Skip to content

Assembling SWTOR Game Areas via the SWTOR Area Assembler Addon for Blender

ZeroGravitas edited this page Oct 21, 2024 · 52 revisions

Notice: the most recent version of this tool is available in the ZG SWTOR Tools Add-on (please check its guide here). We'll update the standalone Areea Assembler Add-on as soon as possible.


The SWTOR Area Assembler Addon for Blender, written by ZeroGravitas and Crunch, assembles SWTOR's in-game areas based on the information produced by Jedipedia.net's World Viewer and File Reader online tools. Those tools are able to export lists of objects in an human-readable-ish .json format, which the Area Assembler Blender Addon uses to import the objects and position them correctly.

In this guide we will show you how to use Jedipedia's tools to find and download the game location data in .json format, how to feed it to the addon, and the possibilities and limitations of the addon.

Alt text



Area Assembler.

The Area Assembler tool (co-written with fellow slicer @crunch) assembles SWTOR's in-game areas based on the information produced by Jedipedia.net's World Viewer and File Reader online tools. Those tools are able to export lists of objects in an human-readable-ish .json format, which this one uses to import the objects and position them correctly.

In this guide we will show you:

  1. How to use Jedipedia's tools to find and download the game location data in .json format.
  2. How to feed it to the Area Assembler.
  3. And its possibilities and limitations.

Requirements.

In order to work, this tool needs:

  • Blender 3.6 (or Blender 4.0 if a 4.0-compatible version of the .gr2 Importer Add-on is installed).
  • The latest .gr2 Importer Add-on for Blender.
  • To have selected in this Add-on's Preferences a 'resources' folder from an assets extraction produced with Slicers GUI's 'Full' preset (a smaller, 'Static' preset-based one ought to work, too, but that hasn't been thoroughly verified yet). The Status panel will indicate if such a folder is selected. If not set, both the indicator and the tool will appear in red.

And to obtain SWTOR areas' data, we'll have to:

Installation and configuration.

  1. Download the latest version of the Area Assembler Addon by clicking here, which is packed in a ZIP Folder. DON'T UNZIP IT! Blender handles that automatically when we install the Addon.

  2. Open Blender and Select from the top Menu Bar Edit > Preferences... > Add-ons > Install... which will open a file import navigation window. From this window locate the SWTOR Area Assembler zip file that we downloaded in the step above (NOT the Area.dat ZIP we downloaded in the previous section), select it, and then click Install Add-on.


  3. After a second or two Blender will reload the Add-on's list, and present you with the newly installed Addon. Click in the empty checkbox to enable the Addon, but stay on the page as you will need to populate some information. If Blender doesn't automatically show you the SWTOR: Area Assembler Addon as the only result and instead you see the full list of Addons, use the search bar on the top right of the Preferences window to narrow your results until you find it (type, for example, 'sw', and it should pop up immediately).


  1. After enabling the Addon and expanding its information box by pressing the small down arrow, you will see a Preferences section with a folder path input box labeled SWTOR Resources. For this field you will need to either locate (by clicking the folder icon to produce a file browser) or enter the full path to the assets extraction you did with the Slicers GUI app or similar, up to and including the resources folder you ought to find in there (please refer to the Slicers GUI guide, if necessary).


  1. Once you've set your path, you can close the Preferences Window. It isn't necessary to close and relaunch Blender.

The Add-on will appear as the "SWTOR Area Tools" tab in Blender's 3D View's Sidebar. You can make sure that these steps were correctly done by checking in its panel's header that everything is "SET".

Importing options.

These are the tool's basic importing options. The default settings are quite alright for most cases.

  • Apply Final Rotation (default=on): disable to skip final rotation. All objects will need to be rotated in the X Axis 90 Degrees by the user. Just leave it on.

    (This is more of a diagnostics tool than anything else. Due to Blender's Z-is-up coordinate system being at odds with SWTOR's Y-is-up one, which is a source of complications when using SWTOR objects transformation data and chaining those transformations when parenting objects, we delay the final 90º rotation to the very last moment. Un-ticking this checkbox would bypass that rotation)

  • Process Named Materials (default=on): it applies the Process Named Materials tool (available in the Materials Tools subpanel) on the fly to auto-texture the imported objects. As we don't cover all SWTOR materials yet, depending on the type of area (indoors, outdoors, spaceship interiors) we can get fully textured environments or ones with lots of untextured stuff showing up as white (vegetation, water bodies, etc.).

  • Apply x10 Scene Scale (default=off): automatically scale the entire scene after import by 10x to better match Blender Units.

    SWTOR uses decimeters as coordinate units, so, objects are internally sized as their real world size divided by 10. As many advanced operations in Blender (physics simulations, automatic weighting, etc.) depend on objects having real world sizes, this option will resize the imported areas x10 for us if we need it.

    Such scalings can be done with the Objects Tools subpanel's Quickscaler tool aftewards, anyway.

  • Skip dbo Objects (default=on): Don't import design blockout (DBO) objects such as blockers, portals, etc. Replace them with Blender's "Empties", instead.

    DBO are game engine objects, invisible during gameplay, used by the level designers to set barriers to movement, mob spawning points, etc. Normally, such objects are just a nuisance for the purpose of assembling a game location in Blender, so, the default setting is on.

    However: we aren't fully sure yet about whether some of those objects determine the position, rotation and/or scale values of some visible objects in the scene (that is, they act as their parent objects) or not. Because of that, at the moment we are replacing them with Blender's Empties, which are a nuisance of their own but at least they aren't rendered in the scene. Once we determine they are safe to remove, we'll have a setting do so.

  • Create Scene Lights (default=off): automatically create very basic scene lighting based on in-game lighting nodes' positions and angles (if their numbers exceed 100, they will be set as excluded from view for Blender responsiveness reasons).

  • Collect Objects by Type (default=off): if checked, the areas' objects are further arranged in Area sub-Collections. For each area (.json file), we get this Collection hierarchy:

    • Area name
      • Area name - Lights (if Create Scene Lights is on)
      • Area name - Terrains (if any)
      • Area name - Objects
  • Merge Multi-Mesh Objects (default=off): joins single .gr2 file-originated multi-meshes into single objects.

    Some of SWTOR's .gr2 objects contain several meshes at once. For example, there is an open crate object that has a separate lid and several tools spread around it, plus some invisible blocking object to keep a player from walking across it. These are the objects that slow down the importing process the most, despite some attempts at optimizing it. Then again, those are great objects for kitbashing if we keep their sub-meshes separate.

    Still, if one doesn't care for that, this option would just merge the sub-meshes into a single object, easier to pick and move around in Blender. This might marginally speed up the importing process if there are duplicates of this object listed in the .json files.

  • Full Report In Terminal (default=off): by default, the Add-on will show the progress of each of the importing job's stages in an abbreviated manner: continuously overwriting a single line except when there is some error to report. Ticking this checkbox will show all the lines, instead.

    If set to on, given the tremendous amount of lines a whole world import can mean, we recommend increasing the console app's buffer settings (or the one in a debugging environment such as VSCode's) to some ludicrous size: we are talking about 500 lines per expected .json file or so. This option doesn't make much sense otherwise, as the full report's first half or more would easily get truncated out without an ample buffer size.

Options for reducing Blender's lag when importing MASSIVE Areas.

If we are importing whole worlds because of course we are 😏:

After importing massive amounts of objects, such as a whole planet's set of .json area files, it's typical that Blender's responsiveness to operations such as a innocent Select All lags horribly. We are talking half minutes or worse just to respond to our cursor hovering over a button (if this button has to do a check for a selection of objects before making itself clickable). Even just moving through an Outliner overloaded with thousands of objects will lag considerably.

The following options mitigate or fully eliminate that problem by hiding the objects in two manners that keep Blender from getting choked, and give us a chance to save the project and organize things a little before unhiding them.

  • Hide Objects (default=off): imported Area objects are hidden in the 3D Viewport (as when turning their 'eye' icon off in Outliner or using the h shortcut) to keep Blender more responsive when having massive amounts of objects per individual Collections.

    Recommended when dealing with single .json files weighting in the MegaByte range, such as certain juicy single location/.json file areas where the game developers put ALL the decorations to test them. That means a single resulting Collection with thousands of objects.

    Lag could persist, as the Outliner still has to list the objects, but it should be fairly tolerable.

  • Hide Collections' Contents' (default=off): imported per area (per .json file) Collections are excluded from the View Layer (as when unticking their checkboxes in the Outliner or using the e shortcut) to keep Blender fully responsive and be able to manage them without any lag at all.

    Recommended when importing a massive number of areas, such as whole worlds.

    Excluded Collections won't list their objects in the Outliner, as usual, which immediately alleviates the Outliner's workload and makes Blender brisky. The idea is to activate the specific area Collection or Collections we want to tinker with and deactivate them to move on to others instead of having it all active and crush both poor Blender and our patience.

Excluding Collections resets the hide/show state of the Collections' contents. That means that, if we enable both options, Hide Objects After Importing won't produce any effect.


How to use the Area Assembler tool.

Locating Area IDs, and Downloading Area .json files.

The first step of importing an area is knowing what area we want to import. For the sake of this tutorial we're going to be focusing on the Sith Warrior's 'Fury' player ship. The steps for this area apply to all other areas as well.

  1. First, we navigate to Jedipedia's World Viewer and locate in the Player ships category the Fury (Sith Warrior) entry. Simply clicking on it will populate the AreaID in the text field below the 1. Choose an area heading, as seen below. We copy this Area ID to the clipboard or a text document for use in the next step, and we're done with the World Viewer!

    AreaID Image Here

  2. Now, we navigate to Jedipedia's File Reader (the link is available at the top left corner of the World Viewer page, too) and click the Add Assets button, which will produce a file browser.

    We have to go to the Assets Directory inside SWTOR's game files and select all the .tor files within the folder (pressing a will do it), then open them. Loading them takes very little: from less than 10 to 60 seconds (there are progress indicators).

    Location of the 'Assets' folder:

    • On a Steam Install: C:\Program Files (86)\Steam\steamapps\common\Star Wars – The Old Republic\Assets

    • On a traditional SWTOR Launcher Install: C:\Program Files (x86)\Electronic Arts\BioWare\Star Wars - The Old Republic\Assets

  3. Once all the .tor files have loaded into the File Reader, we switch to the 'Files' tab on the top left, and then, in the input search field, we paste or type in the AreaID we got earlier. In this case, the AreaID is 4611686051284477081. Doing that will narrow the displayed files down to a handful, all relating to the spc_ps_sith_warrior area: these are the .dat files, commonly referred to as area.dats.

  4. Whilst we could go through each area.dat entry and download them individually, in most cases it's quicker and easier to just import the whole area (unless you know you'll only need a certain section, and are willing to spend the time working out which area.dat is the one you require). For this tutorial, we will be importing the entire Sith Warrior's Fury.

    Continently, the .dat file of any area is called area.dat, and will encapsulate all subareas such as ps_sith_exterior.dat, ps_sith_int_barracks.dat, ps_sith_int_captain_room.dat, etc.

  5. So, let's select area.dat from the search list and scroll down until we see a heading labeled Assets with a large button labeled Extract as zipped JSON.

    After clicking it, it might take it a little bit to actually produce the download, depending on the size of the area (should be almost instantaneous for a player ship). We save the ZIP file generated by this button to somewhere easily accessable, and unzip the included directory which contains the .json files we'll require for the final step in Blender.


Processing the area .json files.

  1. Before proceeding any further: as Blender hardly offers any means for proper progress reports, and importing areas is a rather complex task (and quite lengthy if importing whole worlds), we strongly recommend to open the system console to watch the information that this tool (and others in the same Add-on) produces as it goes through it. The option is at top menu bar > Window > Toggle System Console. It gives us progress percentages of each phase of the importing and assembling, which area and objects are being processed, notes on any failures during the process…

  2. We adjust any importing options we need (all checkboxes have tooltips) and click on the Select Area's json Files button, which will open a file browser.

    Alt text

    Alt text

  3. Then we navigate to where ever our area.json files have been saved, which will be in the directory we created when unzipping the area.zip file in Step 1 of this section. Here we can select as many, or as few of the .json files as we like. The importer will process whatever we tell it to. Obviously, the more files we import, and the bigger they are, the longer the process will take. We recommend that, the first time you try the tool, import a single, big .json file, to get a sense of how fast it runs on your PC.

    For this guide, we will select everything by pressing A, or manually highlight all the displayed .json files used by the Sith Warrior's Fury. With these files selected, we click the Import Area .json button, and wait.

    Alt text

  4. For an area like a player ship, on a decent PC with solid state storage, it ought to take less than ten seconds. If we opened the system console we should be able to see a flurry of activity as the tool imports and processes each .gr2 object file.

    (Example showing full report in terminal instead of the default abrevviated view) Alt text

    If we didn't, Blender will appear to be unresponsive save for its icon-cycling cursor. That's normal. If importing something as relatively small as the Fury ship takes more than one minute, that would be definitely odd and you may want to consider starting again. However, open your Blender System Console this time to see where things might have gone wrong.

    The progress of the importing that the console shows goes through these stages:

    1. A fairly quick one where all the .json files' information is being gathered.
    2. The actual importing of the objects with some additional diagnostic data: imports, duplications, discardings… with speed ups and downs (the tool tries to avoid reimporting objects that were imported already).
    3. A parenting pass where child objects are parented to parent objects, inheriting their positions, rotations and sizes accordingly. Some failure warnings are expected here, as the area data doesn't include certain animated objects that could act as parents. See the Known Limitations and To Do sections at the end of this guide.
    4. An object renaming pass.
    5. An object texturing pass.
  5. When the Add-on has finished processing the files, you should see the fully textured model of the Fury

Congratulations, you've assembled your first area. Now go forth, recreate the game in Blender and enjoy your lag 😅!

How Automatic Lighting Works.

The area data exported in the .json files contains references to light sources' positions and rotations, but nothing else about them (so far). As that information might be useful when lighting the locations in Blender, the tool can produce generic point lights out of that data as a reference (this can be enabled via the importing options'checkboxes).

At this moment, things are set so that all the lights in a "room" (a .json file) share settings (light data block) such as type, intensity, color, etc., so, changing one of them will change all the lights in the room. They can always be individualized by using the Object > Relations > Make Single User > Object + Data > Selected Object (or All if you need that) option.

Adding Terrain.

We have preliminary support for processing heightmap-based terrain objects reference data. The Add-on imports terrain objects exported as .obj files and placed in a new folder at resources\world\heightmaps by a Windows command line tool still in development by fellow Slicer UltimaKaosXIII: SWTOR Terrain Extractor. The tool is able to process all the terrains in the game in one go in a few minutes, the results occupying some 8.3 GB.

There are some caveats, such as gaps between some terrain patches, the lack of UV information to apply textures correctly, same goes for which textures and shaders to use.

General information about Objects Types and Empties.

Similar to the case of lights, many objects in the results of an import are linked duplicates sharing mesh data, as that accelerates importing a whole lot. We are looking at the possibility of making them unlinked and see how that affects importing speed and Blender's responsiveness afterwwards. Meanwhile, any linked duplicate can be turned into a fully independent object through the usual means.

To facilitate debugging, the objects have some Custom Properties keeping some data from the .json files such as object ID, parent's object ID, the name of the .json file it came from, and some positional data that Jedipedia pre-calculated to check against. Those properties can be found in the Custom Properties section of both the 3D Viewport Editor sidebar's Item tab and the Properties editor's Object Properties tab.

Known Limitations.

  • We can't do successive imports inside a same Blender project: due to coordinate system issues, we do a final 90º rotation of the scene at the end of each import, and that would tumble any previously imported objects. We are seeing about some better way to solve this.

  • A nuisance about the results of an import is that everything (95%-100%) is correctly positioned, but their origins or parentings are a mess (objects' origins can be far, far from their bases or center of mass). There's a logic behind it, but still it is inconvenient if we want to manipulate them for, say, kitbashing new environments or objects. Not that it makes it impossible, not at all, but still…

    Also, the tools rely on instancing any duplicates present in the areas' .json files to save time and memory during the import, which is, again, a bother when we want to alter the results or collect stuff for kitbashing (we are seeing about adding some kind of instance-to-standalone pass, and how it impacts the already gargantuan whole worlds imports).

  • We added support for "placeables" (holoprojectors, spaceship cockpits' seats, GTN Booths, etc), but it is incomplete, as some types, such as FXspec objects, are difficult to position in the scene adequately. We are seeing about making them present at the very least, leaving its correct placement to the user.

  • Not an Area Assembler issue per se but a Named Materials Processor tool one: SWTOR has some six main materials, which we cover, and are usually enough for interiors, spaceships, and buildings' exteriors, but we still need to implement lots of simpler things like water body shaders, several vegetation types, etc. Which means that imported exterior areas will show many things as textureless, white objects.

  • Those that expect being able to export these areas to other platforms such as VRChat, Roblox, etc., via FBX, will find that our SWTOR materials for Blender don't move well through such formats: they are just too complex and don't even depend on standardized-ish shaders such as the Principled BSDF one.

    Most area-type materials could be reduced to, say, a diffuse map and a principled shader, maybe adding the gloss map too (although it's not meant to be used in that way). The normal maps would need some preprocessing, though, as they are packed with normals, opacity, and emissiveness data channels (see https://github.com/SWTOR-Slicers/WikiPedia/wiki/SWTOR-materials-and-texture-files).

    There is a tool in the Materials Tools subpanel, the Convert To Legacy Materials one, that converts our current SWTOR materials to an older version that is Principled shader-based and also better lends itself to a baking approach. That could help, if maybe not by much. We are seeing how to produce a proper SWTOR-to-FBX-friendly materials tool, but we are just swamped with work, so, no promises.

To Do.

  • Tool-side:

    • See if any standard Python module such as NumPy can be applied somehow to accelerate processing.
    • Check the implications of making the duplicates of imported objects unlinked (each having their own mesh data) on processing speed and Blender responsiveness.
    • imported objects may have Bone Bounds properties. Their duplicates don't, at the moment. Investigate (it might have to do with armature-driven elements such as doors) and see what else duplicates might be missing and how to solve it.
    • See if we can entirely disregard the Empties replacing DBOs.
  • Named Materials Processor tool-related:

    • There are lots of little materials that we need to cover: animated holograms, vegetation, glass and water, stain cards…

Process Named Materials.

Alt text

The Area Assembler, by default, uses this tool under the hood to auto-texture as many objects as it can without needing any action by the user. That said, it can be convenient to have it accessible as its own tool, as it can be used to re-process objects that for some reason might have failed to auto-texture, and also can be applied to non-area-related objects, such as weaponry, vehicles, or even Creatures and other NPCs

Requirements.

  • Selecting a "resources" folder in this Add-on's preferences settings.
  • An enabled SWTOR .gr2 Add-on.

Introduction.

This tool processes all the materials detected in a selection of objects or in all objects in a scene (that is, it reads their materials' names, looks for .mat material definition files with matching names inside an extracted SWTOR assets stash, and parses them), locating their texture maps and linking them to the relevant SWTOR shaders. It's quite speedy, as all the material definition files are centralized in a single folder at resources\art\shaders\materials.

It processes the basic SWTOR shaders (Uber, Creature, Garment, SkinB, HairC and Eyes) plus a few extra ones typical in environmental objects such as EmissiveOnly (glass, holograms), AnimatedUV (animated computer screens, holograms, and others), etc. The list is growing, given that the recently created Area Assembler Add-on produces lots of objects with special shader needs

(You can see the way it works internaly by reading this guide)

Typically, smallish settings such as a player character's spaceship will be processed instantly, but some sets of objects, such as whole game areas or even whole worlds, can easily have thousands of materials, and Blender might become unresponsive while processing them. Its progress can be followed in Blender's Console output, which will show the objects and materials being processed. Some error messages are prone to appear in the console, due to the tool's current limitations: those are expected, and the tool is meant to discard them.

If a selected object's material is shared with objects that haven't been selected (and that's very typical in architectural objects like spaceships or buildings) they'll show those processed materials, too, as if they would have been included in the selection. This is their expected behavior. If needed, the way to avoid this would be to isolate the material we don't want to be processed by converting it into an independent duplicate and changing its name to one that doesn't exist in SWTOR's shaders folder.

If a SWTOR object imported into Blender shows materials with generic names (such as "default", "01 chestSkinB", etc.) that don't exist in the shaders folder, we can rename those materials with ones that do exist in that folder, to have them processed. For example, it's typical for creature-type models such as Akk Dogs to have several material variants (.mat files), which is why the objects use generic names on import.

Limitations.

The information in the material definition files has its limitations: it covers the basic texture maps, but not the recoloring information necessary to produce color variations in armors, dye applications, player character skin colors, complexion textures, etc. That means that Uber and Creature materials will be fully realized, but everything else will be a partial solution at best, better solved through tools like the Character Assembler (TORCommunity.com's Character Designer has access to a database of SWTOR data that provides the missing information so that the Character Assembler can fill those gaps. We can also use Jedipedia's File Viewer to locate what's missing).

Still, sometimes having the textures at hand and fiddling with the shader's palette controls can produce usable results that are better than nothing at all.

Options.

  • Overwrite Materials (off by default): overwrites already present objects's materials, which allows for regenerating materials that might have lost texture maps, or to apply better versions if they happen to have been advanced in new releases of our tools.

  • Collect Collider Objects (on by default): adds all objects with an "Util_collision_hidden" material type or texture map to a Collection named "Collider Objects". Those are normally invisible objects used by the game engine as barriers, typically a nuisance we rather isolate and delete.


Outliner Collections' Visibility.

Alt text

This is a tool of convenience, useful when dealing with Areas that comprise a massive amount of imported .json files. In the case of a whole world import in which we accidentally let all the objects become visible, trying to disable the Collections holding them "manually" (select all Collections in the Outliner and bring up the contextual menu to disable them) could be excruciatingly slow to achieve. This tool's buttons are far faster to use.

Also, they offer a feature that isn't available in Blender, which is to disable and enable selected Collections' hierarchies of Children Collections.

Table of Contents


INTRODUCTION

IMPORTING SWTOR MODELS INTO BLENDER: A BRIEF OVERVIEW.
Check this intro first. Afterwards, you can jump directly to the guides on extracting PCs, NPCs and others.


TOOLS

No need to read this section right now: each extracting/assembling guide explains its required tools anyway.

Applications and Blender Add-on tools:

Online Tools:

Deprecated Tools:

  • EasyMYP (Windows app).
  • Noesis (Windows app).

THE COMPLETE GUIDE TO AUTO-EXTRACTING AND ASSEMBLING PLAYER CHARACTERS AND NPCs:

READ THE BROAD STROKES FIRST: YOU'LL SEE IT'S EASIER THAN YOU THINK!


ASSEMBLING GAME LOCATIONS AUTOMATICALLY:


LOCATING AND ASSEMBLING ASSETS MANUALLY:


OTHER GUIDES (WIP):


MODDING (to do)

Modding isn't working at the moment due to SWTOR's change to a 64bit codebase. It's going to take a while 🙁.


DATAMINING (to do)

  • Overview.
  • Tools.

SWTOR TECHNICAL INFORMATION:


OTHER RESOURCES:

Clone this wiki locally