Importing SWTOR models into Blender An overview

This is just an overview of the current state of SWTOR "slicing" 😛 as it regards importing models and recreating them in the free, open-source Blender 3D app, so that you may get a general idea of how things work.

• Extracting the game's assets first.
• What's easy to do? What's harder?
  • Recreating Player Characters and companions? Automated and super-easy!
  • Rigging and animating: super-easy, too… but there are issues.
  • Weapons and the like mean manual work.
  • Recreating game locations: automated and easy, if imperfect.
• Going deeper: accessing the game's information.
  • Directly accessible, human-readable data.
  • Non-human-readable binary data files that require online reader tools.
  • A middle ground, in progress.
• Available Tools.
  • Apps and Blender add-ons.
  • Online tools.
    • At Jedipedia.net.
    • At TORCommunity.com.
  • Deprecated tools.

Extracting the game's assets first.

To be able to do anything at all, we have to extract all the assets we might need from SWTOR beforehand: either the whole game (almost 100 GB), or all the character-related assets (around 40-50 GB), or all the environmental ones (ditto), etc. (there's some overlapping between those last two). These big extractions could be redone every time there is a new release of the game with new characters and locations.

Once with them at hand, we can automate the gathering of assets for a given model, or quickly search and preview them manually through our computers' desktop search.

(There are means to extract single specific assets from SWTOR: an object, a texture, a sound… but at the moment that's only possible using some web tools, which isn't a very practical approach. As an example: assembling a simple bald player character in their underwear would involve five objects, each with its own materials requiring at least five texturemaps each. Add hair, facial hair, armor, weapons, and the work multiplies. Getting the files one by one would be arduous)

What's easy to do? What's harder?

Recreating Player Characters and companions? Automated and super-easy!

We just use TORCommunity.com's Character Designer (which imitates both SWTOR's Character Creator and Outfitter) to visually build our characters' looks and armor gear, export that as a .zipped data folder, and process it with the relevant Blender add-on. Single click auto-assembling and texturing!

Rigging and animating: super-easy, too… but there are issues.

Applying the game's animating skeletons to game characters is literally a matter of importing the skeleton object, parenting the character's objects to it, and bam! you are done. Our tools automate that, too.

Applying the game's animations to those skeletons? Select the skeleton, import the animation. That's it.

Caveats:

• The biggest one is that SWTOR 64 bit (Game Update 7.2.1 onwards) broke the importing of animations. BUT! The ones from older SWTOR asset extractions are still applicable (we keep a downloadable copy of the game files to be able to extract those and keep them around).

• Physics-based skeleton bones aren't included in those skeleton files, so, things like lekku, capes, long hair, flaps, etc. have to be rigged manually. Knowing a little about Blender's rigging ways makes it easy-ish, even easy to recycle the results between models. One can go more advanced and apply some physics to them by using Blender's tools and/or some quite nice free add-ons that are around.

• There are some imperfections in our processes that lead so some noticeable distortion when applying animations (that one could use to not animate but to just find interesting poses), specially to Body Type 1 characters. Stuff like necks becoming this bit too elongated and such. Easily correctable in static poses (reset any translations and re-adjust a little) but a nuisance in animations.

• There is a whole category of animations that don't work and result in bodies becoming a horror movie's puddle of flesh: additive animations (that are supposed to work on top of other ones). Luckily, the rest are more than enough stuff to have fun with.

Weapons and the like mean manual work.

We have almost nothing automated there. It doesn't help that many weapons use SWTOR's FX system to animate bits of them, which is quite difficult to handle. This means that we have to look for information in Jedipedia's Search and File Reader to locate the requisite objects and materials. Luckily, the models are most often just two or three objects stuck together.

Their texturing is automated to some extent: most of those models use basic materials that one of our Blender add-ons can auto-locate and apply.

Recreating game locations: automated and easy, if imperfect.

We choose a location from the ones listed in Jedipedia's World Viewer, copy-paste its identifier, and search it in Jedipedia's File Reader to get the relevant Area data files. Once exported as zipped .json files (single "room" ones or just the whole area), we process them with another Blender add-on.

• The results are around 90%-95% accurate. The issue here is that many objects' position and orientation depend on those of other objects, and we aren't able to cover all types (FX-driven ones, specially). In practice, this means that some minor furniture elements might appear in the wrong place, size, and orientation, or not appear at all.

• Terrain patches are most definitely a work in progress: we have a provisional Windows command line executable to process them and save them as .obj files that our Area Assemblers auto-import. No UVs yet, though.

• Also, auto-texturing is imperfect, too, as there are many types of material shaders we haven't recreated yet. Let's say we are 75%-80% there for mostly big architectural and landscape stuff, 95%-100% for interiors.

• Finally, we end up with too many Blender "empties" (invisible utility objects for setting transformation hierarchies, also known as "nulls", "dummies", etc. in other 3D and 2D apps). And importing whole planets, which is something everyone does just because 😁, can drive Blender to its knees (there are strategies and importing options to avoid that).

In general, it is good enough for now and there is margin for making it better, but it'll take its time.

Going deeper: accessing the game's information.

From our SWTOR game's installation we can extract and read both the assets and the information necessary to our "slicing" endeavors (reusing the game's 2D and 3D assets for fan art purposes, modding them back into the game or just telling the game to substitute some with some other ones, theorycrafting research…). We won't be able to do everything locally (solely through tools installed on our computers), though: we'll have to use online ones, too.

Mainly, we have two kinds of game information elements: directly human-readable files, and binary files that require online tools to be understood.

Directly accessible, human-readable data.

It's in the form of .xml files spread through the directories of the extracted game assets. We can just use Notepad or any other text editor to open those files and read what they say: objects' IDs, their meshes' filepaths, material definition IDs that lead to material type information, texturemaps' filepaths, etc.

Using those is the most basic way to find what assets to use to assemble characters and their armor, if rather laborious (for every model involved, we have to read the info in one file, that leads us to other file, that leads us to yet another file, etc.). Also, it is the easiest to automate by locally installed tools such as Blender add-ons and scripts, small apps, etc., as those can read ("parse") that information and do all that file jumping and reading and gathering for us.

Sadly, the information they provide is incomplete:

• They don't include an in-game name-to-model-and-material-IDs table we can read (say, from "Canderous Ordo's Pants" to "leg_pant_[bt]_archetype.gr2" object and "leg_pant_light_ge_mtx46ordo01_u_v01.mat" material files), which is just crucial. That means we have to go to Jedipedia.net or TORCommunity.com to find those in-game name-to-ID relationships.

• The information about an identified game model (say, a piece of armor which we know the ModelID of) will list all the "attachment" objects (shoulderpads, pouches, spiky thingies, capes and flaps, etc.) that are compatible with its main object, but not which ones to use for a specific in-game variation.

• The situation is a little better regarding which materials definition files are we meant to use, but still it is incomplete in important ways: they list all the base texturemaps and shader settings to use, but any non-fixed data (say, skin and eyes color, a default armor piece's dyes, a character's complexion texturemap, etc) isn't there. For non-animated models it is usually enough, but for characters and armor it most definitely isn't.

• And so on and so forth.

To fill those holes, we have to recourse to reading binary files through our online tools.

Non-human-readable binary data files that require online reader tools.

They hold the whole game's data! The most fundamental of it all is the relationship between in-game names and internal IDs. Without them, things are just very painful to do, through trial and error. For example, importing our Player Characters in an exact manner would require eyeballing many of their features: which head model, which complexion, which skin color…

• Some of those tools depend on those sites' internal databases, built out of that data: from their search tools to look for things by in-game name, to TORC's Character Designer and both TORC and Jedipedia's 3D Model Viewers.

• Some of them can directly read the game's files without needing to do a lengthy and PC storage space-heavy asset extraction process, and even extract and save individual assets (a single object, a texture, a sound…), but still we have to load the webpages in our webbrowsers (although the actual processing is wholly local, in our PCs). The most important one is Jedipedia's File Reader.

• Jedipedia.net's File Names page is fundamental for up-to-date asset extractions: as the game's .tor files don't store the assets by name but by numerical identifier (file hashes), we depend on this page to generate and download a current names-to-hashes list that Slicers GUI can use (the tool includes one such, but it is several years old, so, we better replace it with one from that page after installing it).

A middle ground, in progress.

One of our tools used to let us export part of that binary data in human-readable formats (.json and/or .xml files) to keep around and maybe automate its processing. The results were rather bulky, though, which makes distribution hard (in some cases, Blender add-ons would balloon in size to the hundreds of MBs.). Also, they would require constant regeneration to keep pace with the game's updates.

The 64 bit version of SWTOR made everything even more difficult, as it broke that exporter tool, so, we are restricted to stuff previous to Game Update 7.2.1 for certain things. (There are ways to export that data out of Jedipedia's File Reader, but it requires dealing with a webbrowser's Javascript console and knowing one's way through File Reader's available functions, so, definitely not user-friendly 😐)

Still, we are using a little of that in some tools such as the SWTOR Area Assembler Add-on, and we are looking at other possible uses.

Available Tools.

Apps and Blender add-ons.

• Slicers GUI Windows app:
  Stand-alone app providing several disparate tools. The most fundamental of them is the game assets extraction one, and the character asset locating one (nowadays superseded by a new Blender add-on).

  (The rest of the tools were in a diverse state of completion but SWTOR's migration to a 64 bit app broke them)

• Blender:
  Blender is a free, gratis, open source, multiplatform (Windows, Mac, Linux), fully featured, complete 3D production environment. It does not only 3D (including modeling, rigging, animation, simulation, 3D-painting, sculpting, etc.) but video editing and compositing, too. You can check its extensive features in the official site at blender.org.

  One of its virtues is that it is automatable through the use of the Python programming language. Blender scripts and Add-ons (equivalent to "plug-ins", "extensions" and similar in other apps) are relatively easy to write. We have a few that automate some of the drudgery of dealing with SWTOR assets.

  (Before downloading the app, check this post for some considerations on which app versions to use and how to get up to speed with it without investing too much time. No, no Donut tutorial required 😛)

• SWTOR's .gr2 Importer Add-on for Blender:
  (AKA "Granny2-Plug-In-Blender-2.8x"). This is the importer/exporter of SWTOR's custom variant of the .gr2 ("granny") 3D model file format. It is also able to process game characters' information exported by TORCommunity.com's tools for automatic reconstruction, SWTOR's rigging skeleton objects, animation files, and has an experimental attempt at importing the game's cloth physics.

  Also, this add-on builds on the fly replicas of the game's basic shaders for objects, creatures, skin, eyes, hair, and armor.

  Out of all those features, only objects, skeleton objects, and characters importing are compatible with SWTOR 64 bit at the moment, but one can use the 32 bit version of animations on them without trouble.

  This add-on's importing features are used by other add-ons, which build upon them more advanced features or more user-friendly processes, such as importing locations or applying in-game name conventions to imported characters' armor gear. It's typical to require its presence for other add-ons to work.

• SWTOR Character Assembler Add-on for Blender:
  (Not up-to-date yet: use the ZG SWTOR Tools version for the time being).

  Processes Player Character and NPC information exported by TORCommunity.com's Character Designer Tool and the NPC database. It enhances the aforementioned .gr2 importer add-on's features (and obviates the need to use the Slicers GUI app's character assets Locate tool first). It's the preferred automatic character importing, assembling, and texturing tool.

• SWTOR Area Assembler Add-on for Blender:
  (Not up-to-date yet: use the ZG SWTOR Tools version for the time being).

  Processes SWTOR's locations information exported by jedipedia.net's tools to import, assemble, and texture their objects save for a few exceptions.

  Companion SWTOR Terrain Exporter Windows command line tool: converts SWTOR heightmaps to .obj format models, in an usable though quite imperfect manner.

• ZeroGravitas (ZG) SWTOR Tools Add-on for Blender:
  A miscellanea of tools (almost an All-in-One, as it still requires the .gr2 Importer Add-on for several of its tools to work) that can come in handy when playing with imported SWTOR assets. Among those, there are:

  • Integrated SWTOR Area Assembler.

  • Integrated SWTOR Character Assembler.

  • Process Named Materials: it is able to auto-texture objects based on the name of their materials, either embedded in the imported game objects or copypasted. Applying it usually suffices to autotexture creatures, weapons, locations, etc.

  • Convert to Custom SWTOR Shaders: converts the .gr2 importer add-on's SWTOR shaders to tricked out versions that offer some more flexibility and can be further modified.

Online tools.

At Jedipedia.net.

• SWTOR Database:
  it lets us search for any in-game term (characters' names, armor gear, locations, etc.), returning a list of entries covering all the ways the item is mentioned in the database: 3D object references, codex entries, abilities, etc. Delving in them, we can get identifiers and paths for finding those in SWTOR's internal data structure and in our game asset extraction files, helping us automate processes to some extent such as autotexturing objects in Blender.

• File Reader:
  it replaces the functionality on many really old apps such as EasyMYP (to inspect the contents of SWTOR's .tor files), NodeViewer (to explore the data in SWTOR's node tree) or PugTools (to view and export game assets). By feeding it the .tor files (all of them at once, typically), it lets us explore just everything and how it is interconnected. What's more, it lets us visualize and export objects, textures, music, area data, etc.).

• File Names: This page keeps track of the latest additions to the list of known game assets and their internal identifiers, which our Slicers GUI tool uses to extract and name them in an usable manner. There are links to download the list (a simple text file: hash_file.txt) combining the most recently released Game Update of SWTOR with those of the Public Test Servers ("Live"). There's also the one from the game's betas before its original release ("Beta"), and both combined ("All").

  (Slicers GUI has an extraction preset that let us export unnamed assets, too, appearing as long numbers-named files. If we recognize them, we can contribute to the list through this page)

We rely on those tools to manually identify and locate everything that we can't automate.

• World Viewer:
  it lists all the game areas in the database and, by feeding it the .tor files, reconstructs them in 3D and lets us flyby them! The list includes the areas' IDs that will let us export area data from the File Reader to feed the Area Assembler Add-on.

At TORCommunity.com.

• Character Designer:
  This is a fundamental tool for Player Character exporting and assembling! It imitates both SWTOR's Character Creator and Outfit Designer, letting us recreate our characters and export a definition file that our tools can use to auto-import and assemble the character in one click!

  It's limited to data previous to SWTOR's Game Update 7.0, though, as its database hasn't been updated since. There are ways to manually modify its output to include more recent armor parts.

• SWTOR Database:
  It lets us search for any in-game term (characters' names, armor gear, locations, etc.), returning a list of entries covering all the ways the item is mentioned in the database: 3D object references, codex entries, abilities, etc. Delving in them, we can get identifiers and paths for finding those in SWTOR's internal data structure and in our game asset extraction files, helping us automate processes to some extent such as autotexturing objects in Blender.

  By searching for NPCs, we can get not only data but a 3D visor with a button that exports a definition file which lets us auto-import and assemble them in the same way we do our Player Characters.

  Again, it's limited to data previous to SWTOR's Game Update 7.0.

Deprecated tools.

• EasyMYP.
  Not compatible with SWTOR 64 bits.

  Works with SWTOR 32 bit, though, so, it can be used to examine legacy SWTOR .tor files and extract assets from them (such as 32 bit animation files that the .gr2 Importer Add-on and Blender can use) in a more selective manner that Slicers GUI. Save for the extracting part, Jedipedia's File Reader can work as a replacement.

• Noesis.
  Not compatible with SWTOR 64 bits.

  In general, Noesis has always been suboptimal for converting SWTOR objects: it mangles their normals and doesn't support the cases where they carry multiple UV maps. Where it was most useful as of late was as a 3D object viewer (able to show textures for non-recolorable objects) and as a single object exporter (not so much for the mesh conversion but for converting SWTOR's multichannel "packed" texturemaps to more conventional ones alongside the mesh, too). Jedipedia's File Reader, if not as agile, can work as an even better object viewer.