How it works (documentation)

[jonaskello]

Good work on this package so far!

I have breifly looked through the code and are trying to understand the details of how it works. In doing so I thought I would write my findings down and maybe it could be added to a "How it works" section of the readme in the future. It may also help in discussion of improvements to the algoritm. So here it is, please let me know what I have misunderstood :-).

How it works

SystemJS needs three pieces of information to load a package:

P1. Extended package.json information (eg. format: cjs).
P2. The URL to the package root.
P3. Map to dependent packages.

systemjs-config-builder provides the above information for each package by this algoritm:

A1. Walk node_modules recursively and look for package.json. Build a registry (map) of packages with key name@version and value {location}. The recursive part ensures that packages that has private dependencies in their own node_modules folder are included in the registry.

A2. Iterate the registry and call jspm to generate the extended package.json information needed (P1). The registry values now have {location, config}.

A3. Loop through the registry and write information to a generated.config.js file.

A3.1 Output path section (P2). This section is hardcoded to this:

"paths": {
    "nm:": "node_modules/"
  }

A3.2 Output map section. For each package (1...N) set name to registry key minus @version part and output:

"map": {
    "[name1]": "nm:[name1]",
    ...
    "[nameN]": "nm:[nameN]",
}

A3.3 Output packages section. For each package (1...N) set name to registry key minus @version part, and config from the registry value and output:

  "packages": {
    "nm:[name1]": {
      [config1]
    },
    ...
    "nm:[nameN]": {
      [configN]
    }
}

[jonaskello]

So if I understand correctly the (P3) requirement is not fulfilled by the algorithm yet?

[alexisvincent]

Awesome! Needs one or two modifications but thats the overall gist :) I need to sort something out now. But will reply a bit later.

Great work though :D

[alexisvincent]

Sorry that I'm only getting to this now, been a long weekend. I'm just going to say explain the overall approach as if I were explaining it fresh.

To start with, we need the following two things:

1. Where to find packages
2. How to load packages

There are two primary data structures I operate on, tree (badly named since it doesn't really form a tree, trace would be better), and registry.

tree is essentially a representation of all levels of node_modules and looks as follows:

[
   {
     name: 'react',
     version: '15.4.1',
     deps: <array like one above (recursive)>
   },
   ...
]

registry is more interesting, it's a flat map of name@version to an object with the following example information (after transformations):

{
    name: 'react',
    version: '15.4.1',
    key: '[email protected]',
    location: 'node_modules/react',
    config: <package.json file for that package (with extra compatibility info)>
}

In the case that there are two locations of a single version of a package, we pick one (I think the last one). This is fine since we control the mapping, and is actually helpful for caching.

Almost all functions are transformations of {tree, registry}.

To find packages we take the approach of traversing node_modules (more universal then using a yarn lock). This is performed by traceModuleTree. Which returns the final tree object, populating the registry as it goes.

At this point, all packages can be located, however, there are some intricacies loading node modules in the browser. To make the packages browser friendly we piggyback off jspm/npm (which is the same logic that jspm uses to generate the config files that sit at jspm_packages/[package@version].json), to augment the registry config keys with extra info to tell SystemJS how to load it in the browser. To do this, we feed the package's package.json and it's location to jspm/npm, this gives back the how to load this package info. This then gets merged in to the original package.json.

At this point we have all locations and know how to load the packages, so we just need to format this in a way SystemJS understands, so we build a config object.

All top level node_modules (first level in the tree) have their names mapped directly to their location as in:

{
    ...
    paths: {
        ...
        'nm:': 'node_modules/'
    },
    map: {
        ...
        'react': 'nm:react'
    }
}

The nm: mapping is just there to reduce the size of the config.

Now we add package information so that packages can correctly be resolved (only one per version of a package) by making packages entries with the augmented config (filtering only the keys we want).

{
    ...
    "packages": {
        ...
        "nm:react": {
          "meta": {
             "*": {
               "globals": {
                 "process": "process"
               }
             },
             "*.json": {
               "format": "json"
             },
             "dist/react-with-addons.js": {
                 "cjsRequireDetection": false
              },
              "dist/react.js": {
                 "cjsRequireDetection": false
              }
           },
           "main": "react.js",
           "format": "cjs"
        },
    }
}

for nested modules it looks like this:

{
    ...
    "packages": {
        ...
        "nm:collect-all/node_modules/stream-via": {
          ...
        }
    }
}

We're almost done, at the moment we only have mappings for top level node_modules. So we traverse the tree, and add mapping entries to all packages that require their own nested node_modules. Using the collect-all as an example again the following is added to the collect-all packages entry.

    "nm:collect-all": {
      "map": {
        "stream-via": "node_modules/collect-all/node_modules/stream-via"
      },
      ...
    }

The map entry should be fixed (optimized) to map to nm:collect-all/node_modules/stream-via.

Finally, we also include mappings for our core node libs (needs work), as follows:

{
  "map": {
    ...
    "https": "nm:jspm-nodelibs-https",
    "module": "nm:jspm-nodelibs-module",
    ...
    "vm": "nm:jspm-nodelibs-vm",
  }

nm:jspm-nodelibs-* mappings and package configs are already there since they're installed in node_modules.

Some other hacks are currently hardcoded in for the moment until a more general solution is found. But this is basically how everything works. @jonaskello Well done on your write up, you got 98% of everything. Just wanted to give a full description.