How to temporise node spawn

[arthurmougin]

Hi,
I am using the network solution with a big (500-1000 points,2-5k links) dataset in WebXR. In the experience, you dynamicly load different datasets.
My issue is that whenever I load one dataset, the instanciating generate a massive lag spike.
What can I do about this?
I am open with progressively loading the dataset, node by node, but I can't find how to do it.
Please help!

[dsaffo]

Hello, happy to suggest a few solutions!

Could you share a snippet of how you are currently loading the data? Also does the lag spike come from reading in the file or from generating the meshes with bind or bind instance?

[arthurmougin]

Here is the current script with a sample dataset, I have received 20 different datasets, all in CSV format :'(

You may notice that this code is heavely inspired by the sample though

const firstDot = this.instance;
        //loading dataset
        const id = this.tableau.meta.id;
        const datasetPath = "./assets/arts/csv/formulas_count_" + id + ".csv";
        const data = await d3.csv(datasetPath);

        //preping data for force layout
        const dataNodeNames: string[] = [];
        const dataLinks: any[] = [];
        data.forEach((d) => {
            //{'ing1': number, 'ing2': number, 'count': number}
            if (dataNodeNames.includes(d.ing1) === false) {
                dataNodeNames.push(d.ing1);
            }
            if (dataNodeNames.includes(d.ing2) === false) {
                dataNodeNames.push(d.ing2);
            }
            dataLinks.push({
                source: dataNodeNames.indexOf(d.ing1),
                target: dataNodeNames.indexOf(d.ing2),
                strength: Number.parseInt(d.count),
            });
        });
        const nodes: any[] = dataNodeNames.map((name, i) => {
            return { name: name, index: i, group: 0 };
        });

        // Create a simulation with several forces.
        const simulation = forceSimulation(nodes, 3)
            .force("link", forceLink(dataLinks)) //.strength(0.05).id(d => d.id))
            .force("charge", forceManyBody())
            .force("collide", forceCollide()) //.radius((d) => d.count))
            .force("center", forceCenter(0, 0, 0))
            .on("tick", ticked)
            .on("end", () => simulation.stop());

        this.simulation = simulation;

        //create a "container" or empty mesh to act as the root node for our network
        //childObserver true will ensure the bounding box updates to fit the extend of the children nodes
        let container: ANU.Selection = ANU.bind("container", { childObserver: true });
        //TODO : Make the container grabbable

        container.selected.forEach((n) => {
            //add six dof drag behavior to the container n
            n.addBehavior(new SixDofDragBehavior());
        })

        //Bind a selection of instanced nodes using our sphere mesh and data
        //set the properties we want
        let sphereSelection = container
            .bindInstance(this.originalNode, nodes)
            .position((d) => new Vector3(d.x, d.y, d.z))
            .scaling((d) => new Vector3(100, 100, 100))
            .material((d) => this.originalNode.material)
            .id((d, n, i) => d.id);

        sphereSelection.selected.forEach((n) => {
            (n as InstancedMesh).actionManager = new ActionManager(
                this.app.scene
            );
        });
        sphereSelection
            .action((d, n, i) => {
                return new InterpolateValueAction( //Type of action
                    ActionManager.OnPointerOverTrigger, //Action Trigger
                    n, // The Mesh or Node to Change
                    "scaling", // The Property to Change
                    new Vector3(
                        n.scaling.x * 1.2,
                        n.scaling.y * 1.2,
                        n.scaling.z * 1.2
                    ), //The Value of the Change
                    100 // The time to lerp between old and new value
                );
            })
            .action((d, n, i) => {
                return new InterpolateValueAction( //Same as above but to return to normal
                    ActionManager.OnPointerOutTrigger,
                    n,
                    "scaling",
                    new Vector3(100, 100, 100),
                    100 // The time to lerp between old and new value
                );
            });

        //We will be using a lineSystem mesh for our edges which takes a two dimension array and draws a line for each sub array.
        //lineSystems use one draw call for all line meshes and will be the most performant option
        //This function helps prepare our data for that data structure format.
        let updateLines = (data) => {
            let lines: Vector3[][] = [];
            data.forEach((v, i) => {
                let start = new Vector3(v.source.x, v.source.y, v.source.z);
                let end = new Vector3(v.target.x, v.target.y, v.target.z);
                lines.push([start, end]);
            });
            return lines;
        };

        //bind a selection and set the lines option using our data and function from above*

        let line = container
            .bind(
                "lineSystem",
                //@ts-ignore
                { lines: (d) => updateLines(d), updatable: true },
                [dataLinks]
            )
            .prop("color", new Color4(0, 0, 0, 1))
            .prop("alpha", 0.1);

        //use the run method to access our root node and call normalizeToUnitCube to scale the visualization down to 1x1x1

        container
            .run((d, n) => {
                //@ts-ignore
                n.normalizeToUnitCube();
            })
            .positionY(1);

        const containerMesh = container.selected[0] as Mesh;
        containerMesh.setParent(this.origin);
        containerMesh.position = new Vector3(0, 0, 0);
        containerMesh.scaling = new Vector3(0.0005, 0.0005, 0.0005);
        this.containerMesh = containerMesh;
        this.selection = container;
        // Set the position attributes of links and nodes each time the simulation ticks.
        function ticked() {
            //For the instanced spheres just set a new position
            sphereSelection.position((d, n, i) => new Vector3(d.x, d.y, d.z));
            //For the lines use the run method to replace the lineSystem mesh with a new one.
            //The option instance takes the old mesh and replaces it with a new mesh.

            line.run((d, n, i) =>
                ANU.create(
                    "lineSystem",
                    "edge",
                    //@ts-ignore
                    { lines: updateLines(d), instance: n, updatable: true },
                    d
                )
            );
        }````
        
The data was anonymized, so I can share with you an example :
[formulas_count_ADA.csv](https://github.com/jpmorganchase/anu/files/15342696/formulas_count_ADA.csv)
Spoiler Alert : 40K lines

[dsaffo]

Thanks for sharing this sample! I managed to recreate your example and I have some suggestions you could try.

1. I separated the data processing and network code and ran the chrome performance profiler. The whole tasks takes 124ms and the data prep takes up about half of that time (54ms). So if possible pre-proccess those csv into the target json format before run time.

2. When setting lots of properties on lots of nodes its better to use .props() so that you only loop through the selected nodes list once. Also when using instances try to set as many properties as you can on the root node before instancing, since these properties will get copied anyways.

3. If there is an FPS issue on your device you can reduce the number of vertices in your scene by changing the number of segments the sphere mesh was made with. The default is 32 which is pretty high and you can drop it down to like 16 or lower before you really would notice a difference in VR with how small these spheres are going to be. You could also use icospheres if you dont mind another shape which can be made with even fewer vertices.

4. Try using off screen canvas which will help prevent app logic from locking up the render loop. This is not exactly simple and has its own drawbacks to how you design the application but its results could be worth it : )

[arthurmougin]

Thank you for your amazing suggestions!

While I was abble to take car of 1 and 3, I am not very sure to understand what you mean for the 2nd option.
I am sorry I am not an expert in the d3 ecosystem.
Would you please show me how that would work?

[dsaffo]

Sure for example

sphere.position(...) //each chained call will loop through the list of shares and apply the property
           .scaling(....) //loop again 
           .prop(.....) //loop again

// instead if you use the .props operator you can pass an 
// object of properties and value/functions and Anu will only have to
// loop through spheres once to apply all to properties in the call. 

spheres.props({ "position": (d,n,i) => new Vector3(d.sepalLength, d.sepalWidth, d.petalWidth),
                          'scaling.x': 0.1,
                          'name': (d,n,i) => "iris_sphere:" + I,
                          'renderOutline': true});

[dsaffo]

Also I see you are using Typescript which I do not get a chance to test myself that often. If you see an issue or place our types can be improved please feel free to open an issue! Whenever you complete your application would love to see it in action in the show and tell section!