@etpinard these are probably well known to all of us, so it's a 'just in case':

• It's currently in alpha, so it's good to think about it, but I'm not aware of a target release date which makes it hard to incorporate in a roadmap
• feels like an overall refactoring and API improvements for an incrementally better D3, rather than a revolution or leap in some direction; some favorites are better support for non-SVG (Canvas, WebGL) targets, better timer, transitions and of course the Matplotlib Viridis palettes
• .. but a real deep article from Bostock seems to be https://medium.com/@mbostock/what-makes-software-good-943557f8a488#.9lyi5ovkg
• while inclusion of D3 4.0 can be done via ES6 modules, it does not mean that payload size would be reduced: because of the method chaining style of D3, if you need the smallest thing from e.g. d3-scale, it'll pull in the entirety of d3-scale
• these are the main options of including D3 4.0 in the project- personally I'd go for
• the improvements in D3 4.0 often mean that the code needs changing, so migration of a large project takes some time

Observations from an aborted attempt:

• I'm sure rollup.js is a wonderful piece of software, but it's appears rather difficult to work into the existing build process because d3 imports only affect d3 through implied side-effects. 😞 Example: I tried to integrate d3-selection-multi. See: How to integrate with existing d3 build? d3/d3-selection-multi#5 for why this is difficult. The only reasonable answer seemed to be to just copy/paste code from d3 modules into the plotly codebase. A more carefully considered solution pulls off some fancy trick I'm not able to figure out related to ES6 modules and their interaction with browserified code. The suggested approach is to modify d3's main index.js to include what you need. But ya can't fork it because then what ends up in node_modules is not compiled because of the prepublish hooks. So seems like you'd need to include d3's index.js in your project, but then you have to figure out lots of module/browserify stuff.
• style({...}) -> styles({...}). Ditto attrs. This change is mostly made in the d3-upgrade branch.
• selections are no longer subclasses of arrays. This means the .selectAll(...)[0][0] indexing trick is now invalid. which is currently the primary show-stopper to pursuing this further
• .data([0]) works fine, but the style through which it's currently achieved breaks since the resulting selection doesn't actually contain the created elements. It's a simple fix. I think I got most of them.
• d3.round is gone and seems not to have a replacement. I fudged it with a quick fix, but this might require more careful consideration.

Beyond that, it's not like the logic is all that different. It's a lot of bouncing from one error to the next and translating the code. The result that I got was able to display a grid with dragging and some hoverlabels, so it's viable, just would take more work than seems appropriate in the short term.

@rreusser nice summary! Yes, lots of relatively superficial upgrade work even with smaller codebases. Various incompatible changes, e.g. the collapsing of the namespaces, various renamings or both (e.g. d3.layout.force -> d3.forceSimulation), or what things return (e.g. some palettes were scales but now they're arrays) or what methods need to be used.

Besides the d3.round thing, there are subtle changes, such as d3.range no longer attempting to control for numerical inaccuracy (I saw it from Irene Ros, great presentation).

I wonder if the selection[0][0] thing can be replaced with .node(). In any case the array access was never part of the API so it's a good idea to avoid it.

Regarding the promise of smaller components, maybe this new tool can help customize what's included but I haven't yet tried. The reason it might work is that there are numerous modules not used in the codebase. In any case it's an almost negligibly small fraction of the plotly.js codebase.

Based on the current way we use D3 I wouldn't expect significant user-visible improvements from a switch. It can be useful for other reasons, I suppose the community (programmers, bug fixes) move to 4.0 over time. Or if there is a big modularization attempt in plotly.js itself, e.g. now lightweight plots as in journals aren't feasible due to the load and initialization time. Some other things also came up earlier, e.g. our use of the palettes (there's duplication between plotly.js and the gl-vis packages. The force layout in 4.0 is deterministic and way faster than currently, see my three recent blocks or these two gifs from Mike Bostock.

All in all D3 4.0 doesn't seem to address a current technical problem but makes some things easier in the future. It's a tough call because most folks can leave old projects in 3.* and use 4.* on new ones but plotly.js is not a dataviz project.

Haha, but array access is soo easy! Come to think of it, I'm not sure it's actually used in the code. That might just be what I use to debug in the inspector.

Yeah, the other change that maybe had consequences was transitions. The current animation PR falls back to instantaneous data updates when you transition the scale and the data at the same time. The reason seems to be a race condition in which the running transitions (points moving around) don't play well with transform changes (the whole layer moving around). It's difficult to debug because of the way v3 makes it relatively difficult to group transitions (i.e. transition.select(); and specifically, difficult at least within the plotly.js codebase). v4 lets you group more precisely by simply passing a transition to a new transition.

It's ripe for debugging and refactoring, but the current priority is consistency and correctness, then handling more complicated interactions and corner cases. I think v3 will get us there. 👍

Good point. Transitions can be a huge thing to upgrade properly (the new is so much better, taking advantage of it may need full rethinking of the transitions). Since you wrote the animations you are the one who knows about the expected transition upgrade friction.

Referencing @rreusser 's #946 (comment) about our the very common .data([0]) pattern.

FYI d3 v5 should make its official debut soon: https://github.com/d3/d3/releases/tag/v5.0.0

Is this topic up2date? - I would propose the following: instead of replacing all the d3-usages in the code I would create a intermediate-d3-wrapper which forwards functions used in plotly to the related functions in d3. In the wrapper itself we could make a version-check of d3 and depending on the version use the right d3-function. Also with this plotly could maybe support different versions and an upgrade to later versions of d3 are maybe easier.

Does this idea/approach make sense somehow?

Pasting #3052 (comment)

For completeness, the items d3v3 is missing that v5 has are:

%Q - milliseconds since UNIX epoch.

%s - seconds since UNIX epoch.

%u - Monday-based (ISO 8601) weekday as a decimal number [1,7].

%V - ISO 8601 week of the year as a decimal number [01, 53].

%f - microseconds as a decimal number [000000, 999999]. Frustratingly, d3 did something different than we did with %f, which we added (before it was part of d3) to support more digits and increased flexibility over %L. They added it as a fixed 6 digits of precision, whereas we support any precision up to 6 digits (using %{n}f) and we truncate trailing zeros. The fixed precision of only 3 (%L) or 6 digits is, IMO, a significant limitation (though I can see wanting to keep trailing zeros, we could easily add that) so I guess we'll be keeping this particular difference from D3 long-term...

Perhaps not worth it until we upgrade our d3, but it would be possible for us to add Q,s,u,V similarly to how we added f.

d3-format v4+ has new formatting features ...

eg.

d3.format("(0.2s")(-1500);
(2k)

if there's no real plans to update to the newer version of d3, is there some recommendation for users to still end up with a ISO8601 weeknum (%V)?

if there's no real plans to update to the newer version of d3, is there some recommendation for users to still end up with a ISO8601 weeknum (%V)?

Is there any plan to implement this?

Also especially usefully in the new d3-format:

The ~ option trims insignificant trailing zeros

Finally, In order to have nice tick positioning and formatting while zooming, I had to use newer d3 (5.15) on the side. I imagine Plotly would just take care of it, once upgraded to a newer version of d3.

Meanwhile, here is my code.
Upon ploty_relayout event:

        // use d3 to compute ticks (a newer version than the one used by Plotly)
        const xNiceScale = d3.scaleLinear().domain(xaxisRange).nice();
        const xTickValues = xNiceScale.ticks();
        const xTickFormatter = xNiceScale.tickFormat(xTickValues.length);
        const xTickLabels = xTickValues.map(xTickFormatter);

        const yNiceScale = d3.scaleLinear().domain(yaxisRange).nice();
        const yTickValues = yNiceScale.ticks();
        const yTickFormatter = yNiceScale.tickFormat(yTickValues.length);
        const yTickLabels = yTickValues.map(yTickFormatter);

        const layout = {
            "xaxis.tickmode": "array",
            "xaxis.tickvals": xTickValues,
            "xaxis.ticktext": xTickLabels,
            "yaxis.tickmode": "array",
            "yaxis.tickvals": yTickValues,
            "yaxis.ticktext": yTickLabels
        };

if there's no real plans to update to the newer version of d3, is there some recommendation for users to still end up with a ISO8601 weeknum (%V)?

This one is addressed in #5026 and would be available in plotly.js v1.55.0.

A table of d3-v3 methods applied in current plotly.js.
The +/- sign indicates that an equivalent v5 sub-module found and could possibly be replaced.

- d3.behavior
+ d3.dispatch
+ d3.ease
- d3.event
+ d3.extent
+ d3.format
+ d3.geo
+ d3.hsl
+ d3.interpolate
+ d3.json
- d3.locale
+ d3.max
+ d3.merge
+ d3.min
+ d3.mouse
+ d3.nest
+ d3.range
- d3.rebind
+ d3.rgb
- d3.round
- d3.scale
+ d3.select
+ d3.sum
+ d3.svg
- d3.time
+ d3.transition
+ d3.zip

🎉 and now d3-v6 is out:
https://observablehq.com/d/f91cccf0cad5e9cb

This issue has been tagged with NEEDS SPON$OR

A community PR for this feature would certainly be welcome, but our experience is deeper features like this are difficult to complete without the Plotly maintainers leading the effort.

Sponsorship range: $70k-$80k

What Sponsorship includes:

• Completion of this feature to the Sponsor's satisfaction, in a manner coherent with the rest of the Plotly.js library and API
• Tests for this feature
• Long-term support (continued support of this feature in the latest version of Plotly.js)
• Documentation at plotly.com/javascript
• Possibility of integrating this feature with Plotly Graphing Libraries (Python, R, F#, Julia, MATLAB, etc)
• Possibility of integrating this feature with Dash
• Feature announcement on community.plotly.com with shout out to Sponsor (or can remain anonymous)
• Gratification of advancing the world's most downloaded, interactive scientific graphing libraries (>50M downloads across supported languages)

Please include the link to this issue when contacting us to discuss.

Just to be clear... Plotly is based on a version (~3.x) of d3 from 2016... The current version of d3 is now 7.4.5. This seems like a major concern given that it's the foundational package for graphing.

@emilykl before you move on, could you please create a confluence page with your findings?