WebGPU upgrade

[jeffscottward]

Increasing Access

More devices can run sketches smoothly. If complicated enough, a sketch can clog the thread. Needs to be offloaded to GPU especially for more cost effective, lower power devices. More efficient routing of power & compute, means more people can experience creative works!

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature request details

Feature detect if WebGPU is available.
If yes then use that “version” of p5.
https://caniuse.com/?search=webgpu

Personal justification: starting an XR art platform, want to use P5.xr. Want the smoothest experience and for my Quest headset not to overheat or freeze up. Also would be a shame to not make use of Apples full hardware spec when their headset goes live.

[welcome]

Welcome! 👋 Thanks for opening your first issue here! And to ensure the community is able to respond to your issue, please make sure to fill out the inputs in the issue forms. Thank you!

[jeffscottward]

I see WebGL2 was added recently, awesome work! I suspect WebGPU is a way heavier lift.

[aferriss]

Part of the argument for including webGL2 was that it required basically no changes to the code base or renderer. I'm not experienced with webGPU at all but my understanding is that it would require a complete overhaul, as the api is completely different than webGL.

In the caniuse link you shared, it seems webGPU is stil lacking a lot of support, especially for mobile devices. So I think there would need to be a really strong argument to bring it to p5 with that little support. It took webGL years and years to reach broad device support, I suspect a similar time frame may happen with webGPU. You mentioned wanting to use webGPU with apples new headset, but they don't support webGPU in Safari at all right now.

p5 isn't necessarily trying to be the most performant library out there, so another library might fulfill your use case better at this point. And while I agree perf is important, I think for now our efforts might be better spent improving the webGL tools we already have.

[jeffscottward]

Part of the argument for including webGL2 was that it required basically no changes to the code base or renderer. I'm not experienced with webGPU at all but my understanding is that it would require a complete overhaul, as the api is completely different than webGL.

In the caniuse link you shared, it seems webGPU is stil lacking a lot of support, especially for mobile devices. So I think there would need to be a really strong argument to bring it to p5 with that little support. It took webGL years and years to reach broad device support, I suspect a similar time frame may happen with webGPU. You mentioned wanting to use webGPU with apples new headset, but they don't support webGPU in Safari at all right now.

p5 isn't necessarily trying to be the most performant library out there, so another library might fulfill your use case better at this point. And while I agree perf is important, I think for now our efforts might be better spent improving the webGL tools we already have.

Yea a complete overhaul sounds about right. You have to rewrite all the low level bindings. Basically a new "version" entirely guided by a platform support IF statement.

RE Can I use:
It's a little misleading visually but if you look carefully, it's either fully implemented or can be enabled via flag or is in preview across all modern browsers on desktop. Not only is Safari going to support it, you can even test the current beta build. I don't know where you get "at all" from.
For mobile you need to look at the percentages, not just boxes. I'll agree for now even still but as soon as Chrome for Android supports, that's an immediate 40% jump (which should come very soon I'd imagine). The rest of the boxes are all like 1% so it's negligible. Also Safari could change overnight with iOS 17 with the way they do updates.

The point being that since it's such a heavy rewrite anyways, that starting now and meeting the market at the appropriate time rather than way later by only starting to do such a big rewrite after there is such deep penetration means it would put it way way behind. Spatial computing is starting to arrive, so at least getting started would make sense no? I suppose submitting a PR on a parallel branch with a new resource couldn't hurt if I can find them.

WebGL tooling: I'm not aware of the current state of WebGL tooling in P5 so I'll concede it may be more worth it do that right now.

P5 is the tool for the job for me with or without GPU due to the whole community, the playground, the plugins etc, so switching is a no go :) "it's simply too good 😏"

[aferriss]

One thing that would be helpful to understand here are what would be the benefits of bringing webGPU to p5. What are the specific use cases people would need? Is it just to improve performance or are there other benefits as well?

[jeffscottward]

One thing that would be helpful to understand here are what would be the benefits of bringing webGPU to p5. What are the specific use cases people would need? Is it just to improve performance or are there other benefits as well?

My use case is WebXR live coding inside AR/VR. As well as just viewing complicated scenes.

https://x.com/mrdoob/status/1665896482808164352?s=46&t=zsjvsRFuxqFkYW8y9jxV1A

Mr Doob seen here doing it for ThreeJS with live updates (don't mind the failed Bluetooth keyboard experiment).

So one wrong line of code and you can send the CPU to the sun. ThreeJS is working on WebGPU already.

I'm imaging a kid in high school with a low cost Quest 3 headset, live coding something, perhaps in their art class, or at home.

EDIT: I guess I'm just confused if Three.js is doing WebGPU, why isn't p5?
https://github.com/mrdoob/three.js/blob/dev/examples/jsm/renderers/webgpu/WebGPUBackend.js#L30

[davepagurek]

if Three.js is doing WebGPU, why isn't p5?

These are the main challenges as I see them currently:

• The p5 WebGL api isn't what I'd call "feature complete" yet in terms of the tools it provides, so it's less simple for p5 to create another renderer in parallel to meet the spec for the WebGL renderer, as the WebGL renderer is still evolving. E.g. here are some APIs that we don't yet have but might build that would potentially affect p5's architecture:
  • an extendable material system
  • the ability to add custom buffers to geometry
  • using textures for material parameters
  • animated model import (e.g. gltf/glb)
  • geometry processing functionality (e.g. CSG)
• ThreeJS has many active contributors, and the entirety of the project is focused on 3D. P5's WebGL mode is just one part of P5 that partially tries to maintain parity with 2D mode, and it currently has fewer active maintainers. Given our resources, it makes most sense to focus our efforts on the tools that will affect the most people, which is currently still WebGL. (If there are people interested in taking on this task, I'm happy to help guide it!)

Some possible benefits of waiting a bit:

• More options for cross-compiling to both WebGL and WebGPU. Some already exist, like wgpu, but that currently requires building in Rust, which is probably too big of a departure for p5
• p5 is intentionally a singular, big package for now for ease of writing. That's not great for distribution though -- there are some issues talking about splitting p5 into separate importable packages (so that one doesn't have WebGPU code in every build.) This is probably a blocker for releasing WebGPU code since many users can't run it.

I think WebGPU has features we might want, like a more flexible pipeline, which could be good for GPGPU things, or for alternate types of rendering like this: https://wrighter.xyz/blog/2023_08_17_flame_fractals_in_comp_shader

[jeffscottward]

Yea if I can allocate a resource after a fundraise I will.
I need an expert 3D web person to fulltime fix or create the tools across the ecosystem like React Three Fiber, P5, ThreeJS, ModelViewer, etc.

All makes sense though. Wish there was some way I could contribute myself today.

EDIT: perhaps the splitting of packages. Is there an issue for that or just discussion?

[davepagurek]

Yep! There's some discussion here: #5740

[jeffscottward]

Closing this to move the discussion over there.