Check for HDR dependencies in Web Animation APIs [CHECKED: None]

[mdrejhon]

Task: Need to check HDR dependencies in Web Animation API changes

We are making long-term plans to migrate requestAnimationFrame() systems to other Web Aniumation standards as a low-scientific-quality fallback (see issues in #15) for Apple devices. (TLDR: The current Web Animation sanctioned APIs after rAF() deprecation doesn't provide enough feedback information for framedrop-detection necessary for scientific integrity)

We need to know the status of planned HDR support in Web Animation APIs because this will be critical to many future TestUFO HDR tests, including potentially accessing pixel information inside a non-DRM test pattern video file, or direct renders (WebGL, Web Animation API).

Many Apple devices includes HDR support (e.g. iPad Pro), and animations will look better with true HDR support -- or at least a web-based API for SDR-to-HDR tone mapping, to be able to playback existing SDR web animations into a web-developer-specified HDR colorspace.

We are open to many workaround ideas, but this is all for naught if Web Animation API creates some issues preventing HDR support;.

We have witnessed situations where when we use an API within API, we lose certain features (e.g. HDR) -- e.g. wrappering HDR content into a canvas (SDR-only API) can lose HDR information.

If there is no HDR dependency in the new Web Animation APIs, ideally confirm in the standard, and go ahead to close this issue.

[smfr]

I don't believe there is any dependency between animation frame rate and HDR.

I'm not sure why you qualify Web Animations as a "low-scientific-quality fallback". They should provide identical callback reliability to requestAnimationFrame.

[mdrejhon]

I don't believe there is any dependency between animation frame rate and HDR.

Thanks; sometimes unintentional happen, like fallbacks (e.g. HDR getting lost) so I wanted to check.
In this case, this issue can be closed; though kept around as a no-dependency answer;

I'm not sure why you qualify Web Animations as a "low-scientific-quality fallback". They should provide identical callback reliability to requestAnimationFrame.

I just posted my reply to this at: #89 (comment)
(More appropriate venue)
Including two questions for you...

Meanwhile, you may close this item, if you're absolutely 100% sure there's no HDR dependencies.

HDR support in Web Animation API would become another good reason we wouldn't mind moving away from rAF() eventually
(though it will take a long while)

[smfr]

Just to be clear, by "HDR" do you mean "high dynamic range", i.e. brighter colors? I'm not sure why you're saying "HDR in Web Animations" since those are two unrelated things.

[mdrejhon]

Just to be clear, by "HDR" do you mean "high dynamic range", i.e. brighter colors? I'm not sure why you're saying "HDR in Web Animations" since those are two unrelated things.

Yes.

Just wanted to make sure.

rAF() is fundamentally a SDR-only animation API.

For example, the classic CANVAS doesn't natively support HDR colorspace. There's two things involved;

(A) Formats such as 32-bit floating point RGB instead of common integer 24-bit RGB), and there's no guarantee that you can pre-multiply to access HDR colors if you try to use third party HDR canvas https://github.com/brianreavis/canvas.hdr.js

(B) It might be remapped to a SDR subset of the display's HDR color gamut at the OS/driver level, since there's no way to discovery whether the browser is capable of doing it.

Many times HDR support is confined to video files (with HDR information), but not the various drawing commands needed to render the animations TestUFO does.

Apple already supports HDR formats in both iOS and MacOS for apps, we just want to see this ported into web standards;

Apple does it via normalization method:
https://developer.apple.com/forums/thread/677273
Some developer docs on Apple HDR10 APIs
https://developer.apple.com/documentation/metal/hdr_content/displaying_hdr_content_in_a_metal_layer
Via executable apps.

There are similar APIs on Windows now too.

I can settle for normalized as long as discovery is available (detecting HDR capabilities) is accessible via browser APIs, so I know how the normalized colorspace is handled by the display / OS. WebGL is the easiest path to HDR, but we'd rather not use WebGL (if at all possible).

The problem in many situations is it's not possible to execute maximum-brightness pixels by standard CANVAS -- some implementations use a different (smaller) range of possible-accessible pixel brightnesses foir CANVAS than for a video file.

An example is on a ~1500-nit capable display, canvas are often showing only 0-400nit pixels, and forcing me to use other APIs to access ~1500-nit pixels (at 1% windowing -- many displays only allow up to 1% pixels such as sun reflections off metal to become super bright). rAF() canvas is largely a SDR technology, so we have to move away rAF() APIs eventually anyway.

For those people unfamiliar with HDR -- let's talk about the holy grail briefly (because it might be in, say, a future Apple Display). Sony showed off a 10,000 nit prototype display too; in those cases, such brightness only occurs when less than 1% of pixels are given that brightness (e.g. the sun, or streetlamps, or neon, or sun reflections off metal, etc), the pixel brightness is remapped on a sliding scale, the more and more pixels you have (like your iris closing when things get too bright, but display doing it so your iris doesn't have to be disturbed). With future MicroLED displays (LEDs per pixel) you no longer need local dimming, and it's possible to have a single 10,000nit pixel next to total black (deep space), accurately simulating a very bright star pinprick despite being a tiny retina-resolution pixel; the applications are endless and stunning. HDR is not to make the whole screen bright, but to add really nice highlights realistically.

Most people agree that HDR API standardization is a royal mess; I just want to make sure there aren't accidental (unintentional) issues being created, where current planning accidentally makes it hard to add future HDR support to it in future.

Normalization to 0.0000 thru 1.0000 is usually the easiest developer solution. The best mental exercise is a floating point RGB(0.0 thru 1.0, 0.0 thru 1.0, 0.0 thru 1.0) and make sure that future drawing APIs are at least theoretically compatible with future native floating-point-color framebuffers; Then discovery (checking an API) is used to query what subset of 0.0000-1.0000 is mapped as the OS default RGB888 colorspace used for PNGs, it's often only a 0-400 nit subset.

Also remapping RGB888 over the ultra-stretched HDR colorspace (to 10,000 nits) means RGB(0,0,0) and RGB(1,1,1) has extremely large brightness difference much more similar to today's RGB(0,0,0) vs RGB(10,10,10). So stretching/normalizing low precision integer colorspace isn't the proper solution -- you need far more bitdepth than 24bit to prevent banding with HDR. That's why some are needing to use ginormous colorspace like 48bit integer colorspace, however floating point uses less memory (32-bit floating point HDR that uses 10-bit floats per color channel, or things like that) because it's logarithmically spread to fit the human vision better. All the different formats is why there's HDR10 vs HLG vs DolbyVision -- there's really many ways to represent HDR pixels but most now use floating-point color instead of standard integer color.

TL;DR: Integer color often are prevented access to the brightest pixel colors.

Integer RGB buffers and integer RGB APIs usually have no access to the display's brightest pixels.

That's why when you blit SDR content such as a PNG, or use a Canvas2D API, it's often just a subset of the display's HDR colorspace. The OS prevents it -- no access to HDR pixel color range from SDR APIs, etc.

They did it so you can't simply display a white PNG file and suddenly have eye-searing max-nit screen. HDR content also sometimes have realtime windowing metadata attached (e.g. HDR video files, Dolby Vision), so during scene changes you may have a different normalization next frame than previous frame. (this requires access to HDR tone mapping APIs, something not standardized yet in browsers outside the video sphere).

Either way...

There are many HDR workflows, but bottom line: Make sure drawing APIs of future animation standards don't assume RGB888 -- and that there is a graceful upgrade path in the next 5-10 years. In other words, please don't make the new animation API as limited as rAF() from a HDR point of view... Basically, be color-format agnostic and mentally napkin-exercise it to theoretically support both integer and floating point color; etc.

It's quite a long chain of support:

• HDR capable display
• HDR capable OS
• HDR capable graphics buffers
• HDR capable API

The iPad Pro has it complete, as does MacOS on the latest Cinema display, although it is "not-on-steroids" as some other implementations (Sony 10,000 nit display prototype from pre-pandemic).

How does this affect Web Animation API standardization stakeholders?

For now, the simplest napkin mental exercise for protecting future HDR expansion to the new web animation API -- is make sure to support floating point RGB(0.0 - 1.0, 0.0 - 1.0, 0.0 - 1.0) without caring about HDR vs SDR vs how tonemapping is done -- that can be done by a completely separate future API that is global to the entire full screen browser).

The bonus is that floating point RGB is inordinately flexible superset of the precision of all currenlty-invented integer colorspace, so it protects for all possible theoreticals, while leaving gamut mapping to a separate API outside the animation API.

Initially, this is simply long-term planning. Visioning exercises initially, although some browser vendors want to accelerate it a bit... (including hacks like overriding the OS SDR-limiting by stretching RGB88 to floating-point HDR, via blitting SDR buffers (plus math) it to a HDR framebuffer in WebGL, etc. (like Canvas2D hooked to WebGL in a very hacky low-performance way that has banding problems). Better yet to protect a path to eventually supporting floating point color natively, as some GPUs now support it.

[mdrejhon]

Please close this item if there's no confirmed HDR dependencies in new Web Animations API

[mdrejhon]

Update -- Chromium nightly build supports HDR Canvas. Might be worth checking if Web Animations API will work with HDR canvases, for all drawing commands, not just drawing just e.g. a HDR AVIF bitmap. Basically vet all the API to make sure it has an HDR path (e.g. brighter-than-white support)

There are some developing HDR standards:
https://w3c.github.io/ColorWeb-CG/

Testing this requires a bleeding edge browser:

1. Download Chromium Canary nightly

2. Enable the HDR features via chrome://flags Experiment Web Features
   

3. Then view a prototype page with HDR-enhanced 2D Canvas and WebGL:
   https://ccameron-chromium.github.io/webgl-examples/canvas-hdr.html

If HDR-enabled APIs are running, then you will see all 6 images appear instantly. The first two is CANVAS 2D and the second two is WEBGL, while the last two is AVIF (I also understand that Apple HEIC/HEIF is also HDR-enabled).

You may wish to do a quick vetting to make sure that the new Web Animation API can work HDR-enabled in the future; designing a new API needs to be futureproofed as much as possible...

[mdrejhon]

UPDATE:
I also found https://www.w3.org/Graphics/Color/Workshop/ though this is probably different from WHATWG

Within that workshop I noticed a PDF with an interesting proposal:
https://www.w3.org/Graphics/Color/Workshop/slids/Russell.pdf

One thing I noticed is that some WebGL paths use floating-point RGB, including color channel values above 1.0 for colors above SDR, e.g. 2.0 for super-red that is redder than sRGB red.

They haven't finalized how HDR color will be represented in WebGL, but this is worth watching closely.

This is different from the classical planar integar RGB that drawing animations of requestAnimationFrame uses, but I will eventually require drawing APIs capable of deeper color, brighter-than-sRGB colors, etc. There are many ways to theoretically normalize/represent HDR color data.

I realize WHATWG may have a different idea, but regardless, whatever idea is chosen, make sure Web Animations API supports a graceful upgrade path to HDR drawing for things like pixel plotting, line drawing, shape drawing, bitmap sprites (AVIF or HEIF) blending modes, etc.

TestUFO is planned to support various kinds of HDR tests when we replace requestAnimationFrame() later this decade. In fact, as of this year, we actually already have an active HDR contract on HDR prototype test patterns, using the experimental Chromium path, but I want to be ready for the Apple-compatible path too;

From a quick glance, some of the MacOS APIs already support floating point HDR color (for OpenGL and video at least), but we need HDR in standard 2D drawing APIs.

[mdrejhon]

I spent some time to look carefully at possible dependences on Web Animation API, and I don't see any dependency with HDR.

This is because we are preparing to try and support Web Animations API, but we need to make sure we can somehow help influence the API to meet our future needs. Even requestAnimationFrame() works with HDR canvas, so we may be stuck with it for a decade (or more), but we will be attempting to port to parallel APIs (e.g. WebGL or Web Animations API or other)

Our further research still further confirms Web Animation API is still lower science-quality than requestAnimationFrame() for our specific needs. More info why framedrop detection is harder with other APIs than requestAnimationFrame() can seen at #93 ...

Nontheless, we want to make sure we influence Web Animation API to be compatible with our future needs. We'll post a bugtracking item at the appropriate web consortiums (W3C and WHATWG), to see if we can add a minor amendment (a per frame callback specifically for frame timing / frame drop measurement, not for drawing) -- as we need to measure the now() timestamp of frame presentation (i.e. Present() or glxxSwapBuffers() inside the browser engine) -- to Web Animation API to allow it to successfully replace requestAnimationFrame() for scientific-accuracy needs #93

So HDR compatibility and frame-timing-callback ability are mandatory prereq's for our ability to replace requestAnimationFrame()

[grorg]

rAF() is fundamentally a SDR-only animation API.

I don't understand. What does rAF have to do with SDR/HDR? It's just a call to a script. Web Animations is practically the same.

[mdrejhon]

rAF() is fundamentally a SDR-only animation API.

I don't understand. What does rAF have to do with SDR/HDR? It's just a call to a script. Web Animations is practically the same.

It's an outdated claim now -- so I take that part back. It was simply a references to CANVAS being a fundamentally SDR technology, but CANVAS is not necessarily tied to rAF().

After that (see later comments), I discovered the new Chrome Canary build with the HDR CANVAS support, so I take the statement back.

Initially, I opened this item as a precautionary move, to determine for unintended consequences (very common during standardization).

I've already asked to close this item -- I've now checked much more closely (spent >10 hours) and I can safely say to close this issue, I don't see any dependencies anymore, and the CANVAS system is already being upgraded to be HDR compatible in the current ongoing standardizations (at least at the W3C side, which I hope WebKit/WHATWG will (mostly) sync with).

If someone else wants to vet to doublecheck, but I think it's time to close this issue.

I've spent over 10 hours already checking for "unintended consequence" effects (unintended interactions with other standards including web HDR), and I don't see any.

I now consider this issue CLOSED