Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Use ImageBitmap when possible #1131

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

CendioOssman
Copy link
Member

These are much faster than Image object in some browsers, so prefer them when possible.

Found this whilst profiling #1108. Noticed that we were spending a lot of time base64 encoding images and drawing them. So I tried using the newer ImageBitmap objects instead.

This gives a massive performance increase in Firefox (2.7 seconds instead of 5 seconds for my test recording). Unfortunately it seems Chrome has screwed something up here and I get a performance regression there (8 seconds vs 5.5 seconds before).

More investigation is needed, but it looks like it is something worth pursuing.

@CendioOssman
Copy link
Member Author

@jcpunk
Copy link
Contributor

jcpunk commented Dec 3, 2019

Is this still a viable PR?

@samhed
Copy link
Member

samhed commented Dec 8, 2019

I believe we're waiting for a bugfix in Chrome. In a browser that works correctly, this is an improvement.

@CendioOssman CendioOssman marked this pull request as draft June 4, 2020 11:58
@godsic
Copy link

godsic commented Jun 5, 2020

@CendioOssman Are there any plans to merge this? I've tried it with up-to-date code here godsic@5f0f226 and it indeed reduces imageRect(...) footprint from ~14 ms to ~5 ms for my particular case.

@samhed samhed removed the WIP label Jun 5, 2020
@CendioOssman
Copy link
Member Author

Eventually. Would need to re-benchmark it and make sure it isn't causing any regression on one of the browsers.

@godsic
Copy link

godsic commented Jun 8, 2020

@CendioOssman Good, looking forward to. In the meantime, I've properly integrated that promised image into rendering queue, see godsic@2300bb6 . So now it renders correctly.

@vbabenko
Copy link

what is the status of this PR?

@CendioOssman
Copy link
Member Author

Needs a retest to see how it performs on current browsers. Might still be a problem on some browsers.

@godsic
Copy link

godsic commented Oct 1, 2020

@vbabenko You can give it a try at https://github.com/godsic/noVNC .

@jcpunk
Copy link
Contributor

jcpunk commented Oct 22, 2020

Can this be rebased off of current HEAD to help simplify testing?

The render queue doesn't need to know how to wait for images to
load.
Otherwise each element isn't considered a single byte. This breaks
conversion to other types, e.g. Blob.
These are much faster than Image object in some browsers, so prefer
them when possible.
@CendioOssman
Copy link
Member Author

Sure. It's now rebased.

@samhed samhed added this to the Future Features milestone Aug 26, 2021
@symbalis
Copy link

symbalis commented Jun 15, 2023

I have been playing with this patch and was pleased with the performance improvement.
I just found that there was more GC collector issue in my context. I am sending 1000x1000 at 30fps.
I fixed the GC collector issue by forcing TVNC to send the full image instead of 10-15 jpegs of 1000x64.

also forced img.close(); in drawImage and a.img = null; in _scanRenderQ() but not sure it impacted the GC

@Mr-Shibari
Copy link

I was also playing around with this (not knowing of this pull) and implemented some other changes as well. I've created a gist with the changes I made to display.js you can find here: https://gist.github.com/Mr-Shibari/70dfd6a576a3bbbd060833982e11a9cc

It works great in my tests but I observe the same issue with the Firefox garbage collector causing janks (sudden frame drops) as symbalis did. The problem seems to be caused by createImageBitmap not releasing all resources, even after calling bitmap.close() on the generated bitmap causing the GC to kick in periodically. Not calling createImageBitmap solves this so it has te be that function causing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants