Lerp the DCT coefficients of two images

[OmarShehata]

Inspired by this tweet: https://twitter.com/JobvdZwan/status/1400477315738574856

What happens if you lerp between the Fourier representations of two drawings?

What happens if you do that at the 8x8 group level of two JPGs just before the final steps of decompressing them and turning them back into an image?

I think it would be cool to put together a standalone demo that lets you pick two images and have a slider to go back and forth.

I put together a little experiment exploring this idea, that replaces the coefficients of one image with another, one by one, starting with the highest frequency, over time:

fox2cat_spedup.mp4

It's not fast enough to be realtime, but I think that's only because the article is doing a lot of work every time the coefficients change, and maybe that can be optimized a bit.

[OmarShehata]

Just to note this down here, the above demo was created entirely in the browser console. Here's a gist with the code: https://gist.github.com/OmarShehata/895e42474c5604439888426cb57dab75

1. First I copied the coefficients from the "Full Discrete Cosine Transform" editor, of the default cat image
2. Then I opened the article with this URL which opens it using the fox image.
3. Then I defined catData in the fox article:

var catData = `COPY_PASTE_ALL_THE_CAT_COEFFICIENTS`;// from the cat article

4. Then I clear the console so it's not so slow
5. Then I selected the editor with the browser inspector (specifically <div class="ace-editor ace_editor ace-monokai ace_dark">) such that $0 referred to this element
6. Then I pasted in the code from the gist, which will start to swap the coefficients, 100 numbers per frame

[OmarShehata]

Other notes for building a standalone prototype that does this lerp:

• All the editors are written as React components. Here is the "full DCT editor" (that does it for the whole image, not 8x8 blocks): https://github.com/ParametricPress/01-unraveling-the-jpeg/blob/master/components/full-dct-editor.js
• Most of the image decoding/re-encoding code is in ImageUtilities.js
• For example, below is what happens when you make an edit in the editor, and it re-decodes the DCT coefficients and renders the image:

01-unraveling-the-jpeg/components/utils/ImageUtilities.js

Lines 363 to 376 in b655460

	if (this.editMode == 'full-dctLuminance') {
	let values = this.getValuesFromEditor();
	if (this.numberOfCoefficients != undefined) {
	// Fill in missing numbers with 0.
	// This allows the user to easily 0 out chunks at the bottom by deleting them.
	for (let i = values.length; i < this.numberOfCoefficients; i++) {
	values[i] = 0;
	}
	}
	values = this.inverseDct(values);
	this.drawLuminance(values, this.decodedImage.width, this.decodedImage.height);
	}

• And below that is the version that does it for 8x8 blocks in a real JPEG, where decodedImage comes from jpegjs, a 3rd party jpeg encoder/decoder this article is using

01-unraveling-the-jpeg/components/utils/ImageUtilities.js

Lines 378 to 405 in b655460

	if (this.editMode == 'dctLuminance') {
	// Put the edited coefficients back into the image.
	// In the editor, each block MUST be on its own line
	let text = this.editor.getSession().getValue();
	let editorBlocks = text.trim().split('\n');
	let rawYBlocks = this.decodedImage._decoder.frames[0].components[componentMap['Y'] + 1].blocks;
	let Yblocks = [];
	for (let i = 0; i < rawYBlocks.length; i++) {
	for (let j = 0; j < rawYBlocks[i].length; j++) {
	Yblocks.push(rawYBlocks[i][j]);
	}
	}
	for (let i = 0; i < Yblocks.length; i++) {
	let block = '';
	if (i <= editorBlocks.length - 1) block = editorBlocks[i];
	let Yblock = Yblocks[i];
	let values = block.trim().split(' ');
	for (let j = 0; j < 64; j++) {
	let val = 0;
	if (j <= values.length - 1) val = Number(values[j]);
	Yblock[j] = val;
	}
	}
	this.reEncodeImage();
	this.drawDecodedImage();
	}

• It should be easy to make something that just lerps two arrays and uses the existing editor infrastructure to render the frames.
• It may not be realtime. I'm not sure off the top of my head where the bottleneck would be here

[OmarShehata]

And of course it'd be really cool to be able to drag and drop any 2 images to morph them like that