Using ffmpeg.js as media codec and/or media container polyfill

[FluorescentHallucinogen]

Is it possible to use ffmpeg.js as a media codec and/or media container polyfill for the HTML5 <audio>/<video> tag in a web browser to transcode formats that not software/hardware supported?

Example 1: There is a test.htm page with <audio src="test.opus" /> tag, test.opus file encoded using Opus audio codec, but the web browser doesn't support Opus. Is it possible to decode test.opus to uncompressed test.wav (and change the tag to <audio src="test.wav" />)?

Example 2: There is a test.htm page with <video src="test.mp4" /> tag, test.mp4 file encoded using H.264 video codec and AC3 audio codec, but the web browser doesn't support AC3. Is it possible to decode only AC3 audio to uncompressed audio data (PCM) (that supported by browser), but not decode H.264 video (that supported by browser natively) (and view the test.mp4 with working video and audio in the browser)?

[Kagami]

1. Possible, just need to add pcm_s16le encoder and wav muxer.
2. Same, should be possible.

But I don't recommend to use ffmpeg.js in that way. It's not a good idea to force users to wait for entire file to be downloaded, transcoded, etc. You will also need to store it in memory. Take a look at aurora.js
, it should be better suited for your needs.

[FluorescentHallucinogen]

It's not a good idea to force users to wait for entire file to be downloaded, transcoded, etc.

Is it possible to not wait for entire file to be downloaded and transcode it on-the-fly? That's what I mean in #11.

[FluorescentHallucinogen]

FFMPEG is very interesting because it supports almost any
media codec and media container. aurora.js has only a few different decoders.

[Kagami]

You may be able to transcode HTTP stream on the fly, but you won't see the intermediate result in main process because worker should send entire Blob back with sendMessage. Theoretically you may somehow split input in chunks and transcode them one by one but don't do that, it's too hacky.

aurora.js has only a few different decoders

They have plenty of codecs and it should be rather easy to add more.

If you need to decode just any audio format available in libavcodec, then you might consider to write small wrapper in C which will use Emscripten API for HTTP requests (like I mentioned in #11) and call libavcodec API to decode downloadable chunks of audio stream. ffmpeg.js is just not inteded for that purpose, it's more like ffmpeg CLI utility in pure Javascript. You won't play audio files with ffmpeg CLI, right?

You may also take a look at StreamFile.js and OGVPlayer.js from ogv.js project for example of streamable decoder in JS. It downloads stream data via common JS code, fills queue and passes it to C demuxer/decoder compiled with Emscripten.

[duanyao]

You may want to take a look at https://github.com/duanyao/codecbox.js , which provides a decoder API over ffmpeg. Currently only supports Blob as input.

[Kagami]

One crazy idea just came to my mind: you could try to output transcode result to stdout and read that in main process (stdout events). Not sure whether that would work.

[bvibber]

Note that as far as I know there's no way to append data to a live blob URL, so streaming data straight into a video or audio element during transcoding may require using MediaSource Extensions. Beware also that encoding video also tends to be very slow, much slower than real-time at non-trivial resolutions, though if you're only applying it to audio and just remuxing video that might work well enough.

My ogv.js does a full analogue of the media elements, suitable for replacing an audio or video element at runtime for completely script-mediated playback, but the codecs are not yet fully pluggable and it would not handle h.264 or AAC at this time without some poking.

[pinobatch]

You won't play audio files with ffmpeg CLI, right?

What else is the FFplay program intended for?