Media Containers API

[chrisn]

Originally raised in w3c/webcodecs#24:

It comes up as a common question: can we have an API for media containers? It's something that can be done JS/wasm and is arguably orthogonal to WebCodecs. But for some formats that you might consider video (GIF, (M)JPEG), the line is blurry between container and codec.

This is a tracking issue for a conversation around this topic. My current opinion is to leave it out of WebCodecs until it's more mature and then perhaps readdress it later.

Current thinking in the Media WG is that container parsing can be left to JavaScript or WASM, and that open source libraries fill the gap. Please comment here if you still see a need for a browser API.

This issue tracks a media-related issue/proposal/work item/discussion

[dalecurtis]

Some brief thoughts on work since WebCodecs:

• We've had several partners express interest in containers, both for demuxing or muxing. They are often able to make mp4box.js work (or less frequently jswebm), but sometimes need a WASM ffmpeg for more format support. Linking them to the incomplete PR we have for that is often helpful: Add ffmpeg-wasm-demuxer sample webcodecs#549
• Mostly everyone we've encountered just needs basic muxing, demuxing with track selection and seeking. Though some would benefit if stuff like exif metadata could be surfaced.
• Those that do end up using mp4box.js, have made mistakes around extra data handling for aac, h264, h265 and how codec strings are specified within the container.
• I wrote a brief explainer on how we might leverage MediaRecorder at least for muxing, but it didn't get any traction (either internally or from other UAs): https://github.com/dalecurtis/mediarecorder-muxer/blob/main/explainer.md
• Similarly there may be some limited API we can conceive around using MediaSource for demuxing. MSE demuxers are less capable than what something like ffmpeg can do though, so this may not be better than mp4box.js