Activity

1. youennf

   added

   TPAC2026Agenda topic for TPAC 2026Agenda topic for TPAC 2026

   on Oct 9, 2026
2. podborski commented on Oct 13, 2026

   

   podborski

   on Oct 13, 2026

   Member

   More actions

   Thanks for filing this, @youennf. I tested this gap against some samples.

   The problem: VideoDecoder only ever decodes one implicit layer: e.g. AV1 operating point 0, or nuh_layer_id=0 for HEVC. There seems to be no API to discover or select any other layer or operating point.

   Why it matters: layering in a single track is already very common and growing: MV-HEVC (layered alpha, stereo), AV1 (operating points today, with AOM discussing extending this to alpha/depth/stereo), VVC, and AV2 all have multi-layer support. Any fix should be codec-agnostic, not scoped to one codec IMO.
   After investigating some sources it seems that some implementations hardcode operating points (e.g. operating_point = 0 and reject decoding if spatial-layer bit mask > 1) or don’t even consider layer IDs at all.

   Strawman fix, codec-agnostic:

   1. Discover: introspect declared layers/operating points from the existing out-of-band config, no decoder instance needed.
   2. Select: a VideoDecoderConfig field (e.g. targetLayer or operatingPoint), defaulting to 0 so existing code is unaffected.
   3. Decode: unchanged, except non-selected-layer chunks get defined behavior instead of today’s silent/inconsistent failures.
   4. Associate: this one needs more thought. decode() can already emit multiple outputs per call, but each output’s timestamp is inherited from the single input chunk. If an app selects multiple layers at once (e.g. texture + alpha, which share a timestamp in the container today), the decoder would emit multiple VideoFrames with identical timestamps and no way to tell them apart. Some per-output layer/role tag would be needed before multi-layer output is actually usable, not just multi-layer input selection.
3. danielcamposramos commented on Oct 21, 2026

   

   danielcamposramos

   on Oct 21, 2026

   More actions

   Hello Youenn and everyone,

   The question about associating decoded outputs connects directly to our work preserving stereo information across the media pipeline. I build Sparky Stereo OS and co-chair PM-KR; I am contributing here in an individual capacity.

   This work already includes upstream merges in HandBrake for encoding the frame-packing declaration, MKVToolNix for deriving container StereoMode from AVC signalling, mpv for consuming stereo side data, and PeerTube for preserving declared packing through transcoding. These are related implementation results, not an implementation of the multiview WebCodecs API proposed here.

   This matters beyond headset applications. At the visual core, binocular VR is stereo with a tracked viewpoint. Head tracking changes the viewpoint; a stereo image or video can also convey depth on a compatible fixed screen without tracking. Keeping that distinction in the data model lets the same content reach headsets, older or newer stereo-capable projectors, classroom screens and suitably equipped cinemas used for education and scientific presentations alongside films.

   Knowledge becomes useful when people can reach it and build with it. In Malawi, William Kamkwamba drew on a village library science book and spare parts to build an electricity-producing windmill after his family could no longer afford school fees (his story). That is why access to knowledge and usable equipment belong together. I want our open work to give people everywhere something they can use today and build on tomorrow.

   A school should be able to use a working 3D projector it already owns without buying a modern headset for every pupil. We would not deny education to every school that lacks Harvard’s resources; similarly, a useful entry point into spatial material should remain possible with more modest equipment. The path should also retain 2D and accessible alternatives. W3C’s web-for-everyone mission, its Ethical Web Principles, and the TAG’s user-first design guidance support considering these users; WAI explicitly includes economic and hardware access in inclusion. For WebCodecs, the consequence is a display-independent representation of the decoded views, with presentation left to the consumer.

   I agree with keeping the API codec-independent. I would distinguish two input cases while preserving common view semantics at the output:

   • Packed stereo: one decoded picture contains both eyes. The frame-packing SEI in H.264 and H.265, Matroska StereoMode, or MP4 st3d can describe its interpretation. A conventional single-picture decoder can produce the pixels; consumers still need layout, eye order and per-eye geometry.
   • Multiview coding: obtaining both eyes requires the relevant decoder support and view selection. Preserving a layout label cannot substitute for decoding the additional view. Codec layer identifiers also need their declared meaning; a numeric layer ID alone does not say “left eye” or “right eye”.

   Building on the association problem raised above, I suggest these requirements for evaluating either a frame-group design or separately delivered VideoFrames:

   1. Identify each output’s view or role and its association with the other outputs for the same presentation instant. Equal timestamps alone do not identify the eyes.
   2. Define what consumers receive when a requested view is unavailable or dropped, so they cannot silently pair a fresh left view with a stale right one.
   3. Preserve the relevant geometry and metadata through the operations the API supports, including clone/transfer and any defined crop or resize behavior. Metadata supplied by a container demuxer also needs a defined handoff; a decoder cannot recover a container declaration it never receives.
   4. Keep explicit mono, unknown interpretation and an unavailable requested stereo view distinguishable.

   The current VideoFrame interface and VideoFrameMetadata registry are relevant places to discuss the descriptive part. A metadata member by itself would not settle frame grouping, lifetime or missing-view behavior. WebXR Layers’ XRLayerLayout provides a downstream consumer vocabulary to map to, rather than a reason to discard source view identity.

   Our browser/compositor prototype carries both eyes together as full side by side at that boundary. That demonstrates one presentation path, while leaving WebCodecs free to expose separate frames or another representation. The underlying requirement is that a consumer can recover the correct views, geometry and timing without guessing from frame shape or callback order.

   Could we capture those association requirements before choosing between a special VideoFrame and a group of frames?

   Prepared with AI partners; I review the work and take responsibility for this contribution.
4. dalecurtis commented on Oct 22, 2026

   

   dalecurtis

   on Oct 22, 2026

   Contributor

   More actions

   @youennf can you detail how MV-HEVC works in VideoToolbox today? Largely WebCodecs is just a reflection of the OS surfaces available to us, so we wouldn't want to diverge dramatically from how the OS works.

   👍React with 👍1Reacted by Daniel Ramos
5. danielcamposramos commented on Oct 23, 2026

   

   danielcamposramos

   on Oct 23, 2026

   More actions

   A short update since my comment of 21 October, on Dale's question about the OS surface and on the association problem Dimitri raised.

   On the association side there is now a registered vocabulary for the depth case: on 20 October 2026 IANA assigned Matroska Track Plane Types 3, 4 and 5 to the depth maps of the left-eye, right-eye and background planes (https://www.iana.org/assignments/matroska#matroska-track-plane-types). A container can therefore say which plane a track is, not only that two tracks belong together through TrackCombinePlanes. Whatever shape WebCodecs chooses for multi-layer output, a per-output role tag with values of that kind (left view, right view, depth of the left view, and so on) would let a demuxer's declaration survive the handoff that my point 3 asked for. The six packed-frame StereoMode values I proposed are still under expert review and are not assigned, so I do not rely on them here.

   For the OS side, one public pointer while Youenn answers: Apple's own document (QuickTime and ISO Base Media File Formats and Spatial and Immersive Media, 1.9.8 Beta, 9 June 2025) defines vexu as signalling that a track "carries two stereo eyes or is for only one of the two stereo eyes", with an optional hero eye as the default view. That is the same distinction as Dimitri's layer selection: the container names the eyes, the codec layers carry them, and the API needs both to agree on which is which.

   Since 21 October, x265 merged the --frame-packing option (Multicorewareinc/x265#986, 22 October), so an HEVC encoder on the open side now writes the frame packing arrangement SEI the way x264 does; the merges record is at https://github.com/Sparky-OS/sparky-stereo-os/blob/main/docs/upstream-merges.md.

   Offer: test files for each case (a packed HEVC stream with the SEI, and a Matroska file with StereoMode and with the new depth-plane types), and our open implementation as a reference consumer that recovers views, geometry and timing without guessing from frame shape or callback order. Say which case you want first.
6. youennf commented on Oct 23, 2026

   

   youennf

   on Oct 23, 2026

   ContributorAuthor

   More actions

   can you detail how MV-HEVC works in VideoToolbox today?

   @dalecurtis, there is some documentation at https://developer.apple.com/documentation/videotoolbox/kvtdecompressionpropertykey_requestedmvhevcvideolayerids.
   VideoToolbox decoders generate CMTaggedBufferGroupRef which is a group of CVPixelBuffers, when MV-HEVC is enabled.

   In WebCodecs terms, if decoder is configured for MVC, the decoder would output a set of VideoFrames with extra metadata to let the application know how they relate with each other (stereoscopic left/right for instance).