{{ message }}
Repository navigation
VideoFrame.copyTo is manipulating JS objects in a background queue #955
Description
As per https://w3c.github.io/webcodecs/#dom-videoframe-copyto, destination which is a JS visible object is written from a background queue task, which is an anti-pattern.
This creates problems (JS observability, destination mutating during the copyTo tasks).
There are various ways to fix this, for instance:
- make
copyToalgorithm synchronous - Expose a new asynchronous API that would transfer the destination buffer and provide the returned value as part of the promise value.
- Extend
copyToAPI surface to implement 2 within it. - Change the spec to write
destinationasynchronously but in an event loop task with additional checks, even if UA may do it in a smarter way.
As a side note, copyTo algorithm is also synchronously converting to RGB.
It would be convenient to do this asynchronously, which is feasible by cloning the VideoFrame synchronously and using the cloned VideoFrame in the remainder of the algorithm.
Activity
dalecurtis commentedon Oct 1, 2026
Are any of those actually problems? On the Chrome side we did change to using atomic writes for the copy, so we're not exposing undefined behavior. All of the solutions proposed would hurt WASM interop.
Djuffin commentedon Oct 1, 2026
From js observability point of view, It's no different from another worker writing to the same shared buffer. copyTo to non-shared buffers shouldn't have this effect (i.e. it should be synchronous).
youennf commentedon Oct 2, 2026
A naive implementation of the spec would lead to obvious memory issues.
Also, the current spec could quickly lead to interop issues (web page relying on copyTo being sync and UAs having different implementations).
How about a Promise-based
prefetch()method that would ensure that copy is guaranteed to be synchronous?Then, the spec should describe this algorithm.
FWIW, it is surprising to me that a web page gets widely different (and JS observable) behaviour depending on the input buffer flavour.
dalecurtis commentedon Oct 5, 2026
Can you elaborate on the obvious issues? Do you just mean a client who reads w/o waiting for the promise may get partial/corrupted data?
This would result in two copies IIUC?
+1, I didn't realize the spec text didn't already have this clarification. It should be clarified to MUST be synchronous for non-shared type buffers and MAY be asynchronous for SAB.
youennf commentedon Oct 6, 2026
For instance, UA is writing on a buffer in a background queue while web page is resizing it/transferring it on another.
At least in WebKit, I am not sure how we can avoid two copies for GPU-backed video frames.
As part of the copyTo current approach, we do something like:
prefetchwould trigger the first copy,copyTothe second.Can Chromium avoid these two copies?
If we do this, I would make the algorithm first check whether the buffer is SAB or not.
Background queue would only be used for SAB.
I am also not exactly sure how we are supposed to specify the SAB algorithm.
I would tend to synchronously do a structure clone of the SAB and use the clone in the background queue.
But maybe we can hook to some SAB internals instead.
dalecurtis commentedon Oct 6, 2026
Ah you mean for the non-SAB cases? Yes, those do have obvious issues. SAB can't be shrunk and can't be transferred.
Does WebKit not pass the IOSurface to the content process? We don't always in Chromium for legacy, non-security reasons. In Chromium some frames are mappable so have 1-copy perf and some require read-back which has 2-copy perf as you note.
This seems reasonable; I haven't looked at how other specs handle SAB copies though. In Chromium we just ref the underlying buffer.
youennf commentedon Oct 6, 2026
I see. AIUI,
prefetchwould not trigger any additional cost to either strategies.And it would allow applications to call copyTo with a consistent behaviour (no blocking IPC call for some frames, no SAB async behaviour).
dalecurtis commentedon Oct 6, 2026
Can you explain
prefetchmore precisely? I feel like I'm missing something:memcpyon the main thread even if the readback is async.In any case,
copyTois already shipped by Firefox and Chrome, so at most we want to clarify the spec or make behind the scenes adjustments for better interop. Without a strong reason, adding new methods, attributes, etc seems out of scope.@Djuffin can you take a stab at a PR which clarifies the algorithms here?
Clarify VideoFrame.copyTo() parallel steps and buffer handling