perf(producer): multi-worker mp4 renders start encoding while frames are still being captured by miguel-heygen · Pull Request #5952 · heygen-com/hyperframes · GitHub
Skip to content

perf(producer): multi-worker mp4 renders start encoding while frames are still being captured - #5952

Merged
miguel-heygen merged 3 commits into
mainfrom
cloudrender/encode-while-capturing
Oct 21, 2026
Merged

miguel-heygen merged 3 commits into
mainfrom
cloudrender/encode-while-capturing

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What

On a multi-worker mp4 render that captures frames to disk, the encoder now starts with capture instead of after it. It is fed frame 0, 1, 2... strictly in order as each frame file lands, with the same encoder arguments as the after-capture encode, so the output is byte-identical and less encode is left once capture ends.

Why

On this path capture and encode ran one after the other: the encoder only started once every worker had finished, so a render's wall clock was roughly capture plus encode. The streaming path already overlaps them, but the multi-worker streaming mode stays off by default for screenshot capture, so a large share of multi-worker renders took the serial path.

Related work

Refs the default-off multi-worker streaming path (HF_CAPTURE_PARALLEL_STREAM), which this does not change.

How

  • The parallel coordinator reports, per captured frame, the file it just finished writing (ParallelProgress.landedFrame). The write is synchronous and completes before the report, so a reported frame is never partial.
  • encodeWhileCapturing (encode stage) spawns the existing streaming encoder at capture start and feeds frames in index order: it waits for frame N to be reported, reads it from the worker's dir, or from the merged frames dir if the post-batch merge has already renamed it (rename is atomic, so the frame is in one place or the other). Once capture succeeds it reads any remaining frames from the merged dir and closes the encoder.
  • The overlapped run uses the same options builder as the after-capture encode (videoEncoderOptions, now shared). For software libx264 from JPEG frames, image2pipe and the image-sequence input decode the same JPEG bytes into the same frames, so the bitstream is identical.
  • Safety:
    • A capture failure (worker death, missing frames, cancel) goes through the existing catch, which aborts the encoder and waits for it to exit before rethrowing, so it cannot hang.
    • Adaptive retries keep the frames they already have, so the in-order reader just waits for the retried frames.
    • It is off under drawElement capture, where a failed self-check wipes every frame and recaptures.
    • It runs only when a core is spare for the encoder: the render must use fewer capture workers than the CPU count worker sizing sees (the same cpuCount computeWorkerSizing reports). With every core capturing, the encoder only slows capture down, so the render keeps today's serial encode.
    • If the overlapped encode fails for any reason, the frames are encoded after capture exactly as before, with a warning, so a failure costs time but never the render.
  • Scope: software h264 mp4, SDR, JPEG frames, no chunked encode, no GPU encode. PNG sequence, GIF, WebM/MOV, HDR, layered composite, chunked encode and HLS keep today's behaviour.
  • Frame assignment stays contiguous per worker, so the in-order reader keeps pace with worker 0 and overlaps about 1/N of the frames on long films (more on short ones, where workers finish unevenly). I tried round-robin assignment: it overlaps almost all of the encode but changes the captured pixels (the output sha differs from main's, consistently across runs), so it is out of scope here.

Test plan

  • Unit tests added/updated: encodeWhileCapturing.test.ts (integration lane, real ffmpeg, preset medium crf 16). Frames land out of order across two workers' dirs and must be encoded in index order, byte-identical to runEncodeStage on the same frames; frames already moved by the merge, or never reported, are still read; abort stops an encoder that is waiting on frames. Each test failed with the change broken: with the sources at main, encodeWhileCapturing is not a function; when feeding only after capture, the overlap test times out (condition not reached); with one encoder argument changed (crf 17), expected '29745d05…' to be '135a554a…'. A gate test (4 workers with 8 cores runs the overlap; 4 workers with 4 cores and 1 worker do not) fails with expected true to be false once the spare-core check is removed. It passed 3 runs in a row.
  • Producer unit suite passed 3 times in a row with 0 failures (Linux), plus engine and producer typecheck, oxlint, oxfmt and fallow.
  • Manual testing performed: rendered four fixtures on a shared 4-CPU Linux box, forcing the screenshot capture, 4-worker disk path, main and this branch alternating, 3 runs each, each run starting at load 16 or below:
film main wall (median) branch wall (median) encode left after capture, main / branch frames encoded during capture
style-1-prod, 17 s 1080p, video + audio 70.7 s 62.9 s 11.2 / 3.5 s ~60%
42 s portrait, video 124.2 s 104.1 s 23.2 / 13.1 s ~37%
142 s portrait, video 427.1 s 402.3 s 89.1 / 61.7 s ~26%

With the spare-core gate:

  • 8 s 1080p film with no core spare (8 workers, the CPU count worker sizing sees): this branch takes today's serial path (no overlap logged) and produces the same sha. Main 17.6 / 17.7 / 20.9 s (median 17.7), branch 17.0 / 18.7 / 22.9 s (median 18.7). It is the same code path; the gap is run-to-run noise, since the box's load rose through the set. Before the gate, at 4 workers with a 4-CPU budget, this film was 1.3 s slower at the median (18.1 to 19.4 s): the encoder competed with capture because no core was spare.
  • 17 s film at 2 workers (cores spare): main 54.1 / 46.7 / 44.1 s (median 46.7), branch 50.8 / 39.9 / 41.0 s (median 41.0).
  • Earlier ungated runs at 2 workers: 8 s film 27.9 s to 23.4 s.

The CPU count worker sizing reads is the machine's core count. It does not see a container CPU quota: the Linux box above caps each job at 4 CPUs through a cgroup quota, and sizing still sees 8. So on a quota-limited container the gate uses the same core count sizing does.

Byte identity: the 8 s and 142 s films produce the same output sha on main and on this branch in every run. The two other video fixtures are not byte-stable run to run even on main (capture varies), so for all four I also ran today's encode on the very frames each overlapped render used. The video-only output matched in every case (8 s 1af06eca, 17 s f7142e7b, 42 s 60d81447, 142 s 7afb62fb), and the final mp4 shas for the 17 s film on this branch fall in the same set main produces.

Not run: macOS and Windows renders (no reachable machine), HLS, and the drawElement disk path (excluded by design).

  • Documentation updated (if applicable)
  • Comments follow CONTRIBUTING.md "Comments"

…are still being captured

On the multi-worker disk capture path the encoder used to start only after
every worker had finished. It now starts with capture and is fed frame 0, 1,
2... in order as each frame file lands, with the same encoder arguments, so
the output is byte-identical and the encode left after capture is shorter.
Off for drawElement capture (a failed self-check discards every frame) and
for png/gif/webm/mov/HDR/GPU/chunked/HLS output. If the overlapped encode
fails, the frames are encoded after capture exactly as before.
…he encoder

With every core busy capturing, the encoder only slowed capture down, so the
overlap now runs only when the render uses fewer workers than the CPU count
worker sizing sees. Also pins the reviewed producer test-lane source hash and
drops an added comment line.
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact a6db722f33e85e2dc17eb119aee215d7add25071. No blocking finding.

packages/producer/src/services/render/stages/encodeStage.ts:320–426 consumes landed frames in index order, falls back from an atomically renamed worker path to the merged path, and serially re-encodes after overlap failure. The orchestrator aborts and awaits the overlap encoder on capture failure. Shared videoEncoderOptions preserves encoder settings, and the eligibility gate excludes drawElement recapture, GPU/HDR, chunk/HLS and incompatible formats. Adaptive retries retain the frames already reported rather than invalidating fed frames.

The fresh adversarial pass ran the original four real-ffmpeg integration tests plus three additional controls: corrupted landed input recovers through the serial path with matching SHA, fractional 30000/1001 FPS retains byte identity, and documented exclusions stay excluded. Seven controls pass. Local testing used exact-source export aliases/narrow core and engine facades because generated runtime dependencies were unavailable; encoder, process and filesystem implementations were not mocked. These are focused encode tests, not a complete browser render or cross-platform acceptance. The author’s broader render timing measurements remain author-reported.

Audited: new overlap owner, eligibility/options extraction, changed coordinator/frame reporting, capture/adaptive-retry/merge and orchestrator finish/abort integration, test registration and tests. Trusting: unchanged large orchestrator branches outside this path and browser capture internals. No existing substantive review findings to duplicate.

Need: yes · Size: coherent capture/encode overlap · Reuse: existing streaming encoder and shared disk encoder options · Simpler: sequential indexed feeding with serial fallback; no new capture scheduler.

— hf-review

Verdict: APPROVE
Reasoning: Frame ordering, merge races, capture failure cleanup and serial fallback hold in source review and focused real-encoder controls.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit eec3c51 Oct 21, 2026
74 checks passed
@miguel-heygen
miguel-heygen deleted the cloudrender/encode-while-capturing branch October 21, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants