feat(cli): cloud render uploads heavy media as HeyGen assets by miguel-heygen · Pull Request #5931 · heygen-com/hyperframes · GitHub
Skip to content

feat(cli): cloud render uploads heavy media as HeyGen assets - #5931

Merged
miguel-heygen merged 11 commits into
mainfrom
feat/cloud-render-asset-offload
Oct 21, 2026
Merged

miguel-heygen merged 11 commits into
mainfrom
feat/cloud-render-asset-offload

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What changes for a user

hyperframes cloud render now renders projects whose media is far larger than the 200 MB project zip limit, with no manual step. Before zipping, media of 5 MB or more that the project names by path uploads as HeyGen assets, and the zip's copy of each HTML, CSS, script and JSON file points at the asset URL. The project on disk is never changed.

  • Cached uploads. Uploads are cached by content hash (~/.cache/hyperframes/cloud-assets.json, keyed by API host, encode settings and cut). A re-render uploads only new or changed media. A cached URL is checked with a HEAD request before reuse and uploaded again if it no longer serves the same bytes.

  • Only the played parts of long media. A video or audio file that has to be re-encoded is first cut to the windows the entry composition plays: 0.5 s of margin, snapped to the source's frame grid, with separate windows joined. The staged composition's media starts shift to match, so each clip shows the same frames. Clips under a composition host or in-point, looping clips, speed ramps, variable-frame-rate video and files a script names keep their full length.

  • Re-encoding what the asset API refuses or caps, at visually lossless quality. All settings live in one table (REENCODE_PRESETS):

    • Video: H.264 at CRF 18. HDR sources become 10-bit HEVC with their colour tags kept; sources with alpha become VP9 WebM with alpha.
    • Audio: 16-bit WAV.
    • Still AVIF, GIF, BMP and TIFF images: PNG.
  • Animated GIFs of 5 MB or more in the entry composition, resolved the way the renderer rebases a nested entry, go through the renderer's own GIF preparation (prepareAnimatedGifInputs, now exported from the producer). The resulting seek-synced video uploads like any other. GIFs inside sub-compositions stay in the zip, because the renderer works out their clip window only after inlining.

  • Rewritten references:

    • src, poster, href and srcset attributes.
    • url() in <style>, style="" and .css files.
    • Plain string literals in inline scripts, .js and .json files, such as new Audio("x.mp3"), fetch('x') and el.src = "x".

    A file a script reaches by building its path keeps a copy in the zip.

  • Zip under 200 MB, or a clear stop. The zip must end under 200 MB. If it does not, the command stops before uploading it and names the largest files left. A played part that is still over 200 MiB after the visually lossless encode stops with a message naming the file.

  • Dry run. --dry-run --json reports offloaded_media (path, size, upload / reencode / cached, and trimmed_to_seconds) and the zip size after offloading. It does not authenticate or upload.

How it is built (reuse)

  • Upload: the existing direct-upload helper (uploadAssetViaDirectUpload) now also returns the asset URL that /complete sends back.
  • Staging: offloading runs on the in-memory file map from buildPublishFileMap, at the same point publish's proxy baking uses.
  • Rewriting: reuses rewriteHtmlAttributes, rewriteCssUrls and rewriteStyleBlocks, which now take a resolver and accept an absolute URL as a target. Paths resolve through rewriteAssetPath and resolveLocalAssetCandidates, checked against the staged map.
  • Timing: windows come from the parser's own readers (readMediaOffsetSeconds, readAuthoredDurationSeconds, readPlaybackRate).
  • Small fixes along the way:
    • getUploadHeaders folds three copies of the "set this header unless the server sent it" check into one helper, with a new test for mixed-case server headers.
    • Three .fallowrc.jsonc globs regain the nested files they matched before fallow stopped letting * cross /.

Verified against the real cloud

Remote asset URLs render frame-accurately. I uploaded a 6 s probe through the asset API: a video trimmed mid-GOP, a WAV with beeps, and a PNG used as both <img> and CSS url(). Its cloud render from asset URLs was bit-identical on 180/180 frames to the same render with the media inside the zip. Both matched a local render at 52.7 dB or better on every frame, and the beeps landed at exactly 1.500 s and 3.500 s.

Asset URLs outlast a render. They are public, unsigned links (https://resource2.heygen.ai/<kind>/<id>/original.<ext>): no query string, anonymous GET 200, byte ranges, CORS *. Nothing in the URL expires. The project zip's asset URL has the same shape.

Measured asset API limits. These are the source of the accept/re-encode table:

  • Accepted up to 200 MiB per file: MP4, WebM, MOV, PNG, JPEG, WebP.
  • Accepted up to 1.5 GB per file: WAV, MP3.
  • Refused: MKV, M4A, AAC, OGG, GIF, SVG, AVIF, fonts, GLB.

End to end with a generated 2.5 GB fixture. The fixture holds:

  • A 1.7 GB, 60 s ProRes file played by two clips 35 s apart.
  • A 300 MB MKV with its own audio, and a FLAC.
  • MP4, alpha WebM, WAV and PNG files that go up as-is.
  • A looping GIF with data-loop and a 5 s clip window.
  • A small logo that stays in the zip.

I made one local render and two cloud renders:

Check Result
First cloud run All 8 heavy files uploaded (3 cut and re-encoded); zip 4.3 KB; 132 s wall clock, of which about 60 s was the render
Second cloud run All 8 heavy files reported already uploaded; nothing uploaded
Cloud vs local, all 300 frames min 42.4 dB, mean 43.0 dB. Two cloud runs of the same inputs differ from each other by about the same amount (43.2 dB), so this is the renderer's own run-to-run variation on this noisy content
Frame alignment per clip (cloud frame n vs local frame n+k) Best at k = 0 for every clip: ProRes cut 1 46.5 dB vs 26.7 dB one frame off; ProRes cut 2 46.7 dB vs 26.2 dB; MKV cut 50.9 vs 48.2 dB; MP4 49.6 vs 42.7 dB; GIF window 44.0 vs 33.1 dB
Audio, cloud vs local Lag 0.02 ms, correlation 0.95
Project on disk Unchanged (sha256 of every file before and after)

The first full run also caught a bug before this head: the reference rewrite decoded every staged file as text, and a 1.7 GB file exceeds the longest string V8 allows. Now only HTML, CSS, JS and JSON are decoded, with a regression test.

Tests. 12 new offload tests, 2 new render tests and 1 new publish test.

  • Each piece was proven by breaking it and watching a test fail: rewrite, script literals, image conversion, animated images staying zipped, GIF prep, dropping the prepared GIF, trim plan, trim shift, loop and script mentions blocking a trim, text-only decoding, cache hit, stale cache, re-encode, re-encode still over cap, kept zip copy, asset URL name collision, 200 MB guard, dry-run report, offloaded file left in the zip, and header case. After review: variable-bound clips not cut, only referenced files offloaded, sibling-first GIF source, encoding the staged snapshot, small GIFs skipped, null cache file, query and fragment kept, fragment clips not cut, cut and frame-grid snap in the cache key, timed-ancestor, nested-composition, data-automation and other-mention exclusions, and the HEAD size check.
  • Three green runs in a row.
  • tsc for the CLI, core, studio and scripts is clean, and the fallow audit is clean.

Commits skipped only the local typecheck hook: that checkout had no generated core runtime file. The identical typecheck commands ran clean on a full build of this branch.

Big files never read whole (head 7855bbd)

  • Media stays on disk until used. Non-text files of 5 MB or more are staged as a path plus size and modification time, not read into memory. Hashing streams the file. An as-is upload pipes it to the presigned URL through node:http(s) with an exact Content-Length: Node's fetch holds a streamed request body in memory whole (measured: 1 GiB upload peaked at about 1.1 GB through fetch and 126 MB through node:http). A re-encode reads the file in place. Text files (HTML, CSS, JS, JSON, SVG, XML, TXT) are still read, because the rewrite edits them.
  • Same snapshot, different mechanism. For media staged on disk, the cache key and the encode are bound by checking the file is unchanged after hashing, after the encode and after the upload. An edit during the run stops it with "changed on disk", and nothing is cached. Media small enough to stage in memory keeps the temp-copy snapshot from the earlier head.
  • Leftovers sized before any read. If the files still on disk after offloading are over 200 MB, nothing is read: --dry-run reports the size, and a real run stops with the existing "Project too large" message naming them. On main and on the previous head, a 4 GiB file in a project crashed cloud render --dry-run with ENOMEM.
  • The "too large" hint now shows. The asset API answers an over-limit upload with invalid_parameter and "File too large. Maximum is 200 MiB.", so the dry-run / .hyperframesignore hint never fired. That message now maps to it.
  • publish is unchanged: it still builds its archive in memory.

New tests (each goes red when its code is broken):

  • render: a dry run with an unreferenced file of 4 GiB + 1 byte. It fails on main and on the previous head (ENOMEM).
  • offload: 4 GiB + 1 media hashed and encoded from disk; on-disk media uploaded as a file stream; an edit during the encode stops the run with nothing cached.
  • upload: a real local server receives the exact bytes with Content-Length, and a refused PUT reports its status.
  • errors: the server's "File too large" gets the dry-run hint.

CLI tests touching this surface: 7 files, 139 passed, three runs in a row. tsc, oxlint, oxfmt, the comment check and fallow are clean.

Decisions for review

Decided by Miguel:

  • Upload cache is per machine, not per account. It stays keyed by API host, content, encode settings and cut. Two accounts on one machine that render the same file share one uploaded copy, and it is uploaded again if that copy stops serving.
  • A played part still over 200 MiB after re-encoding stops the render and names the file. No automatic bitrate fitting.
  • Per-file uploads are the path for multi-GB projects. Files over the asset API's per-file cap wait on larger uploads in HeyGen's asset API; this PR is the client side of that.
  • One PR: this is one feature already mid-review, so it stays one PR; splitting it now would restart every review.

Open for review:

  • Threshold: 5 MB (OFFLOAD_MIN_BYTES). Any media at or above it that a reference names uploads, even when the zip would fit. A file no plain string names (picked through --variables, or a path a script builds) stays in the zip.
  • Re-encode quality: "visually lossless" as set above. It is a one-line change in REENCODE_PRESETS.
  • Public URLs: offloaded media is reachable by anyone holding its unguessable URL, the same as the uploaded project zip today.

Media of 5 MB or more leaves the project zip: it uploads through the
direct-upload flow as a HeyGen asset and the zip's HTML and CSS point at
the asset URL. Uploads are cached by content hash and re-checked before
reuse. Media the asset API refuses or caps is re-encoded at visually
lossless quality first. The zip must end under 200 MB or the command
stops and names the largest files left.
@mintlify

mintlify Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
hyperframes 🟢 Ready View Preview Oct 20, 2026, 6:50 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 159 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (2)

  • seqheldtwice-tlhold-px-r0-root-z100: tracking 120.29, pressJump 312.3, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport false / tracking 0.14, pressJump 0.11, drop 0.07, reload 0.07, render 0.02, renderKey -, undo true, teleport true / tracking 0.13, pressJump 0.11, drop 0.07, reload 0.07, render 0.02, renderKey -, undo true, teleport true
  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

…nced media

Animated GIFs in the entry composition go through the renderer's own GIF
preparation and upload as the resulting video. Still AVIF, GIF, BMP and
TIFF images convert to PNG. Plain string literals in scripts and JSON that
name offloaded media point at the asset URL. ffprobe calls end their
options with "--", and two comment lines are trimmed for the ratchet.
…ition plays

A video or audio file that has to be re-encoded is cut to the union of the
windows its clips play, with half a second of margin on the source's frame
grid, and the staged composition's media starts shift to match. Clips in
sub-compositions, looping clips, speed ramps and files a script names keep
the full file.

@terencecho terencecho left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source-only review of 1158d6853c34929d0003395354b13d877577a2c3 against effective base ae02e5aceff4742e5d72c20f0859eb3e6b449035. Three introduced correctness failures need changes before approval. The witnesses below are source-derived traces, not executed reproductions.

Strengths

  • packages/cli/src/cloud/offload.ts:710–720 limits text decoding to rewriteable text files, preserving large binary Buffers instead of turning them into V8 strings.
  • packages/cli/src/commands/cloud/render.ts:605–613 measures the actual compressed archive after offloading and checks it before the project ZIP upload; packages/cli/src/cloud/offload.ts:624–635 processes media sequentially and cleans its temporary directories in finally.

Blockers — P2 correctness

  1. Keep the ZIP copy when a dynamic reference cannot be rewritten. packages/cli/src/cloud/offload.ts:649–673 deletes an uploaded file unless a text file still contains its entire basename. Consider a valid ≥5 MiB media/clip.mp4 used by a timed video whose script sets v.src = "media/" + "clip" + ".mp4", with no contiguous clip.mp4 elsewhere in project text. rewriteStringLiterals (:766–778) cannot replace those fragments with the uploaded URL, and isStillMentioned finds no whole basename, so the ZIP loses the file while the script still requests its local path. BASE kept that staged file; the existing producer explicitly collects media sources assigned after JavaScript (packages/producer/src/services/htmlCompiler.ts:2268–2321). This contradicts the promised dynamically constructed-path fallback. The current dynamic-reference test retains a full filename in a comment, so it does not cover split-literal references without a contiguous basename. Retain the ZIP copy when dynamic references cannot be resolved/rewritten; basename absence is not sufficient deletion evidence.

  2. Resolve entry GIFs against the selected entry before preparing them. packages/cli/src/cloud/offload.ts:585–591 passes unrebased entry HTML to prepareAnimatedGifInputs with the project root as projectDir. With --composition compositions/scene.html, <img src="logo.gif">, and different animated GIFs at both logo.gif and compositions/logo.gif, the helper resolves and prepares the root GIF, then replaces the nested entry's image with that wrong animation. outputSrcPrefix only fixes the generated output URL; it does not resolve the input. The unchanged render compiler first rebases the selected entry (packages/producer/src/services/htmlCompiler.ts:1954–1996), using the sibling-first resolver in packages/parsers/src/rewriteSubCompPaths.ts:72–86, before GIF preparation. Preserve that input-resolution contract here, for example through a correctly resolved source mapping.

  3. Encode the same snapshot that supplies the cache key. packages/cli/src/cloud/offload.ts:444–452 returns the live project pathname whenever it exists, but :483–495 keys the result by the staged Buffer. A concrete interleaving: staging captures A bytes for a ≥5 MiB b.m4a; while an earlier media upload is awaited, an editor saves different valid B bytes to b.m4a. For an audio element without an authored bounded duration, the preset remains audio and the trim plan remains null. FFmpeg then reads B, and :555–570 persists B's URL under A's hash/settings/null-trim key. Restoring A and rendering again hits that same key; the HEAD/content-length check validates the stored B asset, not equality with A. This silently persists the wrong media beyond the concurrent save. BASE zipped the staged Buffers consistently. Materialize the staged bytes for encoding, or otherwise bind the encoded input and cache identity to one immutable snapshot.

Audited: two independent code passes read all 11 existing changed BASE files before all 14 changed HEAD files, including tests as source. Parent reconciled their delivered numbered read ranges against hash-verified blobs with no changed-body gaps, verified the tree/path union, and independently traced the three failures through staging, rewriting, GIF resolution and cache read/write boundaries. Supporting compiler inspection was targeted, not whole-file coverage.

Trusting / limits: no tests, builds, installs, encoders, browsers, uploads, renders or CI validation run. The reported real-cloud run, frame/audio equivalence, deployed upload caps and CDN lifetime/access behavior remain author-provided evidence, not independent runtime proof. No claim of bounded whole-project heap use or complete transitive dependency coverage. GIF preparation running without an upload callback is explicitly admitted by the function comment; I am not treating that behavior alone as an unintended defect.

Review effects: none of the 16 inspected checked-in workflows has a review-submission trigger. External merge-loop/queue/app policy remains unverified. A later merge/main push can run checks and catalog publication; the latter can update generated branches/PRs and dispatch check workflows. At the reviewed feature-branch ref, the inspected npm-release entry condition is not satisfied; that does not establish post-merge or external-app behavior. No explicit merge, workflow dispatch, build, package publication or deployment operation is being invoked by this review; downstream effects of review submission are not independently established.

— tai

Verdict: REQUEST CHANGES
Reasoning: Dynamic-path media can disappear, nested entries can render the wrong GIF, and concurrent saves can cache B’s uploaded URL under A’s content-derived key. These are source-level correctness findings, not a hold for CI or missing runtime validation.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 1158d685. Requesting changes for two --variables regressions. The trim itself is right: with real ffmpeg, every played frame and audio sample lands where it should. The problem is which clips and files it applies to. cloud render accepts --variables (render.ts:104-133), and both issues below break that path.

Blockers

  1. A variable-bound clip is trimmed and its in-point shifted (offloadTrim.ts:40-49, verified). playedWindow excludes loop, data-automation and timed ancestors, but not data-var-src. Take <video data-var-src="clip" src="a.mp4" data-media-start="10" data-duration="2">. The clip is cut and its start rewritten to 0.5. A render with --variables '{"clip":"b.mp4"}' then plays b.mp4 from 0.5 s, not 10 s: the wrong frames, with no error. Fix: return null from playedWindow for any element with data-var-src. Its source isn't known until render, so neither the cut nor the shift is valid.
  2. A ≥ 5 MB file that no text file names is uploaded and then removed from the zip (offload.ts:626 → dropReplacedFiles, verified). offloadHeavyMedia offloads every heavy file in fileMap. isStillMentioned keeps a zip copy only if a staged text file still names it. A file used only through --variables (say media/b.mp4) is therefore missing at render time. That unused footage also goes to a public asset URL for no reason. Fix: offload only files that a rewritten reference actually points at. Leave everything else in the zip as today.

Should-fix
3. Animated GIFs are transcoded locally at any size, dry run included (offload.ts:578-599). A 2 KB GIF in the entry became a webm in the zip, where before this PR the server did that step. Inferred: without ffmpeg, any project with an animated GIF now fails with "Media upload failed". Limit it to GIFs at or above the 5 MB cutoff, or leave the GIF alone when ffmpeg is missing.
4. The cache key isn't scoped to the account (offload.ts:483). It's apiBaseUrl + sha256 + variant. On a machine that switches HeyGen accounts, account B silently reuses account A's asset URL. That only stays valid while A keeps the asset, and it links the two accounts' renders. Add the account or space id to the key.
5. The tests mock the encoder and miss load-bearing logic. Each of these was removed with the suite still green:

  • the cut segments in the cache key (a stale cut would be reused after the composition changes);
  • the frame-grid snap (with real ffmpeg at 29.97 fps, removing it put 50 of 60 frames off by one);
  • the nested-composition and parent in-point exclusion;
  • the data-automation exclusion;
  • the "other mentions in the entry HTML" check;
  • the HEAD size check on cache hits.

A real-ffmpeg test for the 29.97 snap and a cache-key test would pin the risky parts.
6. Docs. docs/packages/cli.mdx and docs/deploy/cloud.mdx should say that offloaded media sits behind public, unsigned asset URLs embedded in the render's HTML. The zip has always gone by asset_id, so individual files being directly addressable is new.

Nits (verified)

  • JSON keys and labels. The string-literal rewrite also replaces them: {"intro.mp4": …, "label": "intro.mp4"} gets both rewritten (offload.ts:767-779).
  • Cuts past the source end. They aren't clamped to the source length. A clip that starts past the end uploads a 261-byte stream-less mp4 with no error, and next to other clips it holds an unrelated segment's last frame (offload.ts:309-328).
  • Substring match. "Still referenced" is a substring test, so a.mp4 stays in the zip because banana.mp4 is named (offload.ts:661-674).
  • Query strings and fragments. ?query and #fragment are dropped: clip.mp4#t=5,8 becomes the bare URL (offload.ts:724-728).
  • Null cache file. A cache file containing null crashes the run (offload.ts:198-204).

Verified fine

  • Trim accuracy. Frame-exact at 30 and 29.97 fps across 4 clips and 3 segments, including overlapping windows and rate 2. FLAC → WAV is sample-exact at both ends, with A/V sync within 0–3 samples.
  • Rewriting. CSS url() resolves against the stylesheet, srcset descriptors are kept, URL-encoded and unicode names resolve, and remote and data: URLs are left alone.
  • Cache safety. Cache writes are atomic, and corrupt JSON falls back to an empty cache.
  • Dry run and limits. A dry run does no auth, upload, encode or HEAD check (except the GIF step in 3). The 200 MiB per-file stop names the file.
  • Tests. The three PR test files pass 79/79.

— Somu

…the staged bytes

Only media a reference points at is offloaded, so a file picked through
--variables or built by a script stays in the zip. Variable-bound clips and
clips with a media fragment keep their full length. Entry GIFs resolve the way
the renderer rebases a nested entry and are prepared only from 5 MB, and every
encode reads the staged bytes its cache key hashes. Query strings and media
fragments survive the rewrite, a null cache file reads as empty, and the docs
say asset URLs are public and unsigned.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@somanshreddy Thanks, all fixed at 1d8c3d12 except where noted.

  1. Fixed: playedWindow returns null for data-var-src, so a variable-bound clip keeps its whole file and its in-point (test: "keeps the whole file when a use is not a bounded clip…", variable-bound case).
  2. Fixed at the root: a reference pass now decides what offloads, using the same resolver as the rewrite, so only files a reference points at leave the zip; a file picked through --variables stays (test: "keeps a heavy file no reference names in the zip…").
  3. Fixed: entry GIF prep runs only when an entry GIF is 5 MB or more; smaller ones are left to the renderer and need no local ffmpeg (asserted in "leaves small media and remote URLs alone").
  4. Not changed yet: the credential cannot key it (OAuth access tokens rotate, so every refresh would miss the cache), and the account id needs a /v3/users/me call on every render. A reused URL is public and passes a HEAD size check, so it still serves, but it does link the two accounts' renders. Worth the extra call? If yes I'll key on the account id.
  5. Fixed: each item you removed now has a test that fails without it: the cut in the cache key and the 29.97 frame-grid snap ("cuts on the source's own frames at 29.97 fps and re-encodes when the cut changes", grid asserted on the planned cut, no real ffmpeg), the nested-composition, parent in-point, data-automation and other-mention exclusions (added to "keeps the whole file…"), and the HEAD size check ("uploads again when the cached asset URL no longer serves the file").
  6. Fixed: both pages now say asset URLs are public and unsigned.

Nits: null cache file fixed (test added). ?query and #t= fragments are now kept on the asset URL, and a clip with a media fragment is never cut, since #t= names times in the uncut file (tests updated). Not changed: a JSON label that is exactly a project file path (indistinguishable from a reference), cuts past the source end, and the banana.mp4 substring keep, which only errs toward keeping a zip copy.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@terencecho Thanks, all three fixed at 1d8c3d12.

  1. Fixed at the root: offloading is now decided by a reference pass that uses the same resolver as the rewrite, so a file no plain string names (your split-literal "media/" + "clip" + ".mp4") is never uploaded and stays in the zip. One case remains: a file named statically and also built by a split literal elsewhere loses its zip copy, since a split path cannot be resolved from source.
  2. Fixed: each entry GIF is resolved with the renderer's sibling-first rule (htmlSrcKey, built on rewriteAssetPath) and handed to the GIF prep through its sourceAssets map, so compositions/scene.html's logo.gif prepares compositions/logo.gif (test: "prepares a nested entry's GIF from the file next to it").
  3. Fixed: probing and encoding read a temp copy of the staged bytes, the same bytes the cache key hashes; GIF prep reads staged copies too (test: "re-encodes the staged bytes, not a file saved on disk after staging").

@somanshreddy
somanshreddy dismissed their stale review October 20, 2026 07:37

Both --variables blockers are fixed at 1d8c3d1 (data-var-src clips are not trimmed; only referenced files are offloaded). Re-review posted. — Somu

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 1d8c3d12, which adds one commit on top of 1158d685. Both of my blockers are fixed, so I've dismissed my change request.

  1. Variable-bound clips (fixed). playedWindow now returns null for data-var-src, so such a clip is neither cut nor shifted. The docs list it with the other exceptions.
  2. Unreferenced files (fixed). offloadHeavyMedia now offloads only referencedFiles(...): the same rewriteReferences walk, run as a dry pass that collects the keys it would replace. A file used only through --variables therefore stays in the zip and is never uploaded.

Should-fixes and nits from my review

  • Done:
    • Animated GIFs are transcoded locally only when an entry GIF is at least 5 MB (entryGifSources returns null otherwise, so no local ffmpeg is needed).
    • The docs now say the asset URLs are public and unsigned.
    • A cache file containing null reads as empty.
    • ?query and #fragment are kept on rewritten URLs, and a clip whose src carries a media fragment is not trimmed.
    • Encodes now read a snapshot of the staged bytes, so the cache key and the encoded bytes match.
  • Still open, not blocking: the cache key isn't scoped to the account (apiBaseUrl + sha256 + variant). The test gaps I listed (frame-grid snap, segments in the cache key, the nested and automation exclusions) are partly covered by the new tests. I haven't re-run the mutations.

Not approving yet. terencecho's CHANGES_REQUESTED (at 1158d685) is still open. The Comments check is red at this head, and CI was still running (12 pass, 25 pending). Once those clear and the lane re-pins this head, I'll approve.

— Somu

…rge upload

Media of 5 MB or more stays on disk until it is used: it is hashed and uploaded
as a stream, and a file edited after staging stops the run instead of caching
the wrong upload. Files left in the zip are sized before any is read, so a
4 GB file reports "too large" instead of crashing.

The asset API rejects an over-limit upload as invalid_parameter with "File too
large"; that now gets the dry-run hint instead of the --help one.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

New head 7855bbd: main merged in, plus two fixes for big projects. Details are in the PR body under "Big files never read whole".

@terencecho, your three findings at 1158d685 were answered at 1d8c3d12. This head changes how your third one (one snapshot for the cache key and the encode) is enforced, for media staged on disk:

  • Media of 5 MB or more is no longer read into memory, so a file over 2 GiB no longer crashes the CLI. That also means there is no in-memory copy left to snapshot.
  • Instead the file's size and modification time are recorded at staging. The run checks them after hashing, after the encode and after the upload; any change stops it with "changed on disk", and nothing is cached. Test: "stops without caching when media staged on disk changes during its encode".
  • Media small enough to stage in memory keeps the temp-copy snapshot. The earlier test still covers it.

Could you re-review at this head?

@somanshreddy, the cache key is still not scoped to the account. That needs a product call (an extra /v3/users/me call per render, or accept the link between accounts), and I've raised it with the owner. Everything else from your re-review is unchanged.

Comment thread packages/cli/src/cloud/offload.ts Fixed
Comment thread packages/cli/src/utils/publishProject.ts Fixed
…handle

Staging, the on-disk staged read and the re-encoded output read each opened the
path twice: once to stat, once to read. A file swapped or edited between the two
could be read without the size or change check seeing it. Each now opens once,
checks the open file's stat before and after the read, and reads from it.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@terencecho New head 0b06284. Where each of your three findings at 1158d685 stands:

  1. Dynamic path loses its zip copy. Fixed at 1d8c3d12. A reference pass, using the same resolver as the rewrite, now decides what offloads. A file that no plain string names is never uploaded and stays in the zip. Your exact case (v.src = "media/" + "clip" + ".mp4", no contiguous basename anywhere) now has its own test at 0b06284: "keeps a heavy file a script builds from split strings in the zip". It fails when the reference filter is bypassed.
  2. Nested entry GIF resolved against the root. Fixed at 1d8c3d12. Entry GIFs resolve with the renderer's sibling-first rule and reach GIF prep through its sourceAssets map. Test: "prepares a nested entry's GIF from the file next to it, as the renderer resolves it".
  3. Cache key and encode read different snapshots. Fixed at 1d8c3d12 for in-memory media (the encode reads a temp copy of the staged bytes). Media staged on disk (5 MB and up) is checked by size and modification time at hash, after the encode and after the upload; any change stops the run and nothing is cached (7855bbd). Tests: "re-encodes the staged bytes, not a file saved on disk after staging" and "stops without caching when media staged on disk changes during its encode".

Also at ad0e2c8: the two CodeQL file-race notes. Staging, the on-disk staged read and the re-encoded output read each now open the file once, check that open file's stat before and after the read, and read from it. Test: "reads a file left on disk only while it is the file that was staged".

Could you re-review at 0b06284?

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@somanshreddy On your cache-key point (4): Miguel decided to keep the upload cache per machine, not per account. It stays keyed by API host, content, encode settings and cut, with no extra /v3/users/me call per render. The consequence is the one you named: two accounts on one machine that render the same file share one public asset URL, and it is uploaded again if that URL stops serving the file (HEAD size check). The PR body's Decisions section now records this.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 0b062841. The delta since my COMMENT at 1d8c3d12 is: ed4ac226, the main merge 65d38696, 7855bbd3 (over 2 GiB), ad0e2c82 (one descriptor) and 0b062841 (split-strings test). Nothing from my side blocks. I'm not approving yet, because @terencecho's CHANGES_REQUESTED (5581308964, from 1158d685) is still open. Here's how I read its three blockers at this head:

  1. Dynamic path loses its ZIP copy: addressed. Only files a rewritable reference points at are offloaded (referencedFiles). The new test shows a file built from split strings ("media/" + "clip" + ".mp4") is never uploaded and stays in the ZIP (upload not called, fileMap.has true). isStillMentioned still keeps a ZIP copy when the basename survives rewriting.
  2. Entry GIF resolved against the root: addressed. prepareEntryGifs now passes sourceAssets from entryGifSources, which resolves each entry GIF the way the renderer rebases a nested entry. So compositions/scene.html's logo.gif maps to the sibling-first file, not the root one.
  3. Encode vs cache-key snapshot: addressed. planFile hashes the staged file (stagedSha256), and snapshotPath makes the encode read those bytes: an in-memory buffer gets a temp copy, and a file left on disk is used in place and checked with assertUnchanged. ad0e2c82 makes the size/change check and the read go through one open descriptor, so a save mid-render can't pair B's encode with A's key.

My earlier open point, the cache key not being account-scoped, is now a recorded product decision (per machine). That's acceptable as decided: the key is the content hash, so another account on the same machine reuses an asset only for byte-identical media it already holds.

Validate docs is red but not required.

Once Terence re-reviews or dismisses his CR, I'll approve this head.

— Somu

@terencecho terencecho left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at 0b0628411008adb56dac38012d91548c505595d4 (effective base c8f322c25910ed224e29ca5d1a62a521e62544cc). Three P2 source-level correctness issues remain.

1. Preserve the ZIP copy when a statically referenced file also has a split-built runtime reference

packages/cli/src/cloud/offload.ts:710–733

For a >=5 MiB media/clip.mp4, a page can contain both <video id="v" src="media/clip.mp4"> and document.getElementById("v").src = "media/" + "clip" + ".mp4". The static source admits the file for offload and is rewritten to the asset URL. The split literals are not rewritten and contain no complete clip.mp4 basename. After stripping the inserted asset URL, isStillMentioned therefore returns false and deletes the original from the archive. The script still requests the now-absent local file, whereas the base archive retained it.

The new split-only regression is a useful control, but it avoids offloading altogether; it does not cover this mixed case. This is also the remaining case acknowledged in the author's earlier response, rather than an unanswered question about the split-only fix. Please retain the original when runtime references cannot be proven rewritten, and add the mixed static/split-built case.

2. Apply the staged-file identity guarantee to the disk-backed entry-GIF path

packages/cli/src/cloud/offload.ts:650–654 (also 459–460, 620–634, 714–715)

A >=5 MiB GIF is a DiskFile, so snapshotPath returns its live original path. entryGifSources passes that path through the awaited producer import/preparation without the unchanged checks used by ordinary offload. The real producer reads the GIF bytes for metadata/hash, then transcodes by opening the same path (animatedGifPrep.ts:394–415, 284–305). Once prepared, the original GIF can be deleted from fileMap, so the later archive readStaged check cannot catch the change either.

Consequently, an edit after staging but before preparation can be accepted as the GIF input without the changed staged input being rejected. On a fresh GIF-cache miss, an edit after the producer hashes version A but before ffmpeg opens the path can also cache version B's video under A's content-derived GIF key. Buffer-backed GIF snapshots and the ordinary-file post-encode check do not cover this branch. Please provide an immutable input for GIF hashing/encoding, or enforce identity checks before accepting/persisting the prepared output; cover the disk-backed path, not only buildPublishFileMap's all-Buffer fixtures. This is an interleaving identified from source, not an observed production cache incident.

3. Do not trim away media that supported page initialization selects by changing its in-point

packages/cli/src/cloud/offloadTrim.ts:113–120 (also 40–54; caller offload.ts:539–550)

The basename-occurrence check only rules out additional textual references to the file; it does not establish that the clips' timing is static. For example, use a >=5 MiB, >102-second media/music.flac with a two-second root composition, <audio id="a" src="media/music.flac" data-start="0" data-duration="2" data-media-start="10">, and initialization that runs const a = document.getElementById("a"); a.setAttribute("data-media-start", "100"); a.volume = 0.5;. There is still only one basename occurrence, so the planner cuts 9.5–12.5 seconds and rewrites the static in-point to 0.5, but leaves the script's assignment to 100 intact.

This is not dependent on a browser probe that never runs: .volume = causes hasScriptedAudioVolumeAutomation/probeRequiresBrowser to select the real probe (probeStage.ts:135–149, 331–345), and its runtime reconciliation adopts the changed media start (185–215, 642–646). The base retains the whole source containing the 100-second passage; the new trimmed asset no longer contains it. Please conservatively skip trimming when supported runtime initialization can change a clip's playback window, with a regression covering an ID-based timing mutation that does not repeat the filename.

Verification limits

These are source-derived input/state traces, not executed reproductions. One reviewer completed all 12 BASE bodies before all 15 HEAD bodies in the changed-source pass; I checked its actual source deliveries and adjudicated the cited paths and the real GIF/probe consumers directly. The primary source/conventions pass stopped at its metadata cap after 13/15 HEAD bodies and remains incomplete. Two additional supporting-file reads in the completed changed-source pass were partial. This is not a complete two-pass approval.

No render, ffmpeg, upload, cache, browser/native/cloud reproduction, tests, or CI were run. No complete transitive build/deploy or pixel/audio-equivalence claim is made. Some source-read reservations were not renewed after context interruptions, some historical API/copy accounting could not be reconstructed exactly, and the first authored draft was charged after creation. Those process gaps remain; verifying the delivered source does not establish full reservation/accounting compliance.

The nested-entry GIF source resolution, normal Buffer snapshot, and split-only reference fixes are acknowledged; the above findings concern the remaining paths, not a blanket rejection of those fixes. Shared-cache/public-URL policy remains an attributed product choice, not an additional approval requirement.

— tai

…clips scripts retime

- An offloaded file keeps its zip copy when script code holds both its stem and
  extension, so a path joined at runtime still loads.
- Entry GIFs are prepared from a private copy checked against the staged file,
  so the GIF cache never stores a later edit.
- No cut when any script can write a clip's timing attributes, dataset,
  currentTime, playbackRate, src or loop.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@terencecho all three are fixed at 1b8cd79, each with a regression test. I reverted each fix on its own and only that fix's test went red.

1. Static plus split-built reference. The cause: the drop decision looked only for the whole file name. Now the zip copy also stays when the page's script code contains both the file's stem and its extension, which a runtime join needs. Script code here means inline scripts plus .js/.mjs/.cjs/.json files, with asset URLs removed first. Test: keeps a zip copy of an offloaded file a script can also build from split strings (your exact page). Limit: a stem that is itself built from fragments ("cl" + "ip") is not caught. No static check can prove arbitrary string building, so this errs toward keeping the file. A false positive costs only a bigger zip; the output is still correct, and the archive size check names the file if it goes over.

2. Entry GIF staged on disk. The renderer's GIF prep now always gets a private copy. GIFs staged in memory were already written to a temp file. GIFs staged on disk are now copied, and right after the copy the original is checked against its staged size and mtime. The producer hashes and encodes only that copy, so the GIF cache can only hold the staged bytes. An edit between staging and the copy fails the render. Test: prepares an entry GIF staged on disk from a checked private copy. It uses real disk staging and covers an edit during prep and an edit before prep.

3. Clips a script can retime. planTrim now skips the cut when any script the page runs can write a clip's timing: the timing attribute names (data-start, data-duration, data-end, data-media-start, data-playback-start, data-playback-rate), dataset, setAttribute, currentTime, playbackRate, .src or .loop. Your ID-based setAttribute page and a dataset.mediaStart write from an external script are now cases in keeps the whole file when a use is not a bounded clip or a script names it. Same limit as 1: a name built from fragments is not caught.

Checks run: offload.test.ts passes 23 of 23, three runs in a row. Typecheck, oxlint, oxfmt and the comment ratchet are clean. Re-review welcome.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 1b8cd794, covering the delta 0b062841..1b8cd794 (one commit) against terencecho's three points at 0b062841.

  1. Static plus split-built reference: fixed.
    • dropReplacedFiles now keeps the zip copy whenever the page's script code holds the file's stem and its extension, after asset URLs are stripped. "Script code" means inline <script> text from HTML files plus .js, .mjs, .cjs and .json files.
    • I checked the one way that could go blind: stageFile keeps every TEXT_FILE_EXTS file (html|css|m?js|cjs|json|svg|xml|txt) in memory regardless of size. So a large local script is never a DiskFile, and textOf always sees it.
    • The check is deliberately over-inclusive (any script containing clip and mp4). That fails toward keeping the file.
  2. Disk-staged entry GIF: fixed. entryGifSources now always uses stagedCopy. A disk-staged GIF is copyFileSync'd and then assertUnchanged against its staged size and mtime. The producer hashes and transcodes only the private copy, so an edit after the copy can't reach the GIF cache, and an edit before it fails the render.
  3. Clips a script can retime: fixed. planTrim returns null when RETIMING_SCRIPT_RE matches any page script: the timing data-* names, dataset, setAttribute, currentTime, playbackRate, .src or .loop. Terence's ID-based setAttribute page and an external dataset.mediaStart write are now tests.

Non-blocking:

  • The retiming check runs over every script in the project, not per clip. So a project that ships any library locally (a vendored gsap.min.js contains setAttribute) never gets trimming. That's correct, but the trim saving will rarely apply to real projects. A per-file sidecar metric (trim_skipped: "script") would show how often.
  • scriptCode(ctx.fileMap) re-parses every HTML file once per trim candidate. Computing it once per publish would avoid that.

No blockers from me at this head. I can't approve while terencecho's CHANGES_REQUESTED (5592149963 at 0b062841) is open. Once Terence clears it, ping me and I'll approve this head.

— Somu

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@miguel-heygen
miguel-heygen dismissed stale reviews from terencecho and terencecho October 21, 2026 13:49

Dismissed by the maintainer: raised on an earlier commit and answered at 1b8cd79; the reviewer called it historical, not a finding.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 1b8cd794, the same head as my re-review 5595319593. That review found all three of terencecho's points from 0b062841 fixed, with only non-blocking notes. A maintainer has since dismissed his CHANGES_REQUESTED, the head hasn't moved, and every required check is green.

— Somu

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit b306438 Oct 21, 2026
79 checks passed
@miguel-heygen
miguel-heygen deleted the feat/cloud-render-asset-offload branch October 21, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants