fix(cli): init --video frames the project to the clip's shape, so nothing is cropped by miguel-heygen · Pull Request #5984 · heygen-com/hyperframes · GitHub
Skip to content

fix(cli): init --video frames the project to the clip's shape, so nothing is cropped - #5984

Open
miguel-heygen wants to merge 6 commits into
mainfrom
fix/init-video-frame
Open

miguel-heygen wants to merge 6 commits into
mainfrom
fix/init-video-frame

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What

hyperframes init --video now sizes the new project to the clip's own shape as it plays, with 1080 on the short side and the long side rounded to an even number. A 4:3 clip opens at 1440x1080, a 3:4 one at 1080x1440, and a 21:9 one at 2520x1080. Every preset already has a 1080 short side, so a clip that matches one gets exactly that preset: 16:9 is 1920x1080 (the template's own), 9:16 is 1080x1920, 4:5 is 1080x1350, 1:1 is 1080x1080. The stage matches the clip, so the whole picture shows with no bars and nothing cut. Before, every clip landed on the template's 1920x1080 stage with object-fit: cover, so a 9:16 clip lost 68% of its height.

The talking-head-recut skill no longer asks for the output ratio. It takes the source video's ratio, unless the request already names a ratio or platform, and asks only about layout, style and card count.

Why

The clip already answers the shape question. Asking the person, or cropping their footage until someone notices, is the wrong default. Phone clips are often stored landscape with a 90° rotation, so the probe has to read the clip as it plays, not as it is stored.

Related work

None.

How

  • probeVideo reports the size the clip plays at. Width is multiplied by the sample aspect ratio when the pixels are not square (a 720x576 PAL clip at 64:45 plays at 1024x576). Then the stream's rotation (display-matrix side data, else the legacy rotate tag) turns width and height upright.
  • clipFrame(templateId, clip) picks the frame: the clip's played aspect with a 1080 short side, both sides even, and the template's own size when that comes out 1920x1080. It applies only to the from-file template, the one whose stage is the clip. An explicit --example keeps its own design, and an explicit --resolution always wins. Both the interactive and non-interactive paths pass resolutionPreset ?? clipFrame(...) to scaffoldProject.
  • No new shared preset: applyResolutionPreset now delegates to applyCanvasSize(width, height), which writes the stage size (data attributes, inline html, body CSS, viewport). Only a named preset stamps data-resolution, which nothing reads; render takes the size from data-width/data-height.
  • The --resolution help text says what the default now is.
  • talking-head-recut Step 3's probe also records the stream's rotation and sample aspect ratio, so Step 7 can read the clip's shape as it plays. Step 7 says a ratio the request names wins, and otherwise it chooses the nearest preset. coreSkillContent.test.ts pins both.
  • The after-scaffold step (template telemetry and transcript patch) moved into one finishScaffold helper. Both paths ran the same lines, and fallow flagged them as a clone.

Test plan

  • Unit tests added/updated: init.test.ts runs init --video on six fixtures and checks the composition root's size:
    • a 36x64 clip (1080x1920);
    • a 64x36 clip carrying a 90° rotation (1080x1920);
    • a 48x64 3:4 clip (1080x1440);
    • a 64x48 4:3 clip (1440x1080);
    • a 64x64 clip (1080x1080);
    • a 72x72 clip with 16:9 pixels, which plays at 128x72 (1920x1080).
    • Fails without the fix:
      • with init.ts from main, the portrait, rotated, 3:4, 4:3 and square cases fail (5 failed, 1 passed);
      • at 1967c9d (nearest-preset snapping), the 3:4 and 4:3 cases fail (2 failed, 4 passed);
      • at 564aae1 (no sample aspect ratio), the 3:4, 4:3 and 16:9-pixel cases fail (3 failed, 3 passed).
    • At 17aa857, the whole file passed 3 runs in a row (43 passed) on macOS.
  • Manual testing performed:
    • init --video on a 720x576 H.264 clip at SAR 64:45 prints Video: 1024x576 and scaffolds a 1920x1080 root;
    • on a 640x480 clip it scaffolds a 1440x1080 root and viewport.
  • Documentation updated (if applicable): --resolution help, talking-head-recut Step 7.
  • Comments follow CONTRIBUTING.md "Comments"

lint-skills and the skill mirror check pass.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact bcf042f9f63f7d6e46ebf31c63d3605b07313c56, requested by Miguel. No blocking findings.

packages/cli/src/commands/init.ts:172–205 prefers display-matrix rotation over the legacy rotate tag, swaps upright dimensions for quarter turns, and selects portrait/square only for from-file. Both scaffolding callers use explicit resolution first (:964,1194); landscape and other example designs retain their previous defaults. The finishScaffold extraction preserves telemetry/transcript order.

skills/talking-head-recut/SKILL.md:266–274 expressly gives an already named ratio/platform precedence over source inference. The question now concerns layout/style/card count in both channels; it does not ask users to repeat a specified ratio. The metadata command retains rotation side data. CLI square footage uses1080×1080; the skill intentionally retains its existing near-square→4:5 preset mapping rather than promising arbitrary native dimensions.

Validation: read full init implementation/tests, from-file template and changed skill decision sections; no existing review or inline findings. My14 focused controls passed, including actual ffprobe on both committed fixtures, quarter/half turns, legacy rotation, square/odd/landscape shapes and scoped preset selection. A fresh independent pass passed21 controls, including generated real display matrices and actual rotated non-web MPEG-4→upright H.264 transcoding. Isolated caller/precedence checks are source controls, not an interactive CLI session. No full39-test rerun, TTY wizard, browser render or model-following-skill run claimed; author suite evidence remains attributed.

Nit: skills/talking-head-recut/SKILL.md:362,440 says the canvas matches/comes from the source even when an explicit user override wins or square footage maps to4:5. Say selected ratio there, and describe the inference as choosing a preset. Add a small policy pin for explicit-user-ratio precedence when maintaining the prompt tests.

Need: yes · Size: focused · Reuse: existing ffprobe/scaffold/preset path; no engine dependency introduced · Simpler: small orientation selector and shared existing post-scaffold work.

— hf-review

Verdict: APPROVE
Reasoning: Upright clip dimensions reach both init paths without overriding explicit choices, and the skill preserves user intent while removing the redundant ratio question.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 20, 2026
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a manual request Oct 20, 2026
@miguel-heygen miguel-heygen reopened this Oct 20, 2026
@miguel-heygen miguel-heygen changed the title fix(cli): a portrait clip passed to init --video makes a portrait project fix(cli): init --video frames the project to the clip's shape, so nothing is cropped Oct 20, 2026

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 564aae14cc9c732b85c7f73e25c2d14bad8a71f0, including the delta from my bcf042f9 approval. The shared canvas writer applies the selected dimensions to stage attributes/CSS/viewport, and both init branches retain explicit-resolution precedence. All four committed video fixtures select their expected frame in independent exact-source controls.

P2 blocker — use displayed aspect ratio, including non-square pixels. packages/cli/src/commands/init.ts:172–184 derives shape solely from coded width/height plus rotation; :207–214 uses that ratio to select the canvas. A real H.264 clip encoded 720×576, SAR 64:45 has DAR 16:9. ffprobe confirms that metadata, and real Chrome reports intrinsic 1024×576. Yet the exact current probeVideo + clipFrame select 1080×1080. Before this change the template stayed 1920×1080, matching this clip; the new square frame with the existing object-fit: cover loses 43.75% of its horizontal picture. Include sample/display aspect ratio before applying rotation and choosing the nearest preset, and add an anamorphic regression fixture.

Important — qualify the no-crop claim or preserve the whole picture. templates/from-file/index.html:49 still uses cover. Even the new 48×64 (3:4) fixture cropped into a 4:5 canvas loses 6.25% vertically. Snapping improves the default but does not establish the body's “nothing is cut”; a whole-picture promise needs contain/letterboxing for unmatched ratios.

Validation: independent real ffmpeg/ffprobe fixture plus exact-source selection and browser intrinsic-dimension probe; the four committed fixture selections pass. Full init suite could not collect locally because borrowed dependencies lack @hyperframes/parsers/slideshow; no full CLI wizard/render or model-following-skill run claimed. Re-read changed init/scaffold/preset paths and tests; the unchanged skill policy/nit from my prior review remains.

Need: yes · Size: focused · Reuse: extend the existing ffprobe metadata path and canvas writer · Simpler: compute one displayed aspect ratio before preset selection.

— hf-review

Verdict: REQUEST CHANGES
Reasoning: Rotation handling and 4:5 selection work for square-pixel fixtures, but the new inference introduces a reproducible wrong-canvas regression for anamorphic video.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Addressed at 1967c9d:

  • Anamorphic clips (P2): fixed. probeVideo multiplies width by the sample aspect ratio before rotation and snapping, so your 720x576 SAR 64:45 clip reports Video: 1024x576 and scaffolds 1920x1080. The new anamorphic.mp4 fixture (64x64 coded, 16:9 pixels) fails at 564aae1 (1 failed, 4 passed) and passes now.
  • No-crop claim: qualified in the body with numbers. The stage keeps cover, so a clip between two sizes loses at most 16% (portrait) or 25% (near 4:3, the same as before this PR). Before this change, a 9:16 clip lost 68% of its height. Switching to contain/letterboxing is a design change, so I've left it out of this PR.
  • Skill nit: Step 7 now says "the selected ratio", describes the inference as choosing the nearest preset, and the Step 3 probe records sample_aspect_ratio. coreSkillContent.test.ts pins both the explicit-ratio precedence and the probe fields.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 8ad23620f52e12710c9ac4f94e5768975ec52d1f, including all changes since my request for changes at 564aae14.

Both concerns are resolved. packages/cli/src/commands/init.ts:173–190 applies sample aspect ratio before rotation; :200–210 now keeps arbitrary displayed ratios rather than snapping 3:4 or 4:3 to a mismatched preset. Both scaffold branches preserve explicit-resolution precedence, and explicit non-from-file designs keep their dimensions. The shared canvas writer updates the existing stage/CSS/viewport surfaces. The skill retains user-ratio precedence and its corrected probe fields.

Independent exact-source controls using real ffprobe pass all six committed fixtures, the original 720×576 SAR64:45 counterexample, combined SAR+90° rotation, 21:9 and even-size rounding/fallback cases. I reran that probe successfully. No full CLI/TTY/browser-render or model-following-skill acceptance is claimed.

Need: yes · Size: focused · Reuse: existing probe, scaffold and canvas writer · Simpler: one displayed-size calculation, shared by both init paths.

— hf-review

Verdict: APPROVE
Reasoning: The anamorphic regression and preset-snapping crop are repaired at this head, with no new blocker found in the reviewed paths.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants