{{ message }}
feat(llm): add video and audio media support to Gemini protocol#31889
Merged
Conversation
…ni protocol Adds video file reading (MP4, WebM) to the read tool and widens the Gemini protocol to accept video MIME types via inlineData. MIME sniffing uses ftyp major brand validation for MP4 (excludes AVIF/HEIC/M4A) and EBML DocType verification for WebM (excludes Matroska). Video bypasses photon image normalization and passes through as raw base64. Gemini gets higher media size limits for video (20 MB decoded / 28 MB encoded) while other protocols keep the default image limits (6 MB / 8 MB). OpenAI and Anthropic stay image-only since their APIs do not support inline base64 video. Session: ses_1494c002affeaQLbzZve1wSJEm
Contributor
…pport Video support is provided through the protocol layer only, so plugins can return video via Tool.make toModelOutput. The native read tool stays image/PDF-only per maintainer preference. Session: ses_1494c002affeaQLbzZve1wSJEm
remorses
changed the title
- Replace base64 regex with character-by-character validator to avoid Bun/JSC regex failure on large strings (>4 MB) - Lower Gemini video limits to 14 MB decoded / 20 MB encoded to stay within Gemini total request size budget - Case-insensitive video MIME detection for limit selection Session: ses_1494c002affeaQLbzZve1wSJEm
Add VIDEO_MIMES and MEDIA_MIMES to ProviderShared and widen Gemini protocol to accept video MIME types (mp4, webm, quicktime) via inlineData. Plugins can now return video content to Gemini models through toModelOutput or v1 ToolAttachment. Other protocols stay image-only. Session: ses_1494c002affeaQLbzZve1wSJEm
Previous 6 MB / 8 MB limits were too tight for video. Raise to 20 MB decoded (raw bytes) / 28 MB encoded (base64 string) to support reasonable video clip sizes. Session: ses_1494c002affeaQLbzZve1wSJEm
Add AUDIO_MIMES array with 12 audio types supported by the Gemini API (wav, mp3, mpeg, aiff, aac, ogg, flac, m4a, mp4, opus, pcm, webm) and include it in MEDIA_MIMES. Since Gemini creates its local supported set from ProviderShared.MEDIA_MIMES, audio flows through the same inlineData path as images and video with zero protocol changes needed. Other protocols (OpenAI Chat, OpenAI Responses, Anthropic, Bedrock) are unaffected since they reference IMAGE_MIMES directly. Session: ses_135abc74affeXj589B4g7YkPb0
remorses
changed the title
github-actions
Bot
added
the
needs:compliance
Remove 6 audio types not listed in Gemini docs (mpeg, m4a, mp4, opus, pcm, webm). Keep only the 6 officially supported formats: wav, mp3, aiff, aac, ogg, flac. Ref: https://ai.google.dev/gemini-api/docs/audio#supported-audio-formats Session: ses_135abc74affeXj589B4g7YkPb0
github-actions
Bot
removed
the
needs:compliance
Contributor
markjaquith
pushed a commit
to markjaquith/opencode
that referenced
this pull request
Jun 23, 2026
…alyco#31889) Co-authored-by: Aiden Cline <63023139+rekram1-node@users.noreply.github.com>
BenGu3
pushed a commit
to BenGu3/opencode
that referenced
this pull request
Jun 27, 2026
…alyco#31889) Co-authored-by: Aiden Cline <63023139+rekram1-node@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
N/A
Type of change
What does this PR do?
Extends the Gemini protocol to accept video and audio media as
inlineData, not just images.Video: mp4, webm, quicktime added to
MEDIA_MIMES.Audio: wav, mp3, aiff, aac, ogg, flac added to
MEDIA_MIMES. These are the 6 formats listed in the Gemini audio docs.Gemini's protocol creates its local supported set from
ProviderShared.MEDIA_MIMES, so both video and audio flow through the sameinlineDatapath as images with no protocol file changes.Other protocols (OpenAI Chat, OpenAI Responses, Anthropic, Bedrock) are unaffected because they reference
IMAGE_MIMESdirectly.Also raises media size limits to 20 MB decoded / 28 MB encoded.
How did you verify your code works?
Existing Gemini test suite passes (
bun test test/provider/gemini.test.ts). Other provider tests (OpenAI Responses, Anthropic) still correctly reject audio.Checklist