feat: accept inline text input_file content on typed Responses paths by MetehanAydemir · Pull Request #471 · vllm-project/agentic-api · GitHub
Skip to content

feat: accept inline text input_file content on typed Responses paths - #471

Merged
maralbahari merged 8 commits into
vllm-project:mainfrom
MetehanAydemir:work/469-input-file-text
Oct 22, 2026
Merged

maralbahari merged 8 commits into
vllm-project:mainfrom
MetehanAydemir:work/469-input-file-text

Conversation

@MetehanAydemir

Copy link
Copy Markdown
Contributor

Summary

Closes #469.

The typed Responses executor now accepts a user-message input_file when its file_data carries an inline UTF-8 text
file, and sends it to the model as text. Until now every message input_file got a 400 (#256/#258). That meant a
text-only model behind the gateway could not read a .txt, .md, .json or .yaml attachment a client sent inline.

What is accepted:

  • a base64 data URL with a text media type (text/*, application/json, application/xml, application/yaml).
    Parameters such as charset are ignored, and an explicit media type takes precedence over the filename;
  • raw base64, or application/octet-stream / an empty media type, with a filename that has a common text extension.

How it works:

  • Validation (types/io/input/validation.rs, new types/io/input/file_text.rs). The existing validation runs on
    new input, rehydrated previous_response_id/conversation history and every upstream build (including the
    compaction summary). It now decodes the file (base64 crate, already a dependency, then strict UTF-8). Each other
    case gets a specific 400 reason: file_id (no Files API), file_url (no fetching), missing file_data, non-text or
    PDF data, a non-base64 data URL, invalid base64, non-UTF-8. The input[i].content[j]: prefix is kept, and no reason
    echoes the file data, URL or media type.
  • Model-facing copy only (normalized_model_input). The file becomes an input_text part in the same position,
    prefixed with File: <filename>\n when a filename is set. This is the same presentation proposed in vLLM #62676, so
    both paths would give the model the same prompt.
  • Storage is unchanged. Stored history and compacted windows keep the client's original input_file, and the text
    is re-inlined on each continuation. This follows the fix: preserve and verify image input through Responses #263 rule: never persist content the client did not send.
  • Routing. A store: false request with an inline text file goes to the executor instead of the raw proxy, so
    stateless and stored requests give the model the same input. store: false requests with file_id, file_url or
    PDF files are still raw-proxied byte for byte. The predicate (ResponsesInput::has_inline_text_file) only classifies
    the reference fields and the media type, without decoding. It is checked in the HTTP routing decision next to the
    tool-search state, not in in_process_feature(). in_process_feature() also gates split execution in
    agentic-llm-d, and split execution already builds its upstream body through to_upstream_request, so it can serve
    these files.
  • Compaction (executor/compaction/context.rs). An inline file counts as meaningful context, so a compaction whose
    input is only a file no longer fails with "requires non-empty input".
  • Docs/CHANGELOG. Rewrote "File inputs" in docs/guides/responses-compaction.md. The section appeared twice, so
    this also removes the duplicate. Added an [Unreleased] entry.

Not in scope: file_id resolution (#356), fetching file_url, PDF/DOCX extraction, Chat Completions file parts
(that route is a passthrough), and a separate decoded-size limit. Inline files are bounded by the existing
max_request_body_size_bytes, and the body shrinks after decoding. No new dependencies, no I/O. No changes to
events/, the ingestion state machine, the relay or engine.rs.

Open questions for reviewers:

  1. Should store: false text-file requests go through the executor, as here, or should the raw proxy stay
    byte-transparent for them? Dropping the routing change is a two-line revert in the HTTP handler.
  2. Is the File: <filename> header the format you want?
  3. Do you want an explicit decoded-size cap in ResponsesConfig? Adding a field would break struct literals, so it is
    left out here.
  4. Should the compaction token estimate count the decoded size (about 3/4 of the base64 length) instead of the base64
    length? It is unchanged here, which overestimates.

Test Plan

Unit tests (agentic-server-core):

  • file_text.rs: accepted media types (including charset parameters and uppercase types), an
    application/octet-stream or empty media type falling back to the filename, raw base64 with .yaml/.YML, empty
    and multi-byte UTF-8 files, a PDF rejected by media type (even with a .txt filename) and by filename, file_id,
    file_url, missing data, a non-base64 data URL, invalid base64, non-UTF-8, and routing classification (text
    files are detected without decoding; references, PDFs, images and tool-output files are not).
  • validation.rs: file errors keep their input[i].content[j] position and never echo the URL. Inline text files pass
    in new input and in restored history. The previous input_file case in the position test now uses an unknown part
    type, and file_url is covered by its own test.
  • conversions.rs: the model copy has input_text with the header in the original order. Canonical items keep
    input_file. Input with no files stays Cow::Borrowed.

Integration tests (crates/agentic-server/tests/input_file_repro.rs, mock upstream and in-memory SQLite). All 30
existing tests still pass unchanged. They use a file_url PDF, which is still rejected before inference and before
persistence, and the raw-proxy byte-for-byte controls still hold. 9 new tests:

  • store: true with a data URL: 200. The upstream gets input_text "File: notes.txt\n..." with no file_data, and
    the stored item is the original input_file;
  • raw base64 with filename: config.yaml is inlined;
  • streaming HTTP gives the same upstream body;
  • a previous_response_id follow-up re-inlines the stored file, and storage still holds the single original input_file;
  • WebSocket response.create is accepted and inlined;
  • store: false with a text file runs through the executor: inlined, nothing persisted;
  • store: false with invalid base64 gets a 400 from the gateway instead of being proxied;
  • /v1/responses/compact with a message containing only a file succeeds. The summary request has the text, and the
    returned window keeps input_file;
  • each rejection reason (file_id, file_url, missing data, PDF data URL, PDF filename, non-base64 data URL, invalid
    base64, non-UTF-8) gives a 400 with the specific message: no inference, nothing persisted.

Commands, all run in WSL Ubuntu-22.04 with Rust 1.98.1:

  • cargo fmt --all --check: pass
  • cargo clippy --all-targets -- -D warnings: pass
  • cargo test -p agentic-server --test input_file_repro: 39 passed, 0 failed
  • cargo test -p agentic-server: 710 passed, 0 failed (42 test binaries)
  • cargo test -p agentic-server-core: 1776 passed, 1 failed, 17 ignored. The failure is
    tool_search_characterization_test::provider_parity_recorder_generated_matrix_has_one_semantic_flow, which asserts
    that checked-in cassettes are not executable. It fails only because the checkout is on a Windows drive mounted in WSL
    (/mnt/c), where every file reports mode 0777. This change touches no cassettes.
  • python scripts/check_rust_file_sizes.py (the rust-file-sizes hook): pass, 291 production files checked. The new
    file_text.rs has 155 production lines. types/io/input.rs has 485 production lines, plus one mod line.
  • git diff --check: clean

A message input_file is now accepted when file_data carries an inline
UTF-8 text file: a base64 data URL with a text media type (text/*,
application/json, application/xml, application/yaml), or raw base64 /
application/octet-stream with a common text extension in filename.

The file is decoded only in the model-facing copy built by
normalized_model_input, where it becomes an input_text part in the same
position, prefixed with "File: <filename>" when a filename is set. This
matches the presentation proposed for vLLM. Stored history and compacted
windows keep the client's original input_file, and the text is
re-inlined on every continuation, so nothing is persisted that the
client did not send.

Validation keeps running on new input, rehydrated history and every
upstream build, now with a specific 400 reason for file_id, file_url, a
missing file_data, non-text or PDF data, non-base64 data URLs, invalid
base64 and non-UTF-8 content. No reason echoes the file data or URL.

The HTTP handler routes store:false requests that carry an inline text
file to the executor, so stateless and stored requests give the model
the same input. Other files on store:false keep the byte-transparent
raw proxy. Compaction counts an inline file as meaningful context.

Also removes the duplicated "File inputs" section in the compaction
guide and documents the new behaviour.

Closes vllm-project#469

Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>
Comment thread crates/agentic-server-core/src/types/io/input/conversions.rs Outdated

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing this fix! I added inline comments

Comment thread crates/agentic-server/src/handler/http/responses.rs Outdated
Comment thread crates/agentic-server/tests/input_file_repro.rs
Comment thread crates/agentic-server-core/src/types/io/input/validation.rs Outdated
Comment thread crates/agentic-server-core/src/types/io/input/file_text.rs Outdated
…yFrom

Stop routing every request with an inline text file to the typed executor.
Raw store:false proxy requests now reach the upstream byte for byte again,
so a mixed text and PDF request keeps its upstream status and provider
fields such as chat_template_kwargs are no longer dropped. Text files are
converted only on paths that already need typed execution.

Express the model-facing conversion as
TryFrom<&InputFileContent> for InputTextContent with the public
InputFileError, and keep validation on InputFileContent without building
the filename-prefixed text. normalized_model_input now returns a Result and
propagates conversion failures instead of leaving an unsupported file in
the upstream request. InputFileContent's serde representation is unchanged.

Add regression tests for the raw proxy path and for automatic compaction
sending decoded text to the summarizer while storing the original file.

Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>
Replace the multi-line format literal and push_str in the InputTextContent conversion with one format string, and write the raw proxy test bodies with explicit escapes. Behaviour is unchanged.

Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>
Add record_text_file_input_cassettes.sh, modelled on the image-input
recorder, with notes.txt fixtures whose release codeword appears only in the
file. Every recording is validated to answer with that codeword before it is
promoted.

The gateway set was recorded through Agentic API against vLLM serving
Qwen/Qwen2.5-3B-Instruct: a single file turn and a previous_response_id
continuation, each streaming and non-streaming. The compaction summary and
its follow-up on the compacted window were recorded directly against vLLM,
because the gateway makes those calls internally. TEXT_FILE_RECORD_SET=openai
records the optional OpenAI reference set.

text_file_input_test.rs replays every recording through the gateway and
checks that the model receives the File: notes.txt input_text with no
input_file, that stored history and the compacted window keep the original
input_file, that continuation decodes the stored file again, and that the
summarizer receives exactly the recorded input. OpenAI recordings are
replayed and compared with the gateway set once they exist.

Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>
An explicit media type used to win outright, and only text/*, JSON, XML
and YAML counted as text, so JavaScript, shell, SQL, TOML, JSON-LD and
TypeScript files were rejected. Filenames were matched only by a short
extension list, so Dockerfile, Makefile, dotfiles and many source
extensions never matched.

Move the classification into file_text/kinds.rs with reviewed lists:
any +json or +xml type and text application/* types count as text media
types; the extension list covers common source and config files; and
known extensionless names and dotfiles are matched on the basename. A
text filename now decides for raw base64 and for any media type except
PDF, image, audio and video, and video/mp2t with a .ts, .mts or .cts name
is treated as TypeScript. Strict UTF-8 decoding still rejects binary
content. The not-text error now explains that raw base64 or a generic
media type needs a filename with a text extension.

Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>
Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>

@maralbahari maralbahari left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the comprehensive test support.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Now that the OpenAI cassettes are committed, this conditional fallback is unnecessary. Could you simplify the loading and remove the outdated “optional until recorded” comments?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @maralbahari, cleaned it up the outdated "optional" comments are gone. Thank you for giving your time to review this, I'm happy to contribute!

MetehanAydemir and others added 2 commits October 21, 2026 14:54
The OpenAI reference recordings are committed, so load both providers' cassettes unconditionally, drop the Option plumbing and skip note from the replay test, and remove the wording that described the OpenAI set as optional from the test, the cassettes README, and the recorder script.

Signed-off-by: MetehanAydemir <mmetehanaydemir@gmail.com>
@maralbahari
maralbahari merged commit 2bbb3cc into vllm-project:main Oct 22, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support inline text input_file content on typed Responses paths

4 participants