Switchyard v0.3.0

Switchyard 0.3.0 adds reusable runner integrations, Advisor Gate and Composite routing, Auto presets, and updated Rust and Python embedding APIs. It also fixes provider translation and tool-loop behavior and removes the legacy Python server and coding-agent launchers.

This is a breaking upgrade for library and Python CLI users. Review the migration notes below before upgrading.

Full changelog · Changes since v0.2.0

Routing and integrations

• Advisor Gate lets an executor model work with a stronger advisor for review, approval, and redo decisions, with session review budgets and stall checkpoints.
• Composite routing combines an LLM classifier with Stage Router. Auto presets provide capable and efficient Stage configurations.
• The new switchyard-runner crate loads TOML deployments and runs routing in-process. NeMo Relay and LiteLLM integrations use the native routing path.
• Sub-agent routing, per-target system prompts and reasoning effort, and per-client request deadlines provide more control over agent traffic.
• DeepSWE v1.1 and NeMo Gym/LiteLLM examples document reproducible evaluation setups. Their results apply to the documented configurations, not every deployment.

Library and Python APIs

• Python requires 3.10 or newer; the core package has no runtime dependencies.
• Python streaming exposes Step.CallModel and Step.Done, with response dictionaries and async iterators.
• Routing outcomes report ordered selected_model_ids, bounded decision metadata, and evidence through Rust, Python, logs, and OpenTelemetry.
• The decision endpoint supports routing without executing the selected answer model.

Compatibility and correctness

• Provider translation preserves tool allowlists, parallel-call limits, native tool history, reasoning signatures, citations, and terminal Responses output. Unsupported cross-format native output fails explicitly.
• Streaming fixes retain late response IDs, fragmented tool names, empty tool inputs, and tool-use stop reasons.
• Responses continuations return to the provider that holds their state. Cross-format continuations retain conversation history in process memory; records are lost on restart.
• Media and tool-result translation preserves supported image, audio, document, and file payloads.
• Credential redaction and header handling protect request trust boundaries. Failed or abandoned streams count as errors, and usage reporting preserves translated cache-write and thinking tokens.

Migration and removals

• The Python switchyard command, switchyard serve, coding-agent wrappers, YAML route bundles, legacy server stack, and compatibility bindings are removed. Connect clients to the standalone switchyard-server with native TOML configuration.
• Package extras [server], [all], [cli], [tracing], and [affinity-redis] are removed. The Python package declares no optional dependency extras.
• classify_trigger replaces session_affinity; use classify_trigger = "new_session" for the previous session-affinity behavior.
• Rust and Python embedding callers must adopt the updated step and routing-outcome APIs. Rust struct literals may also need the new Response.upstream_headers and Metadata.subagent_identity_unsupported fields.

Install

After the tag-triggered package publishing workflow completes, install the Python bindings with:

pip install nemo-switchyard==0.3.0

Install the standalone server separately:

cargo install --locked switchyard-server --version 0.3.0

See the getting-started guide for configuration and source builds.

Limitations

The standalone server remains a demo/evaluation component. Release validation covers Ubuntu 24.04 on Linux x86_64; other platforms are outside that scope.

The Relay integration requires >=0.8.0, <1.0.0. Relay 0.8.x and 0.9.0 have a documented native-plugin upstream-error propagation issue. Plugin bundles are published separately; see the Relay integration guide.

Prefill Router and the LiteLLM example remain experimental. Prefill Router does not include a supported checkpoint, exporter, or model assets. See the routing overview and LiteLLM guide.

Switchyard v0.2.0

Switchyard v0.2.0 is a substantial redesign of the project around a native Rust server and the new libsy library architecture. Across 193 commits, this release separates orchestration, provider-neutral protocols, translation, model transport, and serving into focused components that can be used together or embedded independently.

Switchyard remains pre-alpha software. APIs and configuration may change before a stable release.

libsy Redesign

switchyard-libsy provides a provider-neutral framework for multi-LLM orchestration. Its Algorithm abstraction decides which semantic model targets to call, in what order, and how to combine their results.

Algorithms do not own provider SDKs or an HTTP stack. They yield model-call steps back to their host, allowing the same algorithm to run inside the Switchyard server, another proxy, an agent runtime, or an application with custom model clients. The driver supports both buffered and streaming execution, concurrent model calls, routing decisions, request processors, classifiers, and session-scoped state.

The redesign establishes clear crate boundaries:

• switchyard-protocol owns normalized request, response, streaming, routing, tool, usage, error, and metadata contracts.
• switchyard-translation encodes and decodes OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages.
• switchyard-libsy owns orchestration algorithms and their execution model.
• switchyard-llm-client supplies translated HTTP model calls and a ready-made libsy driver.
• switchyard-server composes these crates into a standalone proxy.

libsy includes typed errors, streamed algorithm output, concurrent calls, affinity-aware state, context-window recovery, classifier contracts, per-target prompts, handoff notes, escalation policies, and OpenTelemetry instrumentation.

Native Server

The new switchyard-server serves all three supported APIs from one explicit TOML deployment:

• OpenAI Chat Completions at /v1/chat/completions
• OpenAI Responses at /v1/responses
• Anthropic Messages at /v1/messages
• Anthropic token counting at /v1/messages/count_tokens
• Model discovery at /v1/models
• Health checks, Prometheus metrics, and JSON routing statistics

TOML configuration declares LLM clients, credentials, wire formats, targets, and algorithms explicitly. The server includes TLS, graceful shutdown, bounded upstream retries, model discovery, token counting, optional durable routing logs, and context-window overflow recovery.

Responses now consistently report the model target that actually served the request rather than the client-facing route alias. /v1/models also reports declared capabilities and Codex metadata.

Routing and Orchestration

The native algorithm set now includes:

• LLM task classification using a dedicated judge target.
• Capability, escalation, custom-schema, and multi-target classification policies.
• Signal-driven stage routing based on coding-agent tool use and progress.
• Per-target system prompts and handoff notes.
• Optional LLM fallback when stage-routing signals are inconclusive.
• Session affinity and sub-agent-aware target selection.
• Context-window eviction and rerouting with session-isolated history.
• Probability-based capability policies and configurable classifier prompts.

Classifier history preserves tool-call and tool-result pairs, malformed classifier scores fail safely, and streamed judge responses are fully consumed before a routing decision is made.

Translation and API Compatibility

Translation now preserves substantially more information across API boundaries:

• Buffered and streaming responses.
• Tool definitions, tool calls, and tool results.
• Responses reasoning and final-answer items.
• Usage details, cached-token counts, and cache-creation counts.
• Provider stream events for compatible same-format replay.
• Max-token truncation and incomplete Responses status.
• Prompt-cache metadata across Anthropic-compatible providers.

Known request fields are validated before translation rather than being silently coerced or omitted. Anthropic endpoints return Anthropic error envelopes, accept compatible stream terminators, filter incompatible beta headers and OpenAI-only fields, and omit unsigned thinking blocks rejected by Anthropic-compatible upstreams.

Anthropic prompt caching is enabled by default for translated calls. Streaming now terminates after an in-band upstream error instead of forwarding unrelated trailing events.

Observability

The native path includes:

• Prometheus metrics at /metrics.
• OpenTelemetry spans using GenAI semantic attributes.
• Structured request, target, latency, retry, token, cache, and error telemetry.
• Aggregate statistics at /v1/stats.
• Statistics reset through /v1/stats/reset.
• Optional per-session routing statistics at /v1/routing/session-stats.
• Optional durable JSONL routing logs for evaluation workloads.

Python and Coding-Agent Launchers

Python users can run native algorithms through switchyard.libsy and host the Rust server in-process through switchyard_rust.server.Server.

Claude Code, Codex CLI, and OpenClaw launchers now use the native server, TOML routes, translation, statistics, and OpenTelemetry paths. A packaged OpenRouter deployment provides the default launcher configuration, while --config accepts custom TOML deployments.

An experimental LiteLLM integration is also included.

Install

Python CLI and bindings:

uv tool install --python 3.12 "nemo-switchyard[cli,server]==0.2.0"

Native Rust server:

cargo install --locked switchyard-server --version 0.2.0

Published Packages

Python:

• nemo-switchyard 0.2.0

Rust:

• switchyard-protocol 0.2.0
• switchyard-translation 0.2.0
• switchyard-libsy 0.2.0
• switchyard-llm-client 0.2.0
• switchyard-server 0.2.0

Deprecations and Removals

The Python switchyard serve path, Python server, YAML route bundles, and profile APIs remain available for transition purposes but are deprecated. New deployments should use switchyard-server, native TOML configuration, and libsy algorithms.

The legacy Rust core, components-v2 stack, PyO3 profile bindings, plan-and-execute routing, RouteLLM integration, external router plugin, latency-aware router, and Intake-specific sink have been removed.

The CLI is now focused on switchyard launch and switchyard serve. The previous configure, verify, status, settings, smoke-test, and TUI workflows have been removed. The former cascade route is now named stage_router.

Known Issues

• Buffered upstream work may continue after a client disconnects and still incur provider cost.
• Some fallback and escalation decisions lack routing-target attribution in /v1/stats and /metrics.
• The retry-recovery counter remains zero after a successful upstream retry.
• x-switchyard-session-id is not yet included in native session statistics.
• The native server does not yet forward X-Switchyard-Version upstream.

See the getting-started guide, known issues, and full changelog.

Switchyard v0.1.0

Switchyard v0.1.0 was the first public release. It introduced the Python proxy, OpenAI and Anthropic format translation, YAML route bundles, initial routing strategies, coding-agent launchers, and request metrics.

This release is retained for historical reference. New users should start with v0.2.0.

• nemo-switchyard 0.1.0 on PyPI
• Full v0.1.0 changelog