inference-router

A provider-agnostic inference client, so swapping the LLM backend is a config change — not a rewrite.

Your app code talks to one small InferenceProvider interface; a router maps an environment variable to the backend that actually serves the request (a local Ollama model, Anthropic, a deterministic mock). The serving path is the one place single-provider lock-in is a standing liability, so this keeps a thin abstraction between your code and any one vendor's SDK — plus a continuous smoke test against a second backend, so failover is a tested claim, not a hope.

Backs the post: Renting Compute From Three Clouds Is the Default Now on The Cloud Codex. See also On-Prem AI Is a Bet on Control.

Install

git clone https://github.com/michaeltuszynski/inference-router
cd inference-router
npm install

Node 20+. The default path (tests + quickstart) runs offline with no API keys — it uses the deterministic MockProvider. Anthropic's SDK is an optional devDependency, lazy-imported only when you select the anthropic backend.

Quickstart

Runs offline against the mock backend and prints a smoke-test result:

npm run example

Expected output:

{
  "ok": true,
  "latencyMs": 0.12,
  "provider": "mock",
  "detail": "mock via mock-1: mock: gnip"
}

In code:

import { createRouter, smokeTest } from "inference-router";

const provider = createRouter();              // backend ?? INFERENCE_BACKEND ?? "mock"
const res = await provider.complete({
  messages: [{ role: "user", content: "Say hello." }],
});
console.log(res.text, res.model, res.provider);

console.log(await smokeTest(provider));        // { ok, latencyMs, detail }

Point it at a real backend (same code, env-var only)

Local model via Ollama (OpenAI-compatible API) — no code change:

INFERENCE_BACKEND=openai-compatible \
OPENAI_BASE_URL=http://localhost:11434/v1 \
OPENAI_MODEL=llama3.1 \
npm run example

Anthropic:

INFERENCE_BACKEND=anthropic \
ANTHROPIC_API_KEY=sk-ant-... \
ANTHROPIC_MODEL=claude-opus-4-8 \
npm run example

Copy .env.example to .env for the full list of knobs.

How it works

App code depends on one interface, never on a vendor SDK:

interface CompletionRequest {
  system?: string;
  messages: { role: "user" | "assistant"; content: string }[];
  maxTokens?: number;
  temperature?: number;
}
interface CompletionResponse {
  text: string;
  model: string;
  provider: string;
  usage?: { inputTokens: number; outputTokens: number };
}
interface InferenceProvider {
  readonly name: string;
  complete(req: CompletionRequest): Promise<CompletionResponse>;
}

createRouter is the single switchpoint that maps INFERENCE_BACKEND to a concrete provider:

function createRouter(opts?: { backend?: string }): InferenceProvider;
// backend ?? process.env.INFERENCE_BACKEND ?? "mock"

INFERENCE_BACKEND	Provider	Notes
mock (default)	MockProvider	Deterministic, offline. Reverses the last user message. Zero creds.
openai-compatible	OpenAICompatibleProvider	POSTs to any /chat/completions via fetch — no SDK. Run locally against Ollama.
anthropic	AnthropicProvider	Uses @anthropic-ai/sdk (optional, lazy-imported). Reads ANTHROPIC_API_KEY.

An unknown backend throws immediately with the list of known ids — a typo fails loud, not silently.

smokeTest(provider) runs one cheap completion and returns { ok, latencyMs, detail }. Wire it to a second backend in CI or a cron job: that turns "we could fail over to Ollama" from a hope into a continuously tested claim. examples/smoke.ts exits non-zero on failure so it drops straight into a health check.

Adding Bedrock, Vertex, or Azure is the same move every time: implement the InferenceProvider interface. The interface IS the contract — a new provider class plus a case in createRouter, and nothing in app code changes. That is the whole point: the backend you rent compute from is a configuration detail.

When NOT to use this

This abstraction has a real cost — be honest about it before adopting it.

• You forfeit provider-specific features. Prompt caching, batch APIs, fine-tuning, structured-output/JSON-schema enforcement, extended thinking, computer use, server-side tools — anything that lives in one vendor's SDK is invisible behind a lowest-common-denominator interface. The moment you need those, you either widen the interface (and leak the vendor back in) or bypass it.
• Single-provider depth is often the right call. If you will realistically never switch — you're all-in on one platform, the model's unique capabilities are the product, or the switching cost is irrelevant to your scale — then this layer is pure overhead. Use the vendor SDK directly and go deep.
• The interface is intentionally thin. Streaming, tool/function calling, multi-modal input, and token-level controls are out of scope here. A production version grows these — and every addition is a place where providers disagree and the abstraction leaks.
• Portability is not free. You pay in lost capability and in maintaining the seam. Adopt it when avoiding lock-in on the serving path is worth more than the deepest features of any single provider — not by default.

Development

npm install
npm test        # node:test via tsx — offline, no keys, no network
npm run typecheck
npm run lint    # Biome

Tests cover the mock provider and smoke-test happy path, createRouter env/opts selection (including the unknown-backend error), request/response shapes, and the OpenAI-compatible provider against a mocked global fetch (no real network) — verifying both the outgoing request body and response parsing.

License

MIT — see LICENSE.