Skip to main content
Get started
Rehearse AI agents on a simulated customer world
sandtable is an open-source CLI that rehearses AI agents with simulated callers and stateful mock tools, then grades every trial deterministically into pass^k.
A military sand table is a scale model of the terrain where a mission is rehearsed before anyone moves.
sandtable turns a customer’s operating procedure, API and records into a τ-bench-style simulation. Simulated customers call your agent. Its tool calls hit stateful mocks of the customer’s systems. Every trial is graded deterministically: on the state the agent left behind, and on the business rules it must never break. You get pass^k (does it succeed every time?), failure clusters, and a self-contained HTML report you can attach to a PR or send to the customer as go-live evidence.
What it catches
These come from the bundled example, a fictional freight brokerage whose agent books loads for trucking carriers.Rule breaks that sound fine
The agent books a carrier whose insurance has expired. The transcript reads well; the
eligible-carrier invariant fails the trial anyway.Leaks under pressure
A caller pushes back and the agent says “the best I can do is $1,450”, which is the load’s internal ceiling.
never_disclose catches it.Intermittent failures
An agent can pass a task on one try and fail it on the next. pass^k counts a task only when every trial passes, so the gap shows up before customers see it.
Why
Agents that pass a demo fail on the seventh customer, because each customer has its own rules, systems and edge cases. The usual test setups miss this:- LLM-as-judge scoring reads the transcript, which sounds fine, and misses that the agent booked a carrier whose insurance had expired. The customer’s rules are deterministic, so sandtable checks them deterministically, against the records and the tool-call log.
- Success rate hides unreliability. An agent that is right 80% of the time fails one call in five in production. sandtable runs each task k times and reports pass^k, the probability that all k trials succeed.
- Benchmarks don’t know your customer. Public benchmark domains are useful for comparing models, but you need a benchmark of this customer’s workflow, kept in their repo and run on every prompt or model change.
Who it is for
Engineers who deploy agents into customers’ workflows and need evidence that the agent follows that customer’s rules: before go-live, and again after every prompt, model or tool change. It works with agents built on any stack, as long as they can be called over HTTP or wrapped in a small JavaScript or TypeScript module. It can also run Claude, models on Amazon Bedrock, or any model behind an OpenAI-compatible endpoint with the same prompt and tools, so you can compare models on the customer’s own workflow. If you work in Claude Code, the sandtable plugin lets your coding agent draft, lint and run a domain for you, and triage the failures.Start here
Try it
Run the offline demo, then the same example with Claude, then against your own agent.
Build a domain
Draft one from an OpenAPI spec, an SOP and call transcripts, then review it.
Gate releases
Fail the build when pass^k drops below a threshold.