[Task Proposal] SuperTuxKart race-telemetry reconstruction

Family: agentic_vbench_understanding · Proposed task id: kart-race-telemetry-ledger-s1

Branch pengchx-kart-race-telemetry (this task only, off main): https://github.com/explcre/agentic-vbench/tree/pengchx-kart-race-telemetry. Generator bundled at tasks/agentic_vbench_understanding/kart-race-telemetry-ledger-s1/generator/.

Two revisions after review.

1. Observability fix (the reviewer's main point). STK's profile camera is a chase-cam locked to one kart, so the original "count all twelve karts" framing scored eleven karts whose pickups are off camera — capping any real agent below the oracle. The task now scopes the scored counts to the single camera-followed hero kart (--kart=tux --ai=<rest>), so every scored event is on screen by construction.
2. Hardened within that fair frame. Scoping to one kart made it easy, so this version raises the bar: 12 races × 4 laps (~53 min), and three independent off-HUD counts instead of two. Details below.

Is it hard enough / long-horizon?

A ~53-minute video of twelve AI-driven SuperTuxKart races on twelve tracks (SuperTux, the
hardest AI). The camera follows one hero kart (tux) the whole suite. For each race the agent
recovers three counts for the hero: how many powerup boxes it drove through, how many times
it was blown up, and how many bananas it hit. None is shown as a number anywhere, so
recovering them means following the hero through every lap of every race and tallying three kinds
of discrete event, then ordering the twelve races by each tally. Tracking three independent counts
across a dozen races over ~53 minutes is where the difficulty lives; a single frame or a schema
guess scores ~0.

Cognitive level

understanding — fine-grained event counting on a tracked subject across a long horizon, then a
cross-race ranking per quantity. Not a lookup; not readable from any on-screen counter.

Observability

• Camera model. SuperTuxKart profile mode renders a chase-cam bound to the player kart; the
  generator makes the hero the player, so the camera is locked to tux for all twelve races
  (verified frame-by-frame: tux is the centred kart with its name on the plate, grid to finish).
• Every scored event is on camera. A powerup box is a question-mark box the hero drives
  through (the powerup then appears in its HUD slot); an explosion throws the hero into the air;
  a banana visibly spins it out. All three are large, centred and unambiguous at 720p. The other
  nine karts still race (contesting boxes, bombing the hero), so it is a real crowded race — they
  are simply not what is counted.

Question & output schema

For each race, report the hero's items_collected, times_exploded, bananas_hit, and skid_time (total drift seconds), in race order.

{ "races": [ {"track":"hacienda","items_collected":14,"times_exploded":5,"bananas_hit":1,"skid_time":69} ] }

track, nitro_collected, and positions may be reported for context but are not scored.

Harder variant shipped: the top-center HUD powerup indicator is masked (a black box), so an agent cannot read the pickup off the HUD — it must catch each item from the hero visibly driving through a question-mark box. Still fair (boxes are on-track); strictly harder for the items count.

Ground truth

• source: SuperTuxKart profile mode (--profile-laps) result table — the engine AI drives
  every kart and prints exact per-kart telemetry, so the recorded run is its own ground truth.
  The hero's row — items, explosions, bananas, and skid_time (total drift seconds) — is what is scored.
• tier: machine-truth.
• verification: oracle solution.json = the hero's per-race counts; judge.py scores it
  1.0 through the harness path (12 races; items τ=1.0 over 59 pairs, explosions τ=1.0 over 52,
  bananas τ=1.0 over 49). build_ground_truth.py asserts the hero is present in every race and the
  primary field varies, so a degenerate suite cannot ship.

Scorer (deterministic)

Scored EXACTLY (not just ranked), against STK's full-precision telemetry. Per quantity q:
score_q = clamp(τ_q, 0, 1) · accuracy_q, where accuracy_q = mean over races of max(0, 1 − |pred−gt| / max(1, 0.30·gt)) — you must be within ~30% of the true value — and τ_q is the signed Kendall correlation over the races (the guess gate). reward = 0.30·items + 0.15·explosions + 0.25·bananas + 0.30·skid_time, renormalised over quantities that vary. Rank alone was too forgiving (an agent that systematically under-counts still ranks races right and scored ~0.35); requiring accuracy removes that free credit, and the τ-gate keeps blind guessing at ~0.

Measured on the shipped suite: oracle 1.0, constant-guess 0.0, blind-guess mean 0.019
(p95 0.26), perfect-items-only 0.40, perfect-items+explosions 0.70 — i.e. a one-quantity
agent caps at 0.40 and a two-of-three agent at 0.70; you must count all three to score high.

Honest difficulty

This is a fair, medium entry — the same honest standing as the family's practice (the sibling
Minecraft ordered-ledger sits at 0.164; strict <0.10 is not reached by any counting task here). The
hardening (12 races, ~53 min, three counts, tighter guess floor) lowers a real agent's achievable
score without breaking fairness. A genuinely harder ordered-ledger variant would need machine-exact
per-event order; STK's prebuilt binary exposes only per-race counts (no headless race-replay
recording, and the replay format logs no pickup/explosion events), so counts are the ceiling for
this engine. Strong-agent calibration (Codex gpt-5.6-sol, xhigh): measured 0.0295 — MEETS the family's <0.10 bar (completed 4-quantity run, 542 tool calls, all 12 races' drift-times reported; rollout calibration/rollouts/codex_exact_full_*). Per-quantity: items 0.015 (accuracy 0.043 — essentially never within 30%; it undercounts ~50%), explosions 0.103, bananas 0.0, skid_time 0.032 (drift-duration barely ranked). Requiring accurate counts plus a duration defeats the agent even with meticulous per-race vision analysis. Trajectory: 0.407 (rank, 3 counts) → 0.345 (HUD mask) → 0.0295 (exact-count + drift-time). Honest n=1 caveat: an earlier partial re-score (drift unreported) gave ~0.12; both put it at/below <0.10. Still fair + learnable — oracle 1.0, and a within-30% agent scores far higher.

Input media

• url: https://huggingface.co/datasets/explcre/agenticvbench-understanding-materials/resolve/main/kart-race-telemetry-ledger-s1/race.mp4
• sha256: a84170a13f0d8392e18a0fd0535e5ddad7ceae4888d5c826b6b77850f78bf322, pinned in environment/Dockerfile.
• length: 53.4 min. resolution: 1280×720. audio: none.

Programmatic large-scale generation

generator/run_suite.sh renders a parameterised suite (tracks, hero, field, laps, race count all
knobs) and emits video + machine-exact ground truth in one command, zero annotation. Races render
in parallel (per-display) for speed. The space of fresh, machine-labeled instances is effectively
unlimited, so the benchmark can mint new instances at any length/difficulty and re-generate if a
clip leaks.

Reproducibility

generator/ (SuperTuxKart 1.5 --profile-laps --kart=tux, Xvfb + software GL + x11grab):
run_suite.sh → per-race run_race.sh + parse_profile.py → build_ground_truth.py. No manual
annotation.