SD-AgentFoundry-3D

Embodied simulation of a VLM-driven agent in a 3D digital twin

🇯🇵 日本語版 README はこちらです → README.ja.md

This repository is the implementation of SD-AgentFoundry-3D described in the manuscript "Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments" (Ryuki Hyodo).

It is published not as a finished application, but as a minimal foundation meant to be modified. It prioritises making the perceive → reason → act → observe loop readable and traceable over covering many features. Everything runs locally (macOS / Windows / Linux), and the VLM runs on a local Ollama server.

Because of that, some parts are deliberately left unpolished, some parts lack flexibility, and interpreting free-form text can fail. See Extending this repository for details.

simulation.mp4

Where this fits

SD-AgentFoundry consists of two minimal foundations at different levels of abstraction.

	SD-AgentFoundry-2D	This repository (3D)
Driving model	LLM	VLM (vision-language model)
Environment	2D integer grid (a highly transparent environment)	3D digital twin in USD (JAXA's Kibo module)
Input	Numeric state + messages from neighbours (text)	First-person RGB image + dynamically assembled text
Number of agents	Many (20 by default)	One
Response format	JSON (conservative fallback parsing)	Free-form text (read by a deterministic interpreter)
Focus	Communication and collective behaviour	Visual grounding and embodied action

A 2D environment captures communication and social organisation efficiently, but it abstracts away much of the spatial and perceptual structure that physical action carries. This repository complements it by grounding decisions in images.

The intended users are students, researchers, and practitioners. The point is to build your own scenarios, agent designs, measurements, and interfaces on top of this, which is why the machinery is left simple.

How to interpret the output

The output of this simulation is not a validated prediction of human behaviour.

• It is a tool for generating hypotheses. It can be used for exploratory purposes such as navigation, facility usage, human-robot interaction, and reviewing motion on a digital twin, but backing up any claim about the real world requires separate empirical validation
• The runs and figures included here are qualitative usage examples, not controlled evaluations, evidence of task completion, or performance benchmarks
• Descriptions of objects, materials, and situations that appear in the model's replies are generated by the model, not the state of the simulator. The simulator holds only pose, geometry, and camera settings. Keep this in mind when reading the Reasoning panel on screen and raw_text in run.jsonl
• What this repository implements is a VLM-driven, VLA-like perception–decision–action loop, not a trained VLA policy. Actions are interpreted from natural language; they are not learned as a policy
• There is no physics (pure kinematics). Contact, grasping, and forces are not handled

Overview

This is a simulation foundation that places a single walking agent inside a digital twin in USD format, shows its first-person image to a local open VLM, and lets the model decide the next action.

The bundled demo runs inside the Japanese Experiment Module "Kibo" on the ISS, with the task Find JAXA logo and get close to it.

When you run it, a window opens showing an overview of the whole space (a translucent dollhouse view, where the agent's position and trajectory show through the module) with the agent's field of view as an inset in the lower right. What the inset shows is exactly the image passed to the VLM.

What makes this demo different

It does not require a heavyweight simulation platform. USD is loaded as geometry data rather than as something to render (via usd-core from pip), and drawing is done with VTK. Setting up takes nothing but pip install, and it is not picky about the machine.

It does not enforce an output format. The VLM is given no JSON schema and no tool use. The free-form text that comes back is interpreted as an action directly.

Model reply       "The yellow box is ahead. I'll move forward 75 centimeters."
                     ↓
Simulator's read   distance 0.75 m / bearing +0 deg   ← the unit is not even metres

Removing the schema does not remove the simulator's need for numbers, so the responsibility shifts from "the model emits valid JSON" to "the simulator reads natural language". When it cannot read the reply, it does not fill in anything on its own: it records that the reply could not be read, and the agent does not move. Whether a model can give executable instructions in free-form text is exactly that model's ability, so papering over that would destroy the one thing this demo can measure.

Inference time is never added to simulation time. See Time model for details.

Repository layout

The only things you touch are config.yaml and python main.py at the repository root.

.
├── config.yaml              # Simulation settings (the task prompt lives here too)
├── main.py                  # Entry point
├── src/vlmsim/              # The simulation itself
│   ├── cli.py               # Argument parsing, logging setup, dispatch
│   ├── simulation.py        # Run loop (observe → infer → interpret → apply → record)
│   ├── agent.py             # Pose and movement integration (pure kinematics)
│   ├── clock.py             # Time model (keeps inference time separate)
│   ├── prompt.py            # Prompt assembly
│   ├── interpreter.py       # Free-form text → action
│   ├── motion.py            # Action representation and clamping to limits
│   ├── recorder.py          # run.jsonl, first-person images, config snapshot
│   ├── replay.py            # Re-renders the live screen from the recording
│   ├── usd_scene.py         # USD → world-space triangle mesh (the only place with frame conversions)
│   ├── usd_gprims.py        # Tessellation of Cube/Sphere and friends
│   ├── render/              # The only place allowed to import VTK/PyVista
│   │   ├── viewer.py        # The live window (overview + inset + HUD)
│   │   ├── ego_camera.py    # The first-person image handed to the VLM
│   │   ├── scene_actors.py  # Scene render actors
│   │   ├── agent_figure.py  # Humanoid figure (overview view only)
│   │   └── hud.py           # Text and layout of the information panel
│   └── vlm/                 # Where models are swapped in
│       ├── ollama.py        # Default
│       ├── openai_compatible.py
│       ├── transformers_local.py
│       └── scripted.py      # No model needed (for tests and smoke checks)
├── assets/
│   ├── kibou/               # Kibo digital twin (bundled, 69 MB)
│   └── sample_room/         # Procedurally generated minimal scene (3.4 KB)
├── visualization/           # Standalone tools for reviewing a run afterwards
│   ├── viewer.html          # Browser viewer
│   └── generate_video.py    # Builds an MP4 from a recording
├── tests/                   # Pass without a model, a GPU, or a network
└── output/                  # Run results (1 directory = 1 run)

Modules outside render/ do not import VTK / PyVista. This guarantees that it runs headless, that tests stay light, and that drawing cannot influence simulation results; tests/test_layering.py checks it mechanically.

Requirements

• Python 3.10 or later (3.12 recommended)
• Ollama (local VLM server)
• The Python packages listed in requirements.txt
• FFmpeg (only if you generate videos)

A GPU is not required (drawing takes about 1 ms/frame at 1280×720). VLM inference speed, however, depends on your machine.

Setup

1. Create and set up the virtual environment

macOS / Linux

# Run the setup script (recommended)
chmod +x setup_mac.sh
./setup_mac.sh

# Or set it up by hand
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -e .

Windows

REM Run the setup script (recommended)
setup_win.bat

REM Or set it up by hand
python -m venv venv
venv\Scripts\activate.bat
python -m pip install --upgrade pip
pip install -e .

Use pip install -e ., not pip install -r requirements.txt. The implementation lives in src/vlmsim/, so python main.py cannot import it unless the package is installed. requirements.txt remains the single source of truth for the dependencies themselves.

2. Activate the virtual environment

source venv/bin/activate        # macOS / Linux
venv\Scripts\activate.bat       # Windows

3. Set up Ollama

Install Ollama

• Install Ollama.

Start the Ollama server

It normally starts as a background service when installed, but to start it by hand:

ollama serve

By default it listens on http://localhost:11434. To check that it is up:

curl http://localhost:11434/api/tags

Download and choose a model

⚠️ A vision-capable model is required. When an image is passed to a text-only model, Ollama does not raise an error: it silently ignores the image, so the run continues with "the agent answering without looking at anything" and you never notice. This simulator checks capabilities at startup and stops with a clear error if the model has no vision support.

# List the models you have already downloaded
ollama list

# Pull a model (example)
ollama pull <model name>

Model confirmed to work: with qwen3.6:27b, the agent looks around the module and approaches the logo well enough for the demo to hold together.

We recommend trying models yourself. The right choice depends on your machine (RAM / VRAM) and on how far model development has come at the time you read this. Recommending a specific model here would go stale quickly, so no list is given on purpose. Check which vision-capable models you have with ollama list and try a few. Because this demo hinges on interpreting free-form text, behaviour differs between models. That difference is itself the thing to observe.

How to specify the model name: the surest way is to copy the value in the NAME column of ollama list into vlm.model in config.yaml (:latest may be omitted).

vlm:
  model: "qwen3.6:27b"    # Change this to a model name from ollama list

4. The demo scene (Kibo)

Nothing needs to be downloaded. The USD of the Japanese Experiment Module "Kibo" on the ISS is bundled in assets/kibou/ (179 files, 69 MB). The commands in this README work as soon as you clone the repository.

assets/kibou/
├── KIBOU.usd          The stage root. This is what config.yaml points at
├── Kibou/             The module itself (geometry, materials, textures)
└── Intball2/          Int-Ball2, the small robot that floats inside the module

The maps that are not used for drawing (normal / roughness / metallic) and the FBX files have been dropped, so this comes to 40% of the original asset. The materials still reference them, but this renderer only resolves diffuseColor, so it makes no difference.

If you want to try a lighter scene, the bundled procedurally generated scene (3.4 KB) also works:

python main.py --usd assets/sample_room/sample_room.usda

5. Review the configuration file

Open config.yaml and adjust it. These two in particular:

• vlm.model — the name of a vision-capable model on your machine (check with ollama list)
• task.prompt — the task given to the agent. This is the main thing to work with in this demo

Usage

Note: before running, activate the virtual environment and make sure the Ollama server is up.

Basic run

python main.py --usd assets/kibou/KIBOU.usd --model qwen3.6:27b --duration 10 --replay

A window opens showing the overview view and the inset with the agent's field of view. Afterwards you can open visualization/viewer.html, choose output/, and review the run.

Flag	Meaning
--usd assets/kibou/KIBOU.usd	The demo scene (Kibo). Already bundled. For a lighter scene, assets/sample_room/sample_room.usda
--model qwen3.6:27b	The vision model to use (check with ollama list)
--duration 10	Stop after 10 decisions (the config default is 40)
--replay	Also build the frames viewer.html needs when the run ends

All of these can be written in config.yaml and omitted. The command above merely spells them out so that one line shows where everything is decided.

💡 Tip for your first run: the VLM is called once per step, so the larger the model, the longer it takes. Start with a light model and a small --duration (say 5) before scaling up.

About --replay. What the browser viewer plays is not a screen recording but a re-render from the log. It therefore does not exist yet when a run ends, which is why generating it is a separate step. --replay does that step immediately after the run. Without it, the command to run next is printed when the run ends.

One frame is drawn per world sub-step at the window resolution used during the run, so the default of 40 decisions produces 400 frames and several hundred MB. That is why it is not done by default (and why the command above uses --duration 10, which keeps it to 100 frames).

Headless run

On WSL, over SSH, or on a machine with no GUI, the window that --visualize would open cannot be created. Use --headless instead (recording works as usual).

python main.py --headless

Inspecting a USD file

Run this first when you bring in your own USD. It prints the mesh count, triangle count, bounding box, and the list of UV sets.

python main.py --inspect-usd assets/kibou/KIBOU.usd

Main options

Option	Description
-c, --config	Path to the configuration file (default: config.yaml)
--inspect-usd [PATH]	Load a USD, print what is in it, and exit
--usd PATH	Swap in a different digital twin USD
--model NAME	Swap in a different VLM model name
--backend NAME	ollama / openai_compatible / transformers / scripted
--duration N	Number of decisions
--seed N	Random seed
--headless	Do not open a window
--save-frames / --no-save-frames	Whether to save the first-person images
--output-dir DIR	Output directory
--replay	Generate the frames for viewer.html when the run ends

Command-line options take precedence over config.yaml.

Running without a model

The scripted backend returns a prepared sequence of replies. It lets you check that the scene, the start pose, and the window are right without waiting for inference.

python main.py --backend scripted --duration 10 --replay

Reading the screen

Element	Content
Full window	The overview view. The space is drawn translucent (dollhouse view), so the humanoid agent walking inside the module and its trajectory show through
Inset, lower right	The agent's first-person view. This is exactly the image passed to the VLM (no markers, no HUD)
Panel, upper left	State, pose, time, plus Reasoning (the model's reply verbatim) and how it was interpreted
Chip, upper right	Parse status by colour: green ok / amber PARTIAL / red UNPARSED, with a running count of failed reads

What the upper-left panel contains:

sim 12.3 s   decision 12/40   [ok]                    ← sim time, progress, parse status
pos (-2.10, 20.13) m   heading 3 deg   gaze +0 deg    ← pose
walked 0.9 m   tick 3/10                              ← distance walked, sub-step within the decision
inference time 5.3 s   (not added to sim time)        ← wall-clock inference time (never added to sim time)
total inference 61.0 s   2.05x real time

Reasoning
step 12   move 1.00 m at bearing +0 deg               ← the action the simulator executed
  clamped to the movement limits                      ← only when the limits cut it down
  stopped by the scene bounds                         ← only when a wall stopped it

The JAXA logo is on the panel ahead and slightly      ← the model's reply, verbatim
to the right. I will move forward 1 m to get closer.

(space) pause   (f) follow agent   (q) quit

The point is that "what the model said" and "what the simulator understood" sit one above the other. When a reply cannot be read, the reason (parse_notes) appears in the Reasoning panel, so when your own model does not work you can see why on the spot.

task.prompt asks the model to give its reasoning first and put the action in one sentence at the end. Since interpreter.py is designed to take the last action expression that appears, making the model put its conclusion last aligns the two conventions. This is a plain request in the prompt, not a schema.

Control	Effect
Left drag / wheel	Orbit and zoom the overview camera
space	Pause / resume
f	Switch the overview camera between "whole scene ⇄ follow the agent (close up)"
q / Esc	Quit (everything recorded so far is kept)

The agent is drawn as a humanoid figure, and its facing is readable from the body direction, the face and sunglasses, the chest plate, and the leg swing while walking (no direction arrow is drawn). A 1.6 m figure is small in the whole-scene view, so the f key switches to a close follow camera.

If the overview is heavy, setting visualization.overview_opacity to 1.0 speeds it up considerably (but the agent is then hidden behind the outer shell).

Configuration file (config.yaml)

The main parameters (every key carries a comment stating its meaning and unit):

• simulation: simulation settings

  • duration: number of decisions
  • sim_dt: world update step [s]
  • decision_interval: decision period [s] (must be a multiple of sim_dt)
  • seed: random seed

• scene: the digital twin space

  • usd_path: the USD file (.usd / .usda / .usdc / .usdz)
  • exclude_prims: paths of prims to exclude from drawing
  • uv_set: the name of the UV set the textures use ("auto" is allowed)
  • load_textures: false draws flat colours (fast, but logos and the like disappear)

• agents: agent settings (a single agent)

  • start_x / start_y / start_yaw: initial position [m] and facing [deg]
  • eye_height: eye height [m] (default 1.6 = a standing adult)
  • pitch_min / pitch_max: gaze range [deg] (straight up and straight down are not reachable)
  • max_translation / max_turn / max_pitch_change: the maximum allowed per decision
  • wall_clearance: minimum distance kept from the bounds of the space [m]
  • memory_size: how many recent actions are included in the prompt

• camera: the first-person camera (= the image given to the VLM)

  • width / height [pixels], fov (vertical field of view [deg]), near / far [m]

• task: task settings

  • prompt: the instruction given to the agent. Rewriting this is the main way to use this demo

• vlm: VLM settings

  • backend: ollama / openai_compatible / transformers / scripted
  • model: model name (an Ollama model name, or a HuggingFace model id)
  • base_url: API endpoint
  • temperature / max_tokens / repeat_penalty / repeat_last_n / min_p
  • think: false suppresses thinking output (replies get shorter and easier to interpret)

• interpreter: free-form text interpretation settings

  • on_unparsed: hold (do not move) / repeat_last (repeat the previous action)
  • default_step / default_turn / default_pitch: defaults for words that carry no number
  • diagonal_bearing: what "diagonally" / "斜め" is taken to mean [deg]

• visualization: display settings

  • show_window, inset (position of the inset), overview_opacity, save_frames
  • show_agent_marker: whether to draw the humanoid agent
  • follow_agent: false for the whole scene, true to close in on the agent (f toggles it during a run)

Coordinate system

• Units: metres. The USD metersPerUnit is read and converted internally
• Up axis: +Z. Y-up USD files are rotated on load to match
• Agent position: the position of the feet (x, y). The camera sits eye_height above it
• Facing (yaw): 0 deg is +X, increasing towards +Y (counter-clockwise seen from above)
• Gaze (pitch): 0 deg is horizontal, positive is upwards
• Action bearing: relative to the agent itself. 0 is ahead, +90 is left, -90 is right

Using your own USD

You only replace scene.usd_path. Check the following, though.

1. First run python main.py --inspect-usd path/to/your.usd. Check that the bounding box is what you expect.
2. Exclude huge background objects with exclude_prims. For instance, the Kibo USD contains a sphere of the Earth about 1000 m across; including it shrinks the module to a dot.
3. If textures look wrong, change uv_set. A mesh may carry several UV sets (st / st1 / st2) while the material never declares which one to use. --inspect-usd lists the available UV sets.
4. Set agents.start_x / start_y inside the space. If they fall outside, a warning is printed and the run starts from the centre of the space.

Limits (because usd-core does not include Hydra / usdImaging):

USD feature	Supported
Geometry (Mesh, Cube/Sphere/Cylinder and friends, instances)	✅
displayColor, and the diffuse colour / diffuse texture of UsdPreviewSurface	✅
GeomSubset (several materials on one mesh)	✅
Normal / metallic / roughness maps	❌ Ignored (VTK's three-point lighting stands in)
MDL materials (Omniverse-specific)	❌ Ignored
USD lights, skeletal animation, physics	❌ Ignored

Output

Results are written to output/ (visualization.output_dir).

output/
├── frames/frame_0000.png ...   The first-person images passed to the VLM, exactly as given
├── run.jsonl                   1 line = 1 decision
├── config.snapshot.yaml        The settings actually used (for reproduction)
└── replay/                     Re-rendered frames built by the visualization tools (after the run)
simulation.log                  Run log

1 directory = 1 run. When a simulation starts, the files written by the previous run are deleted. Numbered files are not fully overwritten by a short run, so the tail of the previous run survives, and without this you end up with the previous run's video presented as the new one (which is what actually happened).

Only the files listed above are deleted; nothing else you put in output_dir is touched. If there is a run you want to keep, change visualization.output_dir per run.

One record of run.jsonl:

{
  "step": 3,
  "sim_time": 3.0,
  "pose_before": {"x": 1.20, "y": 0.40, "z": 0.0, "yaw": 25.0, "pitch": -5.0},
  "frame": "frames/frame_0003.png",
  "raw_text": "The yellow box is ahead. I'll move forward 75 centimeters.",
  "command": {"distance": 0.75, "bearing": 0.0, "yaw_delta": 0.0,
              "pitch_delta": 0.0, "vertical": 0.0},
  "parse_status": "ok",
  "parse_notes": ["'75 centimeters' -> 0.75 m", "'forward' -> bearing +0 deg"],
  "clamped": false,
  "blocked_by_bounds": false,
  "pose_after": {"x": 1.88, "y": 0.72, "z": 0.0, "yaw": 25.0, "pitch": -5.0},
  "inference_wall_ms": 12100
}

The point is that raw_text (what the model said) sits next to command / parse_notes (what the simulator understood). parse_status takes three values: ok, partial (filled in with a default), and unparsed (could not be read, so the agent did not move).

Visualization tools

visualization/ holds tools for reviewing a recorded run afterwards.

What they show is the live window itself. The overview view, the humanoid agent, the trajectory, the first-person inset, and the information panel in the upper left: everything you saw during the run comes out as it was. It is not a summary figure with a different layout.

This is not a screen recording but a re-render from the log. Given run.jsonl (the pose and action of each decision) and config.snapshot.yaml (the scene and display settings), the state sequence can be reconstructed, and it is drawn off-screen by the same Viewer used during the run. Because the implementation is not split in two, the images cannot diverge (tests/test_replay.py compares them pixel by pixel).

Two things are not reproduced.

• The pauses waiting for inference: that is the advantage of a replay. The recorded trajectory has no gaps, so laying the sub-steps out at a constant rate gives continuous motion with no waiting
• Viewpoints you moved with the mouse: mouse input is not recorded, so a replay always shows the automatic camera

Frames are written to <run>/replay/, and both the browser viewer and the MP4 share them. At 10 frames per decision, at the window resolution used during the run, they are not small (about 7 MB per decision at 2074×1342). If you only need the MP4, you can delete <run>/replay/ after generating it.

Browser viewer (viewer.html)

Generate the frames first, then open it (not needed if you already ran main.py --replay).

python visualization/generate_video.py output/ --frames-only
open visualization/viewer.html      # Choose the output/ directory

If it says replay/frames.json is missing, this command has not been run yet. The run directory is emptied for every new run, so if you re-run the simulation you have to re-generate the frames.

The composited frame is shown on top, with the model's raw reply and the interpretation below it (selectable and copyable).

Images are shown at native size by default (1 image pixel = 1 screen pixel). Stretching them to the full width of the browser invents 2–3 pixels for every real one, especially on a Retina display, and the HUD text is the first thing to smear even though the frame itself is sharp. If you want a larger picture, press ⤢ Fit width.

Action	How
Play / pause	The ▶ Play button, or Space
Previous / next frame	⏮ Prev / ⏭ Next, or ← →
Previous / next decision	Shift + ← →
Jump to a frame	The slider
Playback speed	The Speed slider (a multiplier, with 1.0 = real time)
Native size / fit to width	The ⤢ Fit width button (⤡ Native size to go back)

Chrome and Edge can select a whole folder at once.

Video generation (generate_video.py)

This collects the same frames into an MP4. FFmpeg is required.

brew install ffmpeg                 # macOS
sudo apt install ffmpeg             # Ubuntu / Debian

python visualization/generate_video.py output/
python visualization/generate_video.py output/ -o result.mp4 --width 1920 --height 1080

Option	Default	Description
data_dir (required)	—	Path to the run directory
-o, --output	simulation.mp4	Output MP4 file name
--fps	1 / sim_dt (10 by default)	At the default, playback runs at simulation speed
--width / --height	The window size used during the run	Override the resolution. Pass both together (see below)
--frames-only	—	Only build the frames, no MP4 (FFmpeg not needed)
--force	—	Re-render even if <run>/replay/ already holds frames

The resolution defaults to the window used during the run. The window size resolved at run time is recorded in config.snapshot.yaml, and the replay reads it (a --headless run had no window, so it becomes 1920×1080).

This is not only about sharpness. The HUD margins, the font size, and the field of view of the overview are all derived from the window size, so changing the aspect ratio changes the composition itself. Always pass --width and --height together (passing only one is an error).

One decision is split into decision_interval / sim_dt frames (10 by default). The default --fps is the reciprocal of that, so playing it back advances the world at the same speed as the run.

VLM agent design

Basic design

One decision proceeds in this order.

Render the first-person image  →  Assemble the prompt  →  Query the VLM
      →  Interpret the free-form text as an action  →  Clamp to the limits
      →  Integrate into the pose  →  Record

All the agent is given is the task prompt, the first-person image, its own pose, and its recent action history. It is given neither the position nor the distance of the target.

There is no physics engine (pure kinematics). Grasping and pushing are not handled; the only result of an action is that the agent's position and facing change.

Time model

Simulation time is determined solely by the number of decisions.

sim_time = number of decisions × decision_interval

However many seconds VLM inference takes, it is never added to simulation time. The time actually spent is only recorded as inference_wall_ms in the metadata. As a result, for the same scene, the same seed, and the same sequence of replies, the trajectory matches exactly whether the model answers in 20 ms or in 20 s.

The world is also updated in steps of sim_dt while the agent decides once every decision_interval (multi-rate). One decision is applied across several world updates, so movement in the overview view looks continuous.

About the window freezing during inference: the run loop is synchronous, so the screen is not updated while the model is thinking. But what stops is drawing time, not simulation time. No gap appears in the recorded trajectory. If you want smooth footage, build a replay with generate_video.py after the run: it gives you the same screen as the live run, with only the waiting for inference removed. The alternative of "advancing the world during inference and applying the action when it completes" is deliberately not used (that would be a different time model).

Action interpretation (no enforced schema)

The model is asked for no JSON, no schema, and no tool use. The following are read from the free-form text that comes back.

What is read	Examples
Distance and unit	2 m / 75 centimeters / 50cm / 1.5 メートル / 3 feet
Direction of travel	forward / back / left / right / 前 / 後ろ / 左 / 右
Diagonal	diagonally / 斜め
Body turn	turn right 30 degrees / 右に30度回る / turn around / 振り返る
Gaze	look up 30 degrees / 30度上を見る / 見上げる
Holding still	stay / wait / 止まる / そのまま

The interpreter accepts Japanese phrasings as well as English ones; the Japanese examples above are the literal wording it recognises.

Design points:

• Turns and gaze are read first, and those parts are removed from the text before movement is read. This prevents the right in turn right from being read a second time as "move right".
• The last expression that appears wins. A model that writes out its reasoning puts its conclusion at the end.
• Direction words with no verb are not executed. "The hatch is on the right" describes the situation rather than instructing anything, so nothing moves unless there is a verb or an explicit distance.
• When no number is given, the defaults are used and the step is recorded as partial (interpreter.default_*).
• If nothing can be read, unparsed is recorded and the agent does not move. Nothing is filled in.

Every step leaves "what was read and how" in parse_notes, so you can diagnose "why is it not moving" yourself, with your own model and your own prompt.

What is kept out of the first-person image

The agent marker, the trajectory, and the HUD drawn in the overview view never appear in the first-person image. Only scene geometry is registered with the first-person renderer, so the structure of the code guarantees that the input given to the VLM is not contaminated.

Tests

pytest

No model, no GPU, and no network are needed (the scripted backend is used). If a public repository's CI needed a model, contributing from outside would become effectively impossible.

The main tests:

• test_interpreter.py — free-form text interpretation (Japanese and English, different units, instructions inside prose, and situation descriptions that must not be executed)
• test_usd_scene.py — USD loading (unit conversion, Y-up conversion, gprim tessellation, traversing instances, excluded prims)
• test_agent.py — kinematics (bearing convention, order of turning and translating, staying inside the bounds, neutralising NaN)
• test_simulation_time.py — the invariant of the time model (the trajectory matches even when inference delay is injected)
• test_render_smoke.py — that the first-person image is not blank, and that two images taken from different poses do not match
• test_layering.py — a static check that nothing outside vlmsim/render/ imports VTK

Limitations

• The visuals are simple. Materials are read only as far as the diffuse colour, and USD lights and PBR maps are ignored (see the table in Using your own USD)
• There is no physics. Grasping, contact, and joints are not handled
• Collision is checked only against the bounds of the space. Furniture inside a room is passed straight through
• There is a single agent. There is no communication between agents

Extending this repository

This repository is built on the assumption that you will develop and extend it yourself. Because being minimal comes first, the following trade-offs are made.

• Some parts are deliberately left unpolished. The "Limitations" above are exactly those parts. Add what you need yourself
• Some parts lack flexibility. There are only three kinds of action — translation, turning, and gaze — and actions such as grasping, pushing, and speaking are not defined
• Interpreting free-form text can fail. That is unavoidable given the choice not to enforce a schema. Failures are not hidden: they are recorded as unparsed and the agent does not move. If you want more phrasings to be recognised, add an extractor to vlmsim/interpreter.py

There are three ways in. Each can be replaced without touching the other layers.

What you want to do	Where to work
Use a different model or a different inference method	Add a VlmBackend implementation under vlmsim/vlm/
Recognise more phrasings	The extractors in vlmsim/interpreter.py
Add more kinds of action	MotionCommand in vlmsim/motion.py and the integration in vlmsim/agent.py

Extending toward Physical AI

This repository uses a time model that separates inference time from simulation time and has no physics. If you want to extend it towards Physical AI, aimed at transfer to real hardware or at evaluating control performance, consider connecting it to Isaac Sim or ROS 2. You will mainly need the following.

• A physics engine — contact, grasping, joints. The kinematics here cannot handle them
• A time model that accounts for latency — a scheme that reflects inference latency, communication latency, and control latency in simulation time. This repository deliberately implements only the time model that keeps them separate
• Sensor / actuator abstractions — in a form that fits ROS 2 topics and services

Even then, vlmsim/interpreter.py (free-form text → action) and vlmsim/prompt.py (prompt assembly) do not depend on the simulator, so you can take them with you as they are.

Citation

When publishing work that uses this code, please refer to the following.

Ryuki Hyodo. Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments.

http://arxiv.org/abs/2608.22833 (doi.org/10.48550/arXiv.2608.22833)

For the 2D, LLM-driven version, see SD-AgentFoundry-2D.

License

Copyright 2026 Dr. Ryuki HYODO

This software is released under the Apache License 2.0. See LICENSE.txt for details.