Set TRITON_CACHE_DIR in the Jetson 6.2.0 image by JeremyGracey-AI · Pull Request #3072 · roboflow/inference · GitHub
Skip to content

Set TRITON_CACHE_DIR in the Jetson 6.2.0 image - #3072

Merged
grzegorz-roboflow merged 4 commits into
roboflow:mainfrom
JeremyGracey-AI:fix-jetson-6.2.0-triton-cache-dir
Oct 2, 2026
Merged

grzegorz-roboflow merged 4 commits into
roboflow:mainfrom
JeremyGracey-AI:fix-jetson-6.2.0-triton-cache-dir

Conversation

@JeremyGracey-AI

@JeremyGracey-AI JeremyGracey-AI commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Set TRITON_CACHE_DIR in the Jetson 6.2.0 image

What

Adds TRITON_CACHE_DIR=/tmp/triton-cache to the runtime ENV block of
docker/dockerfiles/Dockerfile.onnx.jetson.6.2.0, next to the existing
ORT_TENSORRT_ENGINE_CACHE_PATH=/tmp/ort_cache. One line, plus a unit test in
tests/inference/unit_tests/deployment/test_jetson_triton_cache_dir.py (same style as
test_model_trust_defaults.py): every Jetson image that ships Triton and does not pin RF-DETR to the base
preprocessor must set TRITON_CACHE_DIR=/tmp/triton-cache. It fails on main (naming 6.2.0) and passes on this
branch.

Why

RF-DETR preprocessing JIT-compiles Triton kernels
(inference_models/inference_models/models/rfdetr/triton_preprocess.py). With no TRITON_CACHE_DIR, Triton's
cache manager creates ~/.triton/cache, i.e. /root/.triton/cache. The
Jetson install docs
run the container with --read-only and only /tmp mounted writable, so every RF-DETR request
fails with HTTP 500:

OSError: [Errno 30] Read-only file system: '/root/.triton/cache'
  .../rfdetr/triton_preprocess.py -> triton/runtime/driver.py -> triton/runtime/cache.py:69 os.makedirs

YOLOv8 models on the same server are unaffected (no Triton path). The JetPack 7.2.0 image already sets
TRITON_CACHE_DIR=/tmp/triton-cache (Dockerfile.onnx.jetson.7.2.0:593, asserted by
docker/scripts/verify_jetson_server_runtime.sh:9); this brings 6.2.0 in line with the same value.

How it was tested

  • Device: Jetson Orin Nano 8 GB (Super), JetPack 6.2.1 (L4T R36.4.7).

  • Image: published roboflow/roboflow-inference-server-jetson-6.2.0:latest (server reports Inference 1.7.2),
    started with the documented hardened command (--read-only, /tmp volume, --cap-drop=ALL,
    no-new-privileges).

  • Before: rfdetr-nano, rfdetr-base, rfdetr-medium -> HTTP 500 with the trace above; yolov8n-640 -> OK.

  • After: same image and command plus -e TRITON_CACHE_DIR=/tmp/triton-cache at runtime (equivalent to the
    ENV line in this PR), no other Triton variable set, starting from an empty cache -> all three RF-DETR sizes return
    predictions (person 0.90-0.96 on two 1280x720 test frames). /tmp/triton-cache did not exist before the first
    request and held 24 compiled files afterwards, so Triton compiled into exactly the path this PR sets.

  • Commands (the -e MAX_ACTIVE_MODELS, ACTIVE_LEARNING_ENABLED and TELEMETRY_OPT_OUT flags were also set
    and are unrelated to this change):

    sudo docker run -d --name inference-server --runtime nvidia --read-only -p 127.0.0.1:9001:9001 \
      --volume /opt/nvme/inference-cache:/tmp:rw --env-file ~/.roboflow.env \
      -e TRITON_CACHE_DIR=/tmp/triton-cache \
      -e MAX_ACTIVE_MODELS=2 -e ACTIVE_LEARNING_ENABLED=False -e TELEMETRY_OPT_OUT=True \
      --security-opt=no-new-privileges --cap-drop=ALL --cap-add=NET_BIND_SERVICE \
      roboflow/roboflow-inference-server-jetson-6.2.0:latest
    # run once per model_id in rfdetr-nano, rfdetr-base, rfdetr-medium, on two 1280x720 JPEG frames
    import base64, json, os, urllib.request
    body = json.dumps({"model_id": "rfdetr-base", "api_key": os.environ["ROBOFLOW_API_KEY"], "confidence": 0.4,
                       "image": {"type": "base64", "value": base64.b64encode(open("frame.jpg", "rb").read()).decode()}}).encode()
    req = urllib.request.Request("http://127.0.0.1:9001/infer/object_detection", data=body,
                                 headers={"Content-Type": "application/json"})
    print(json.load(urllib.request.urlopen(req))["predictions"])
  • Unit test: pytest tests/inference/unit_tests/deployment/test_jetson_triton_cache_dir.py -> 1 passed
    (and 1 failed against main's Dockerfile). black --check and isort --check-only pass on the new file.

  • Not done: rebuilding the 6.2.0 image from this Dockerfile (Triton is compiled from source; not feasible on the
    test device). The verification uses the published image with the variable set at runtime.

Related (not in this PR)

By code inspection only (not tested on x86 GPU hardware): Dockerfile.onnx.gpu and Dockerfile.onnx.cu13.gpu ship the
Triton JIT toolchain for RF-DETR preprocessing but do not set TRITON_CACHE_DIR, and the Linux GPU install docs also
use --read-only with a /tmp volume, so they may hit the same error. Happy to follow up after testing on a GPU host.

On an 8 GB Orin Nano, loading several models with the default MAX_ACTIVE_MODELS produced
CUBLAS_STATUS_ALLOC_FAILED (nvmap ENOMEM); -e MAX_ACTIVE_MODELS=2 avoided it. Happy to open a separate docs PR
for an 8 GB note if that is useful.


Investigation and draft prepared with Claude Code; tested on-device and reviewed by me.

RF-DETR preprocessing JIT-compiles Triton kernels. Without TRITON_CACHE_DIR,
Triton writes its cache to /root/.triton/cache, which is read-only when the
container runs with --read-only as the Jetson install docs recommend, so
every RF-DETR request fails with OSError: [Errno 30]. The 7.2.0 image already
sets TRITON_CACHE_DIR=/tmp/triton-cache; use the same value here, next to the
existing ORT TensorRT cache under /tmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Sep 28, 2026 •

Copy link
Copy Markdown

JeremyGracey-AI and others added 3 commits September 27, 2026 21:46
Jetson images that ship Triton and do not pin RF-DETR to the base
preprocessor must point Triton's cache at /tmp, because the documented
container runs --read-only. Same style as test_model_trust_defaults.py.
Fails on main (Dockerfile.onnx.jetson.6.2.0), passes with this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@grzegorz-roboflow

Copy link
Copy Markdown
Collaborator

Hi @JeremyGracey-AI , thank you for this contribution! I'm merging it to our main and will be part of today's release! Let me know if you plan to provide corresponding fix to other dockerfiles :)

@grzegorz-roboflow
grzegorz-roboflow merged commit cca67a9 into roboflow:main Oct 2, 2026
5 checks passed
@JeremyGracey-AI

Copy link
Copy Markdown
Contributor Author

Thanks @grzegorz-roboflow, and thanks for the quick merge. Yes, I'll take the other Dockerfiles.

Scope

  • Dockerfile.onnx.gpu and Dockerfile.onnx.cu13.gpu are the two x86 images that ship the Triton JIT toolchain (both run verify_triton_jit_toolchain.py at build). Neither sets TRITON_CACHE_DIR, so they get the same line with the same value, /tmp/triton-cache.
  • Dockerfile.onnx.jetson.7.2.0 already sets it, and jetson.5.1.1 pins RF-DETR to the base preprocessor, so neither needs a change.
  • gpu.slim and gpu.parallel also install triton through torch but skip the toolchain check. I'll confirm whether they can reach this path and include them if they can.

Who it affects (from reading the code, not from a GPU test yet)

  • inference server start looks safe: it passes HOME=/tmp/home, so Triton's default ~/.triton/cache lands on the writable /tmp mount.
  • The documented docker run --read-only command leaves HOME=/root, the same failure as on the Jetson.

Plan

  1. Reproduce on an x86 GPU host with the documented command, then confirm with -e TRITON_CACHE_DIR=/tmp/triton-cache, the same method as this PR.
  2. Add the ENV line to both Dockerfiles and extend test_jetson_triton_cache_dir.py to cover every image that runs the Triton toolchain check.
  3. Open the PR next week with before/after output.

One limit: my GPU is an RTX 5070 Ti, so I can run the cu13 image end to end. The cu12.4 image ships torch 2.6.0+cu124, which predates 50-series support, so I'll test that one on an older rented GPU or mark it untested in the PR.

One question: classify_triton_jit_failure returns None for this OSError, so an unwritable cache dir raises a 500 instead of falling back to the reference path. Do you want a separate small PR that makes it recoverable, or should it keep failing loudly?

Code inspection done with Claude Code and reviewed by me. Hardware testing comes before the PR.

@JeremyGracey-AI
JeremyGracey-AI deleted the fix-jetson-6.2.0-triton-cache-dir branch October 2, 2026 16:17
JeremyGracey-AI added a commit to JeremyGracey-AI/JeremyGracey-AI that referenced this pull request Oct 2, 2026
JeremyGracey-AI added a commit to JeremyGracey-AI/prevera-guardian-lidar that referenced this pull request Oct 2, 2026
@JeremyGracey-AI

Copy link
Copy Markdown
Contributor Author

grzegorz-roboflow pushed a commit that referenced this pull request Oct 9, 2026
RF-DETR's Triton preprocessing creates ~/.triton on its first kernel
launch. Under the documented --read-only docker run command that path
is read-only, so every RF-DETR request returns HTTP 500. Point the
cache at /tmp, as #3072 did for the Jetson 6.2.0 image.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants