Repository navigation
Set TRITON_CACHE_DIR in the Jetson 6.2.0 image - #3072
grzegorz-roboflow merged 4 commits into
Conversation
RF-DETR preprocessing JIT-compiles Triton kernels. Without TRITON_CACHE_DIR, Triton writes its cache to /root/.triton/cache, which is read-only when the container runs with --read-only as the Jetson install docs recommend, so every RF-DETR request fails with OSError: [Errno 30]. The 7.2.0 image already sets TRITON_CACHE_DIR=/tmp/triton-cache; use the same value here, next to the existing ORT TensorRT cache under /tmp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jetson images that ship Triton and do not pin RF-DETR to the base preprocessor must point Triton's cache at /tmp, because the documented container runs --read-only. Same style as test_model_trust_defaults.py. Fails on main (Dockerfile.onnx.jetson.6.2.0), passes with this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hi @JeremyGracey-AI , thank you for this contribution! I'm merging it to our main and will be part of today's release! Let me know if you plan to provide corresponding fix to other dockerfiles :) |
|
Thanks @grzegorz-roboflow, and thanks for the quick merge. Yes, I'll take the other Dockerfiles. Scope
Who it affects (from reading the code, not from a GPU test yet)
Plan
One limit: my GPU is an RTX 5070 Ti, so I can run the cu13 image end to end. The cu12.4 image ships torch 2.6.0+cu124, which predates 50-series support, so I'll test that one on an older rented GPU or mark it untested in the PR. One question: Code inspection done with Claude Code and reviewed by me. Hardware testing comes before the PR. |
docs: roboflow/inference#3072 merged by Roboflow; test count 281 on main
RF-DETR's Triton preprocessing creates ~/.triton on its first kernel launch. Under the documented --read-only docker run command that path is read-only, so every RF-DETR request returns HTTP 500. Point the cache at /tmp, as #3072 did for the Jetson 6.2.0 image.

Set
TRITON_CACHE_DIRin the Jetson 6.2.0 imageWhat
Adds
TRITON_CACHE_DIR=/tmp/triton-cacheto the runtimeENVblock ofdocker/dockerfiles/Dockerfile.onnx.jetson.6.2.0, next to the existingORT_TENSORRT_ENGINE_CACHE_PATH=/tmp/ort_cache. One line, plus a unit test intests/inference/unit_tests/deployment/test_jetson_triton_cache_dir.py(same style astest_model_trust_defaults.py): every Jetson image that ships Triton and does not pin RF-DETR to thebasepreprocessor must set
TRITON_CACHE_DIR=/tmp/triton-cache. It fails onmain(naming 6.2.0) and passes on thisbranch.
Why
RF-DETR preprocessing JIT-compiles Triton kernels
(
inference_models/inference_models/models/rfdetr/triton_preprocess.py). With noTRITON_CACHE_DIR, Triton'scache manager creates
~/.triton/cache, i.e./root/.triton/cache. TheJetson install docs
run the container with
--read-onlyand only/tmpmounted writable, so every RF-DETR requestfails with HTTP 500:
YOLOv8 models on the same server are unaffected (no Triton path). The JetPack 7.2.0 image already sets
TRITON_CACHE_DIR=/tmp/triton-cache(Dockerfile.onnx.jetson.7.2.0:593, asserted bydocker/scripts/verify_jetson_server_runtime.sh:9); this brings 6.2.0 in line with the same value.How it was tested
Device: Jetson Orin Nano 8 GB (Super), JetPack 6.2.1 (L4T R36.4.7).
Image: published
roboflow/roboflow-inference-server-jetson-6.2.0:latest(server reports Inference 1.7.2),started with the documented hardened command (
--read-only,/tmpvolume,--cap-drop=ALL,no-new-privileges).Before:
rfdetr-nano,rfdetr-base,rfdetr-medium-> HTTP 500 with the trace above;yolov8n-640-> OK.After: same image and command plus
-e TRITON_CACHE_DIR=/tmp/triton-cacheat runtime (equivalent to theENVline in this PR), no other Triton variable set, starting from an empty cache -> all three RF-DETR sizes returnpredictions (
person0.90-0.96 on two 1280x720 test frames)./tmp/triton-cachedid not exist before the firstrequest and held 24 compiled files afterwards, so Triton compiled into exactly the path this PR sets.
Commands (the
-e MAX_ACTIVE_MODELS,ACTIVE_LEARNING_ENABLEDandTELEMETRY_OPT_OUTflags were also setand are unrelated to this change):
sudo docker run -d --name inference-server --runtime nvidia --read-only -p 127.0.0.1:9001:9001 \ --volume /opt/nvme/inference-cache:/tmp:rw --env-file ~/.roboflow.env \ -e TRITON_CACHE_DIR=/tmp/triton-cache \ -e MAX_ACTIVE_MODELS=2 -e ACTIVE_LEARNING_ENABLED=False -e TELEMETRY_OPT_OUT=True \ --security-opt=no-new-privileges --cap-drop=ALL --cap-add=NET_BIND_SERVICE \ roboflow/roboflow-inference-server-jetson-6.2.0:latestUnit test:
pytest tests/inference/unit_tests/deployment/test_jetson_triton_cache_dir.py->1 passed(and
1 failedagainstmain's Dockerfile).black --checkandisort --check-onlypass on the new file.Not done: rebuilding the 6.2.0 image from this Dockerfile (Triton is compiled from source; not feasible on the
test device). The verification uses the published image with the variable set at runtime.
Related (not in this PR)
By code inspection only (not tested on x86 GPU hardware):
Dockerfile.onnx.gpuandDockerfile.onnx.cu13.gpuship theTriton JIT toolchain for RF-DETR preprocessing but do not set
TRITON_CACHE_DIR, and the Linux GPU install docs alsouse
--read-onlywith a/tmpvolume, so they may hit the same error. Happy to follow up after testing on a GPU host.On an 8 GB Orin Nano, loading several models with the default
MAX_ACTIVE_MODELSproducedCUBLAS_STATUS_ALLOC_FAILED(nvmap ENOMEM);-e MAX_ACTIVE_MODELS=2avoided it. Happy to open a separate docs PRfor an 8 GB note if that is useful.
Investigation and draft prepared with Claude Code; tested on-device and reviewed by me.