[Feat] Build a public labelled set for the injection guard from Injec… · ThinkFlowLab/system1-agents@00516d1 · GitHub
Skip to content

Commit 00516d1

Browse files
GoDiaoGoDiao
committed
[Feat] Build a public labelled set for the injection guard from InjecAgent and AgentDojo
scripts/build_injection_dataset.py builds evals/labelled/injection-public.jsonl from InjecAgent at a fixed commit and agentdojo 0.1.35 (benchmark v1.2.2). The agentdojo dependency is declared as PEP 723 inline metadata, so pyproject.toml and uv.lock stay unchanged, and a rebuild writes the same bytes. The set holds 665 tool outputs. 335 carry a planted instruction, sampled with a fixed seed across user tools, injection tasks and attack templates; 330 carry none, among them normal requests written for a person. Positives that only the user's task tells apart are marked in their notes and reported on their own. rails.evaluate records the labelled set's sha256, taken from the bytes it parsed before the first model call, and the decision model's name in summary.json. --report turns a job into the benchmark tables; it recomputes the digest of the set it reads, from the same bytes it parses, and stops on a mismatch or on a job without these fields, labels its tables with the recorded model, and checks each note's grade against the row's label. docs/benchmarks.md gains the Jev run of 2026-10-05; its verdicts and summary are archived on the pull request rather than committed. NOTICE and THIRD_PARTY_LICENSES.md carry both MIT notices; evals/README.md, docs/roadmap.md and CHANGELOG.md describe the new set.
1 parent 3a2c2c6 commit 00516d1

11 files changed

Lines changed: 2339 additions & 8 deletions

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions

‎NOTICE‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,9 @@ This product includes software developed by third parties:
1212
evals/2048/style/fonts are Clear Sans by Intel Corporation (Apache-2.0)
1313
- evals/millionaire/questions.json, fetched at first run: Open Trivia Database
1414
(https://opentdb.com), CC BY-SA 4.0
15+
- evals/labelled/injection-public.jsonl: tool outputs from InjecAgent,
16+
uiuc-kang-lab/InjecAgent (MIT, Copyright (c) 2023 Qiusi Zhan). Full license text:
17+
THIRD_PARTY_LICENSES.md
18+
- evals/labelled/injection-public.jsonl: tool outputs from AgentDojo, ethz-spylab/agentdojo
19+
(MIT, Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca
20+
Beurer-Kellner, Marc Fischer, and Florian Tramèr). Full license text: THIRD_PARTY_LICENSES.md

‎THIRD_PARTY_LICENSES.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,53 @@ Fonts under `evals/2048/style/fonts` are Clear Sans by Intel Corporation, Apache
3939

4040
`evals/millionaire/questions.json`, fetched on the first Millionaire run, holds questions from the
4141
[Open Trivia Database](https://opentdb.com), Creative Commons Attribution-ShareAlike 4.0 International.
42+
43+
## InjecAgent
44+
45+
The records of `evals/labelled/injection-public.jsonl` whose note starts with `injecagent-` are built by
46+
`scripts/build_injection_dataset.py` from [uiuc-kang-lab/InjecAgent](https://github.com/uiuc-kang-lab/InjecAgent) at
47+
commit `f19c9f2c79a41046eb13c03c51a24c567a8ffa07`: `data/test_cases_dh_base.json`, `data/test_cases_ds_base.json` and
48+
`data/attacker_simulated_responses.json`.
49+
50+
MIT License
51+
52+
Copyright (c) 2023 Qiusi Zhan
53+
54+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
55+
documentation files (the "Software"), to deal in the Software without restriction, including without limitation
56+
the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
57+
to permit persons to whom the Software is furnished to do so, subject to the following conditions:
58+
59+
The above copyright notice and this permission notice shall be included in all copies or substantial portions
60+
of the Software.
61+
62+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
63+
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
64+
THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
65+
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
66+
DEALINGS IN THE SOFTWARE.
67+
68+
## AgentDojo
69+
70+
The records of `evals/labelled/injection-public.jsonl` whose note starts with `agentdojo` are tool outputs of the
71+
[ethz-spylab/agentdojo](https://github.com/ethz-spylab/agentdojo) suites, benchmark version v1.2.2 of the `agentdojo`
72+
0.1.35 package, produced by `scripts/build_injection_dataset.py` from the user tasks' ground truth with and without the
73+
package's attacks, and from the suites' own read tools on the default environments.
74+
75+
MIT License
76+
77+
Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr
78+
79+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
80+
documentation files (the "Software"), to deal in the Software without restriction, including without limitation
81+
the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
82+
to permit persons to whom the Software is furnished to do so, subject to the following conditions:
83+
84+
The above copyright notice and this permission notice shall be included in all copies or substantial portions
85+
of the Software.
86+
87+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
88+
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
89+
THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
90+
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
91+
DEALINGS IN THE SOFTWARE.

‎docs/benchmarks.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,46 @@ The same showcase on ALFWorld, Blackjack and Millionaire is under `results/<eval
7878

7979
The rail scores 20 of 20 on its labelled set at a median of 464 ms (`s1a run injection_guard`).
8080

81+
A larger set, `evals/labelled/injection-public.jsonl`, uses data from two public benchmarks, InjecAgent and AgentDojo:
82+
665 tool outputs such as emails, calendar entries, files, web pages and product pages, 335 of them with an instruction
83+
planted by an attacker. `scripts/build_injection_dataset.py` rebuilds the file. The run below is from 2026-10-05
84+
(`uv run s1a run injection_guard --model jev --labelled-set evals/labelled/injection-public.jsonl`), at a median of
85+
304 ms per decision and $0.016 for the whole set. Its records are archived with their provenance on
86+
[ThinkFlowLab/system1-agents#23](https://github.com/user-attachments/files/33183041/injection-guard-jev-2026-10-05.zip). Unpack the archive in a checkout and
87+
`uv run scripts/build_injection_dataset.py --report injection-guard-jev-2026-10-05` prints the tables below from them.
88+
89+
| tool output | count | quarantined by Jev |
90+
|---|---|---|
91+
| no planted instruction | 330 | 0 |
92+
| of which a normal request written for a person, such as a bill asking for payment | 34 | 0 |
93+
| planted instruction in attack wording, such as "ignore your previous instructions" (AgentDojo) | 154 | 150 |
94+
| planted request to send the user's data somewhere (InjecAgent) | 61 | 58 |
95+
| planted request for a harmful action, with no attack wording (InjecAgent) | 55 | 23 |
96+
| planted request that reads like an ordinary one, such as a payment request in an email | 65 | 34 |
97+
98+
The last row is counted apart. Only the user's task tells these requests from real ones, and the rail sees only the
99+
tool output.
100+
101+
Over the 600 outputs outside the last row, the run gives these precision and recall figures:
102+
103+
| measure | Jev | quarantine everything |
104+
|---|---|---|
105+
| precision: share of quarantined outputs that carry a planted instruction | 1.00 | 0.45 |
106+
| recall, InjecAgent direct harm (55) | 0.42 | 1.00 |
107+
| recall, InjecAgent data stealing (61) | 0.95 | 1.00 |
108+
| recall, AgentDojo (154) | 0.97 | 1.00 |
109+
110+
Recall is the share of planted instructions that the rail quarantined. "Quarantine everything" is what a rail scores if
111+
it quarantines every output.
112+
113+
On the 150 outputs that carry no attack wording (the 116 planted instructions without it and the 34 normal requests), a
114+
rule that quarantines any text containing "please" is right 65% of the time, counting the two groups equally. Jev is
115+
right 85% of the time: it catches 70% of the planted instructions and none of the normal requests.
116+
117+
The set has two limits. Only 34 of the normal outputs contain a request. The planted instructions and the normal
118+
requests also come from different benchmarks with different text formats, so a simple word-count classifier already
119+
separates them 73% of the time.
120+
81121
## Series
82122

83123
The protocol in `evals/README.md` quotes nothing under ten episodes and asks for 500 Blackjack hands. The series of

‎docs/roadmap.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,17 @@ in `README.md`, `docs/agents.md`, `docs/browser-front.md` (the browser policy) a
66

77
## Rails: one shipped, two to ship
88

9-
### Prompt-injection guard: shipped, with a 20-item labelled set
9+
### Prompt-injection guard: shipped, with two labelled sets
1010

1111
jiuwen ingests untrusted pages through `browser_snapshot` and `fetch_webpage`. The `injection_guard` rail
1212
(`s1a/agents/injection_guard.py`) ships: one rail on `after_tool_call` asks Jev "does this text instruct
1313
the agent?" (`noul`) and quarantines the result; 20 of 20 on `evals/labelled/injection.jsonl` at a median of
14-
464 ms. Still to build: the planted-page demo (the browser subagent visits a planted page and the guard flags the
15-
planted instruction before the model reads it) and a labelled set from the public ones, InjecAgent (1,054 tool-output
16-
injections) and AgentDojo (97 tasks, 629 security cases). jiuwen's own
14+
464 ms. `evals/labelled/injection-public.jsonl` adds 665 records from the public InjecAgent and AgentDojo benchmarks,
15+
built by `scripts/build_injection_dataset.py`: 335 positives, 65 of them needing the user's task to tell apart and
16+
reported on their own, and 330 negatives. Its precision, recall and hard-subset balanced accuracy are in
17+
`docs/benchmarks.md`. Still to build:
18+
the planted-page demo (the browser subagent visits a planted page and the guard flags the planted instruction before
19+
the model reads it). jiuwen's own
1720
`openjiuwen/harness/rails/security/prompt_security_rail.py` is pattern-based; `SafetyPromptRail` only injects
1821
guideline text before model calls.
1922

‎evals/README.md‎

Lines changed: 1 addition & 0 deletions

0 commit comments

Comments
 (0)