Search results

Agents wire a placeholder verification secret into a live source, and nothing surfaces it

findingA result we do not yet understand and should investigateA result we do not yet understand and should investigate

Status: Open.

#75 In hookdeck/evals;

· leggetter opened on Sep 2, 2026

·  

5. Sustain

Re-running publish-results on a held run fails to push

harnessRunner, scorers, provisioner and CIRunner, scorers, provisioner and CI

Status: Open.

#74 In hookdeck/evals;

· leggetter opened on Sep 2, 2026

·  

4. Trust the numbers

Publishing is held: clear EVALS_PUBLISH when a clean snapshot lands

Status: Open.

#66 In hookdeck/evals;

· leggetter opened on Aug 28, 2026

·  

3. Finish publishing

The -no-skills arm reaches documentation differently, so part of the skills delta is not skills

findingA result we do not yet understand and should investigateA result we do not yet understand and should investigate

Status: Open.

#61 In hookdeck/evals;

· leggetter opened on Aug 26, 2026

·  

2. Prove the skills earn their place

A published snapshot mixes execution dates, so counting snapshots overcounts measurements

findingA result we do not yet understand and should investigateA result we do not yet understand and should investigate

Status: Open.

#60 In hookdeck/evals;

· leggetter opened on Aug 26, 2026

·  

4. Trust the numbers

Five benchmark scenarios have never been failed by any agent

scenarioScenario coverage, design or correctnessScenario coverage, design or correctness

Status: Open.

#47 In hookdeck/evals;

· leggetter opened on Aug 24, 2026

·  

1. Cover the product

Hookdeck's webhook notification config cannot be read back, and the natural way to set it returns 200 while doing nothing

documentationImprovements or additions to documentationImprovements or additions to documentation

scenarioScenario coverage, design or correctnessScenario coverage, design or correctness

Status: Open.

#43 In hookdeck/evals;

· leggetter opened on Aug 21, 2026

·  

5. Sustain

The provider-key preflight only runs in CI, not for a local run

harnessRunner, scorers, provisioner and CIRunner, scorers, provisioner and CI

Status: Open.

#42 In hookdeck/evals;

· leggetter opened on Aug 21, 2026

·  

4. Trust the numbers

The Outpost skill never says you need a key for an Outpost project

findingA result we do not yet understand and should investigateA result we do not yet understand and should investigate

Status: Open.

#40 In hookdeck/evals;

· leggetter opened on Aug 21, 2026

·  

5. Sustain

The Outpost API answers "Not Found" when your key belongs to a non-Outpost project

findingA result we do not yet understand and should investigateA result we do not yet understand and should investigate

Status: Open.

#39 In hookdeck/evals;

· leggetter opened on Aug 21, 2026

·  

5. Sustain

Nothing detects an agent reading this repository during a run

harnessRunner, scorers, provisioner and CIRunner, scorers, provisioner and CI

Status: Open.

#38 In hookdeck/evals;

· leggetter opened on Aug 21, 2026

·  

4. Trust the numbers

The operator events API is undocumented: absent from both OpenAPI definitions and from the docs

documentationImprovements or additions to documentationImprovements or additions to documentation

findingA result we do not yet understand and should investigateA result we do not yet understand and should investigate

Status: Open.

#34 In hookdeck/evals;

· leggetter opened on Aug 20, 2026

·  

5. Sustain