Summary

Apply the existing score-command prediction ID checks to compare. Previously duplicate rows silently overwrote predictions and rows outside the selected split disappeared, yielding a plausible paired comparison from an invalid submission.

Evidence and validation

• Reproduced on unchanged upstream main ccf586d43d2a580319b6535c893668904d909eb9 with the regression tests in this PR.
• Focused native tests: python -m pytest -p no:cacheprovider -q tests/test_compare_prediction_ids.py tests/test_imajev_bench_v2.py tests/test_imajev_bench_scoring.py pass after the patch; pre-patch failures are preserved locally.
• git diff --check passes.
• Unchanged upstream main in this environment: 4 failed, 713 passed, 136 skipped. Two failures import the unavailable optional mlx_vlm dependency; two import the absent test_p3_labeltrain fixture helper.
• Full python -m pytest -p no:cacheprovider -q: Exact PR head 7daff21b2089: 4 failed, 718 passed, 136 skipped, 35 warnings in 59.64s. Only those same four upstream/environment failures remain; this patch introduces no additional failures.
• This repository currently reports no GitHub status checks; that is not a claim of passing remote CI.
• Tests use local synthetic fixtures; no released model weights, private data, hosted-provider calls, or model quality claims.

Contract references

• cli.main score already requires unique string IDs inside the selected split; compare now reuses that same validation. No change to missing-prediction scoring.

AI assistance

Prepared with AI assistance; the reproduction, patch and tests were reviewed before submission.