Related Issues

• part of deepset-ai/haystack-private#498

Proposed Changes:

Third part of the harness optimization work: a harness evaluator for Advanced RAG agents, so the experiment from #4069 can optimize them.

• AdvancedRAGHarnessEvaluator scores an agent on RetrievalEvalCases:
  • per eval case as AdvancedRAGEvalCaseMetrics: retrieved and cited recall, citations, tool calls and errors, and a digest of the run
  • in aggregate as mean_budgeted_recall, which is 0 for a run over its tool-call budget
  • to_dict/from_dict, so the optimizer agent holding it can be serialized
• Evaluation helpers it uses: ToolRunStats, tool budgets (resolve_tool_budgets, budgets_exceeded) and digest_agent_run with AgentRunDigestPolicy.
• The Advanced RAG system prompt asks for one filter covering every source a question spans.
• Examples: advanced_rag_harness_optimization.py runs a full experiment; advanced_rag_eval.py scores one agent.

How did you test it?

Unit tests for the evaluator and the evaluation helpers, plus tests/advanced_rag/test_optimization_e2e.py, which runs an experiment against scripted agents. hatch run test:unit, hatch run test:types and hatch run fmt-check pass locally.

Notes for the reviewer

• Stacked on feat: Add harness optimization experiment #4069.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.