Related Issues

• part of deepset-ai/haystack-private#498

Proposed Changes:

First part of the harness optimization work: an agent that edits a configuration's YAML into one improved candidate. The loop that calls it (HarnessOptimizationExperiment) follows in #4069.

• create_harness_optimizer_agent(*, evaluator, loader, ...) builds the optimizer agent. It has:
  • editing tools read_config, edit_config, validate_config, submit_candidate, restore_candidate, finish, plus inspect_component
  • submit_candidate and finish as exit conditions
  • optional read-only search over the Haystack docs (documentation_tools=True)
  • It round-trips through to_dict/from_dict, evaluator included, and defaults to gpt-5.6-terra with low reasoning effort.
• propose_candidate runs one turn. It passes the starting point to the agent as state (ConfigurationDraft: the YAML to edit; KnownConfigurations: every configuration measured so far) and returns a ProposalResult (the submitted candidate or None, the finish reason, failed validations).
• The editing tools in optimization/tools.py read and write that state. ValidateConfig runs the evaluator's validate on each draft. A candidate must be validated before it can be submitted, and known configurations can't be resubmitted.
• OptimizationObjectives gates candidates (quality floor, reported usage) and ranks them by cost, duration or quality.
• HarnessEvaluator gains to_dict/from_dict (implemented by RetrievalHarnessEvaluator), so the agent can serialize its evaluator.
• New dependencies: pyyaml, and mcp-haystack for tests.

How did you test it?

Unit tests in tests/optimization and for RetrievalHarnessEvaluator serialization. hatch run test:unit, hatch run test:types and hatch run fmt-check pass locally.

One optimizer turn on its own (needs OPENAI_API_KEY):

from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.document_stores.in_memory import InMemoryDocumentStore

from haystack_integrations.agent_pack.evaluation import EvalMetrics, RetrievalHarnessEvaluator
from haystack_integrations.agent_pack.optimization import (
    OptimizationObjectives,
    create_harness_optimizer_agent,
    propose_candidate,
)
from haystack_integrations.agent_pack.optimization.utils import dump_pipeline, load_pipeline

reference = Pipeline()
reference.add_component("retriever", InMemoryBM25Retriever(document_store=InMemoryDocumentStore(), top_k=5))
turn = propose_candidate(
    optimizer_agent=create_harness_optimizer_agent(evaluator=RetrievalHarnessEvaluator(k=5), loader=load_pipeline),
    reference=reference,
    reference_yaml=dump_pipeline(reference),
    prices={},
    objectives=OptimizationObjectives(quality_metric="mean_recall_at_k", primary="quality"),
    baseline=EvalMetrics(durations=[1.0], all_tokens_reported=True, details={"mean_recall_at_k": 0.5}),
    history=[],
)
print(turn.candidate.diff if turn.candidate else turn.finish_reason)

How the experiment calls it each turn: HarnessOptimizationExperiment.run in #4069.

Notes for the reviewer

• Builds on feat: Add HarnessEvaluator protocol and model pricing to agent pack evaluation #4038 (merged).
• optimization/utils.py is left out of the API docs.
• feat: Add harness optimization experiment #4069 (the experiment) is stacked on this PR, and feat: Add Advanced RAG harness evaluator #3932 (the Advanced RAG harness evaluator) on that.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.