Feature Proposal: Tracing and Observability Support (e.g., Agent-Devtools)

Summary

optillm acts as an optimizing proxy that routes, combines, and modifies queries across multiple techniques (e.g., Monte Carlo Tree Search, Mixture of Agents, CePO, PlanSearch).

To improve debugging, latency tracking, and visibility into multi-step reasoning processes, I propose adding a lightweight tracing hook or native plugin integration for observability platforms like Agent-Devtools.

Problem / Motivation

When using complex optimization strategies (such as mcts, moa, or plansearch):

• Multiple downstream API requests are made under the hood before returning a single response.
• It can be difficult to visually inspect the intermediate completions, ratings, or plan generation steps.
• Developers debugging custom workflows or latency issues lack a standardized way to trace the execution tree of a single proxy request.

Proposed Solution

Provide an optional plugin or tracing interface (e.g., via a --trace-devtools flag, plugin, or environment variable like OPTILLM_DEVTOOLS=true) that logs or streams intermediate reasoning steps and tool execution data to an inspection tool.

This could be integrated by:

1. Adding an Observability/Tracer Plugin: Extending optillm's existing plugin system (optillm/plugins/) to hook into technique pipelines.
2. Standardized Event Emitting: Emitting structured events during multi-step inference loops (e.g., candidate generation, verification, tree search depth steps).

Benefits

• Full Execution Visibility: Easily inspect how approaches like moa or cepo decompose and evaluate requests step-by-step.
• Improved DX: Streamlines debugging for developers building on top of optillm as their primary inference proxy.
• Performance Profiling: Help identify bottlenecks or high-latency LLM calls within complex pipelines.

Next Steps

I would love to gather feedback from the maintainers on whether adding tracing hooks or an Agent-Devtools integration aligns with the repository's roadmap!