{{ message }}
Repository navigation
Conversation
Replace the agentx-harness 1.0.12 client with ai-dynamo/aiperf at 8aa7039f872cd029f05a4c313fb8759e9a482cd3. The replay runs `aiperf profile --scenario inferencex-agentx-mvp` and passes the fork's agentx preset as explicit flags and AIPERF_* defaults, which callers can still override. Job events record the pinned AIPerf commit. 中文： 将 AgentX 客户端从 agentx-harness 1.0.12 切换为 ai-dynamo/aiperf 8aa7039f872cd029f05a4c313fb8759e9a482cd3。回放改为运行 `aiperf profile --scenario inferencex-agentx-mvp`，并以显式参数和 AIPERF_* 默认值传入 fork 中 agentx 预设的设置，调用方仍可覆盖。 作业事件记录所固定的 AIPerf 提交。
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AgentX currently runs on
agentx-harness==1.0.12, SemiAnalysis's fork of AIPerf. Upstreamai-dynamo/aiperfnow contains everything AgentX relies on: the fork was ported in ai-dynamo/aiperf#1413, and the SPAWN fixes landed as #1540 (roll back delayed SPAWN children at the sending cutoff) and #1567 (pin SPAWN children under their own sticky entry). This PR moves AgentX to upstream AIPerf at8aa7039f872cd029f05a4c313fb8759e9a482cd3, removing the fork from the benchmark path.infx/bench/agentic/requirements.txtinstalls AIPerf from the pinned commit's source tarball, so the serving containers don't need git.aiperf profile --scenario inferencex-agentx-mvp. The fork-onlyagentxpreset (feat(scenario): support CLI and runtime environment defaults ai-dynamo/aiperf#1495 adds scenario defaults upstream but is not merged, and no PR adds the preset) is passed as explicit flags plusAIPERF_DATASET_CONFIGURATION_TIMEOUT,AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT,AIPERF_UI_REALTIME_METRICS_ENABLEDandAIPERF_HTTP_TCP_USER_TIMEOUT. Recipe and workflow values still win.job_event.jsonrecords the pinned commit asversions.aiperf_sha.agentx_harnessstays in the schema for older records and is null for new ones.aiperf profile.Behavior changes vs 1.0.12 (why this is a perf-changelog entry for every agentic config):
X-Session-Affinitywith the correlation ID (upstream defaultAIPERF_HTTP_X_SESSION_AFFINITY_FROM_CORRELATION_ID=true).Apart from these, the benchmark config AIPerf builds is unchanged. For the default, FAST, under-900 s and all-opt-ins points, the CLI parse, plan build and scenario check on the installed pin give the same values as 1.0.12 for all shared settings, except the scenario name (
agentx→inferencex-agentx-mvp).Open issues (draft):
dynamo-*frameworks with conversation-aware routing,replay.pypasses--use-dynamo-conv-aware-routingand--dynamo-session-timeout-seconds. Neither 1.0.12 nor upstream AIPerf accepts these flags ("Unknown option"), so those points fail onmaintoday, and this PR keeps the same flags.validate_perf_changelogrejects the"*"entry because matrix planning fails onbenchmarks/single_node/srt-slurm-recipes/qwen3.5/sglang/h100-fp8/agentic.yaml(KV offloading / DRAM override mismatch). This PR does not touch that config."*"entry sweeps every agentic config, so the sweep scope needs agreeing before a primary sweep label is applied.AIPERF_SLICE_DURATION; this PR fixes--slice-duration 1.0.AI model disclosure
claude-opus-5-5(Anthropic, as exposed by the Hermes runtime). Role: gap analysis of agentx-harness vs upstream AIPerf, config-parity checks, code, tests, docs, changelog entry and this description.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概要
AgentX 目前使用
agentx-harness==1.0.12，即 SemiAnalysis 维护的 AIPerf fork。上游ai-dynamo/aiperf现已包含 AgentX 所依赖的全部功能：fork 的改动已通过 ai-dynamo/aiperf#1413 移植，SPAWN 相关修复也已合入，分别为 #1540（在发送截止时回滚延迟的 SPAWN 子会话）和 #1567（SPAWN 子会话使用各自的 sticky 路由条目）。本 PR 将 AgentX 切换到上游 AIPerf 提交8aa7039f872cd029f05a4c313fb8759e9a482cd3，使基准测试不再依赖该 fork。infx/bench/agentic/requirements.txt从固定提交的源码压缩包安装 AIPerf，服务容器无需安装 git。aiperf profile --scenario inferencex-agentx-mvp。fork 独有的agentx预设以显式参数传入（feat(scenario): support CLI and runtime environment defaults ai-dynamo/aiperf#1495 为上游增加了场景默认值，但尚未合并，也还没有添加该预设的 PR），并设置AIPERF_DATASET_CONFIGURATION_TIMEOUT、AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT、AIPERF_UI_REALTIME_METRICS_ENABLED和AIPERF_HTTP_TCP_USER_TIMEOUT。recipe 和 workflow 中的设置仍然优先。job_event.json以versions.aiperf_sha记录固定的提交。agentx_harness字段保留在 schema 中以兼容旧记录，新记录中为 null。aiperf profile。相对 1.0.12 的行为变化（因此为所有 agentic 配置新增 perf-changelog 条目）：
X-Session-Affinity，其值为 correlation ID（上游默认AIPERF_HTTP_X_SESSION_AFFINITY_FROM_CORRELATION_ID=true）。除上述变化外，AIPerf 生成的基准测试配置保持不变。对 default、FAST、低于 900 秒以及启用全部可选参数这几种情况，在已安装的固定版本上执行 CLI 解析、计划构建和场景检查，所有共有设置的值都与 1.0.12 相同，唯一的区别是场景名称（
agentx→inferencex-agentx-mvp）。未解决问题（草稿）：
dynamo-*框架，replay.py会传入--use-dynamo-conv-aware-routing和--dynamo-session-timeout-seconds。1.0.12 和上游 AIPerf 都不接受这两个参数（报 "Unknown option"），因此这些测试点目前在main上就会失败，本 PR 保留了相同的参数。validate_perf_changelog拒绝了"*"条目，原因是 matrix 规划在benchmarks/single_node/srt-slurm-recipes/qwen3.5/sglang/h100-fp8/agentic.yaml上失败（KV offloading / DRAM override 不匹配）。本 PR 没有改动该配置。"*"条目会对所有 agentic 配置执行 sweep，因此在添加主 sweep 标签之前，需要先确定 sweep 范围。AIPERF_SLICE_DURATION，而本 PR 将--slice-duration固定为 1.0。AI 模型使用说明
claude-opus-5-5（Anthropic，由 Hermes 运行环境提供）。工作内容：对比 agentx-harness 与上游 AIPerf 的差异、检查配置一致性，并编写代码、测试、文档、changelog 条目和本说明。