{{ message }}
Repository navigation
track: SOTA comparator benchmark drive (perf/sota-comparator-benchmarks) #2125
Description
Goal
Get ruflo's agent layer to defensible SOTA vs LangGraph, AutoGen, CrewAI on an identical workload, with end-to-end timing, multi-platform, concurrency, and a v3.7→v3.8 delta. Then push past SOTA with shipped speedups.
Branch / PR
- Branch:
perf/sota-comparator-benchmarks(off main @bf0f505c9, post v3.8.0) - PR: perf: M1-M3 SOTA comparator benchmarks — ruflo vs LangGraph/AutoGen/CrewAI #2124 (ready for review)
- Driver: autopilot worker in /loop 5m
Milestones
- M1 — Workload spec (
docs/benchmarks/sota-workload-spec.md) - M2 — Comparator selection (LangGraph 1.2.1, AutoGen 0.4.9, CrewAI 0.80.0, ruflo 3.8.0)
- M3 — Harnesses + first verified matrix (Mode A, stub LLM)
- M4 — Cross-platform matrix (darwin-arm64 + linux-x64 on ruvultra, 32-core Ubuntu 24.04)
- M5 — End-to-end with real Anthropic model (BLOCKED: GCP secret stale — harness ready at
benchmarks/bench-mode-b.mjs) - M6 — Concurrency N=1/10/50/100 → 421K tool dispatches/sec at N=100
- M7 — v3.7.0 vs v3.8.0 delta → 1.83× createWasmAgent
- M8 — Real plugin enum (21 plugins) → effectively free after all 4 speedups
- M9 — Gist published + v3.8.0 release notes patched → https://gist.github.com/ruvnet/298f8c668c8859b369f91734a0e9cbbe
- M10 — 4 speedups shipped: manifest cache + isDestructiveTool fast-path + Buffer hoist + loadAgentWasm memoize → compose 0.351ms → 0.146ms (2.4×). Witness manifest updated.
Summary: 9/10 milestones done. Only M5 blocked (key needed).
Current standings — darwin-arm64 (Mode A, verified-real WASM, after M10)
Current standings — linux-x64 (ruvultra, 32 cores, Mode A)
| Dimension | ruflo | AutoGen | LangGraph | CrewAI |
|---|---|---|---|---|
| Cold start (ms) | 2.66 | 104.1 | 213.0 | 1420.9 |
| Compose 50 tools (ms) | 0.146 | 4.85 | 26.9 | 0.096† |
| Single turn (ms) | 0.053‡ | 4.94 | 31.3 | 0.091† |
| N=10 parallel (ms) | 0.751 | 48.9 | 349.2 | 0.093† |
| RSS peak (MB) | 60.2 | 77.4 | 78.6 | 251.2 |
† CrewAI = proxied lower bounds. ‡ WASM darwin-arm64 only; linux proxied by compose.
Activity
ruvnet commentedon May 24, 2026
Witness manifest updated — commit 8458389
What happened:
node scripts/regen-witness.mjsafter the M10 production speedups (commit 88ab2c9)perf/sota-comparator-benchmarks@ v3.8.0smoke-init-bundle-invariantsandsmoke-wasm-plugin-bridgeboth PASSverification/macos/history.jsonlMilestones: M10 witness obligation fulfilled (ADR-103 requirement).
Matrix delta: No number changes — this is bookkeeping only. Current standings unchanged:
M5 status:
ANTHROPIC_API_KEYis not in the current shell environment. M5 (real model timing) is gated on the key being available. Will proceed when key is present.Next: M4 CI workflow is pushed at
.github/workflows/sota-bench.yml— Linux numbers will appear when PR CI runs. M9 (gist + release notes) is blocked until M4 Linux numbers are confirmed. Can continue with additional M10 speedups in the meantime.ruvnet commentedon May 24, 2026
M4 complete + M10 Speedups 3+4 — commits e1ead3c, eff3df1, e2a3031
M10 Speedups 3 and 4 shipped:
import { Buffer } from 'node:buffer'to module top — eliminatesawait import('node:buffer')microtask on everywasm_agent_composeandgallery_load_rvfcallloadAgentWasm()via_agentWasmPromisesingleton — all ~20 MCP tool handlers now return the same settled Promise instead of firing a new microtask per callCombined result: compose_50_tools down from 0.351ms (pre-M10) → 0.146ms (2.4× total improvement). Plugin-enum baseline now 0.083ms (was 0.127ms).
M4 Linux numbers captured on ruvultra (Ubuntu 24.04, 32 cores):
† CrewAI proxy (lower bound). ‡ ruflo linux: WASM binary is darwin-arm64 only; single_turn proxied by compose.
Linux speedups confirmed: cold start 39×–533× faster, N=10 parallel 65× vs AutoGen, 465× vs LangGraph. Full data:
docs/benchmarks/sota-matrix-linux.json.M5 status: GCP ANTHROPIC_API_KEY secret returns 401 (stale). M5 harness written at
benchmarks/bench-mode-b.mjs. Blocked.Next: M9 (gist + release notes) is now unblocked by M4. Will create the gist now.
ruvnet commentedon May 24, 2026
M9 complete — gist published + v3.8.0 release notes patched — commit 1803909
Gist: https://gist.github.com/ruvnet/298f8c668c8859b369f91734a0e9cbbe
Contains: workload spec, full cross-platform matrix (darwin-arm64 + linux-x64), M10 speedup table, concurrency scale (421K tool_dispatches/sec at N=100), v3.7→v3.8 delta, repro command, methodology caveats.
v3.8.0 release notes patched at https://github.com/ruvnet/ruflo/releases/tag/v3.8.0 — includes the SOTA comparison table, 4 speedup descriptions, and gist link.
Current milestone completion: 9/10
benchmarks/bench-mode-b.mjswritten, ready to run when valid key is available.What's left:
ruvnet commentedon May 26, 2026
Closing — shipped in v3.8.0.
Milestones complete:
M5 (end-to-end ruflo with real model) remains as a separate workstream (not blocking this tracker).
Results:
docs/SOTA-PROGRESS.md,docs/benchmarks/sota-workload-spec.md,docs/benchmarks/sota-matrix.json, gist298f8c668c8859b369f91734a0e9cbbe.