Potion (external result, preprint): request for a BEAM listing, and two sections of the paper that concern AMB · Issue #50 · vectorize-io/agent-memory-benchmark · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Potion (external result, preprint): request for a BEAM listing, and two sections of the paper that concern AMB #50
Opening this here at Naren Ramakrishnan's suggestion, in reply to my email of 25 September.
The paper. "Potion: Constant Context over Growing Histories, and What a Memory Leaderboard Measures", Robin L. de Bruijn, 25 September 2026, https://doi.org/10.5281/zenodo.22966163 (CC BY 4.0; the DOI resolves to the current version). v1.1 of 28 September corrects text, figures and references, gives the context-size comparison on AMB's tokenizer as well as on the paper's own, and states the further stage's answer shapes in §11; no reported result changed. Potion is a long-term memory for language agents that I developed independently; no closed model is used anywhere in it. The paper reports it on BEAM at 100K, 500K and 10M with an open-weight 27B reader, scored on AMB's judge configuration (Appendix A.1), and it reports the 10M contexts run through AMB's own answer path and judge. Every number in the paper is released with a per-row score file in the Zenodo deposit, including both pairings against the verified Hindsight 10M entry (under AMB's judge and under BEAM's scorer), so all of them recompute from the deposit.
Request 1 — a listing. I would like Potion listed on the BEAM page as an external row citing the preprint, with its conditions stated as the paper states them: its own open-weight reader on AMB's judge configuration, 63.1 at 10M with two disclosed hint lines (worth 2.5 points on the page they were added to), and 67.1 at 500K and 69.3 at 100K without them. The retrieved contexts are not released at this time, because their form discloses how the memory is built; I expect to be able to share more once that is settled, and until then the per-row files are the most I can offer.
Request 2 — your read of §3 and §4. Two sections concern AMB directly. Section 3 measures the rater effect: the same answers move 6.8 to 8.8 points per tier between AMB's judge configuration and a newer Gemini judge. Section 4 documents the pre-June answer prompt's per-category annotations, their scope, and their signature in published answers. The paper states plainly that this applied to every entry alike, that AMB itself removed it in June, and that no misconduct is alleged; it is written as an audit of a test, in the vocabulary of measurement. If either section contains an error of fact, I will correct it in the next version and credit the correction. I would welcome your read before I say anything further about it publicly.
Thank you for publishing the full harness; it is the reason any of this could be checked.
Activity