Artifacts
What I build, what I think, What I learn - I generally write here
- 18
Building a safe async policy for RL rollouts in vLLM
Oct 202637viewsActually, I've been spending a lot of time around rollouts these days. And one thing that I like is how much of RL training actually depends on inference infra.
GitHub ↗https://github.com/awesome-pro/rolloutcoreImplementation ↗https://github.com/awesome-pro/rolloutcore/tree/main/src/rolloutcoreResult ↗https://github.com/awesome-pro/rolloutcore/tree/main/resultsvLLM audit ↗https://github.com/awesome-pro/rolloutcore/blob/main/docs/source-map-vllm-main.mdhttps://abhinandan.one/artifacts/building-a-safe-async-policy-for-rl-rollouts-in-vllm · serial 18 · published 2026-10-15T06:38:50.871+00:00 · updated 2026-10-16T10:51:24.398+00:00 - 08
Building a quantized HiCache for SGLang
Oct 202693viewsKV caching is probably one of the most impactful optimizations in LLM inference. If u think about it: without KV caching, every time we generate a new token, we're unnecessarily…
GitHub ↗https://github.com/awesome-pro/hiqcacheSGLang PR ↗https://github.com/sgl-project/sglang/pull/41365Experiment A ↗https://github.com/awesome-pro/hiqcache/blob/main/docs/experiment-a-results.mdExperiment B ↗https://github.com/awesome-pro/hiqcache/blob/main/docs/experiment-b-results.mdhttps://abhinandan.one/artifacts/building-a-quantized-hicache-for-sglang · serial 08 · published 2026-10-13T16:56:17.791+00:00 · updated 2026-10-13T17:04:57.535+00:00 - 06
Prefill & Decode made Crystal Clear
Sep 202694viewsThese two terms are very common. And it's very possible that you may have heard them before, or maybe you even know what they mean. But I feel that even though earlier I knew…
https://abhinandan.one/artifacts/prefill-decode-made-crystal-clear · serial 06 · published 2026-09-22T13:10:55.521+00:00 · updated 2026-10-02T13:58:38.825+00:00 - 07
Understanding Attention From Scratch
Sep 202641viewsI think if you have ever spent even a single day learning about current models, then attention is most probably the first term you may have heard. And when it comes to the…
https://abhinandan.one/artifacts/understanding-attention-from-scratch · serial 07 · published 2026-09-16T17:30:00.202+00:00 · updated 2026-10-06T17:42:39.065+00:00 - 05
I do not know if this was my poor luck or me being late, but while trying to understand and measure speculative decoding myself, I realized that SGLang had already merged adaptive…
https://abhinandan.one/artifacts/measuring-adaptive-speculative-decoding-in-sglang · serial 05 · published 2026-09-04T08:00:00+00:00 · updated 2026-10-03T11:46:38.499+00:00 - 04
Building a Mini Inference Engine from Scratch
Jul 202687viewsMost of us know that LLMs are auto-regressive, i.e., they generate token one by one. We also know that they have a concept of KV caching and can process multiple requests…
GitHub ↗https://github.com/awesome-pro/miniserveYT Series ↗https://www.youtube.com/playlist?list=PLIi0G24WvbAAhttps://abhinandan.one/artifacts/building-a-mini-inference-engine-from-scratch · serial 04 · published 2026-07-21T10:38:43.277+00:00 · updated 2026-09-23T11:25:27.735+00:00 - 03
Building a Semantic Cache that Does Not Trust Cosine
Jul 202639viewsUsually semantic caches for LLMs work on embeddings and cosine similarity. You take the new prompt, turn it into a vector, find the closest old prompt, and if the score is high…
https://abhinandan.one/artifacts/building-a-semantic-cache-that-does-not-trust-cosine · serial 03 · published 2026-07-03T07:37:17.016+00:00 · updated 2026-09-23T08:34:02.344+00:00 - 02
Running a Health Voice: All Private on Your Mac
Jun 202639viewsThere is this promise that a doctor can just talk, and a note will write itself. It feels good & easy when you say it like that. Talk, note, done. But the patient's voice has to…
GitHub ↗https://github.com/awesome-pro/health_voiceCase Study ↗https://www.aidaddy.tech/learn/16-case-studies/13-voice-ai-healthcarehttps://abhinandan.one/artifacts/running-a-health-voice-all-private-on-your-mac · serial 02 · published 2026-06-26T18:19:07.916+00:00 · updated 2026-09-23T08:25:36.27+00:00 - 01
Building a RAG based Search System for an Enterprise
Jun 202646viewsThere will be only 0.000001% people in tech who have not heard the word RAG. Embed a few documents, find the close ones, paste them into a prompt. The answer comes back. It feels…
B_GitHub ↗https://github.com/awesome-pro/enterprise_rag_backendF_GitHub ↗https://github.com/awesome-pro/enterprise_rag_frontendhttps://abhinandan.one/artifacts/building-a-rag-based-search-system-for-an-enterprise · serial 01 · published 2026-06-25T18:19:07.916+00:00 · updated 2026-09-23T08:31:13.386+00:00