📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

图解 vLLM 推理引擎：100 课动手练习册 + ~500 行可运行的迷你推理引擎 | An illustrated, hands-on course on how vLLM works

A small, readable PyTorch inference engine for Qwen3-4B with continuous batching, native paged attention, prefix caching, and an OpenAI-compatible chat completions API.

A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM

⚡ High-throughput LLM serving runtime to run frontier 70B–753B models on a single 24GB GPU (RTX 4090/L4). Features 75% KV compression, 57% subspace activation pruning, and dual OpenAI + Anthropic APIs.

Miniature LLM serving runtime with continuous batching, chunked prefill, KV-cache management, preemption, prefix caching, and real Llama-family execution.

Implementation of PagedAttention from vLLM paper - a breakthrough attention algorithm that treats KV cache like virtual memory. Eliminates memory fragmentation, increases batch sizes, and dramatically improves LLM serving throughput.

Virtual paged attention KV-cache memory manager with logical block mapping, non-contiguous physical page allocation, and zero-copy fragmentation tracking.

Virtual paged attention KV-cache memory manager with logical block mapping, non-contiguous physical page allocation, and zero-copy fragmentation tracking.

Deterministic sparse-index localization for Apple silicon (MLX/Metal) and CUDA/Triton, with stable ordering, a Python oracle, and reproducible evidence.

Experimental LLM-serving systems reference with paged memory management, continuous batching, and tensor-partitioning simulations.

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

Measurement harness for the sliding window attention premium in the vLLM TPU Ragged Paged Attention v3 kernel: per layer decode cost, block size control, throughput, and goodput for Gemma 4 31B on TPU v6e.

Minimal, educational continuous-batching LLM inference engine with Paged KV-cache, prefix caching, and speculative decoding for Qwen2.5 in pure PyTorch.

Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.

An AI inference engine from scratch. Like nanoGPT, but for serving.