Python / PyTorch / CUDA
vLLMini
A minimal inference stack for understanding how modern LLM serving systems handle model loading, quantization, attention, and token sampling.
return to projects
learned
- How KV cache layout affects generation speed and memory pressure.
- Why quantization choices are a systems tradeoff, not just a model-quality setting.
- How much complexity hides behind a clean sampling API.
writeup queue
- Write a deeper note on paged attention and cache allocation.
- Add benchmarks across batch size, context length, and quantization mode.