Back
infer.c: C/CUDA LLM Inference Engine
CCUDALLMsInferencingSystemsFlashAttention
C/CUDA inference engine with Paged KV Cache, Speculative Decoding, cBLAS/cuBLAS, Batching, FlashAttention, CUDA Graphs, and systems-level optimizations.
Building a high-performance LLM inference engine from scratch in C and CUDA. The project explores the full stack of inference optimizations: paged KV cache management, speculative decoding, batched execution, FlashAttention kernels, CUDA Graphs for reduced launch overhead, and BLAS-accelerated matrix operations. The goal is deep systems understanding of how production inference engines actually squeeze latency and throughput out of GPUs.
Key Features
- Paged KV Cache for efficient memory management
- Speculative Decoding for faster token generation
- cBLAS/cuBLAS-accelerated matrix operations
- Dynamic batching for throughput optimization
- FlashAttention integration
- CUDA Graphs for reduced kernel launch overhead
- Systems-level profiling and optimization
Technology Stack
Backend
CCUDAcBLAScuBLAS
AI/ML
LLM InferenceFlashAttentionSpeculative Decoding
Tools
CUDA GraphsProfiling
Challenges
- Balancing memory efficiency with inference speed in KV cache design
- Implementing speculative decoding with correct rollback semantics
- Achieving competitive performance against mature engines while learning the stack
Key Learnings
- LLM inference internals: attention, KV caching, and decoding strategies
- CUDA programming and GPU memory hierarchy optimization
- Trade-offs between latency, throughput, and memory in production inference
Additional Links
Achievements
- Built inference engine exploring Paged KV Cache and Speculative Decoding
- Integrated FlashAttention and CUDA Graphs for GPU optimization