LLM KV Cache Systems Engineer

Job Description

About the role We are looking for an LLM KV Cache Systems Engineer to design and build KV Cache subsystems for large-language-model inference. You will work on memory allocation, lifecycle management, reuse, compression, transfer, tiering, and KV-aware scheduling across single-node and distributed serving systems. This is a systems role. Direct source-level experience with an LLM KV Cache subsystem is more important than familiarity with serving-framework APIs alone. What you will do • Design block/page allocators, KV Cache lifecycle management, fragmentation control, eviction, prefix caching, cross-request reuse, and preemption/recovery mechanisms. • Optimize KV layouts, GQA/MQA handling, quantization, compression, or sparsity while balancing accuracy, capacity, memory bandwidth, and compute overhead. • Build KV transfer and disaggregated-serving mechanisms, including prefill/decode separation, RDMA/NVLink/CXL transport, and GPU/CPU/SSD memory tiering or offload. • Develop KV-aware request routing, batching, admission control, scheduling, and multi-tenant isolation under latency and SLA constraints. • Integrate and productionize KV technologies in systems such as vLLM, SGLang, TensorRT-LLM, LMDeploy, Mooncake, or DistServe, with observability and failure handling. Show More