LLM serving stacks tackle cold starts with reusable weights and snapshots

Inference replicas do not add useful capacity the moment a container starts. Before serving a request, an engine may still need to read and transform model weights, initialize GPU resources, compile or tune kernels, capture execution graphs and warm the runtime.

A new CNCF engineering survey argues that the emerging fixes should not be treated as one interchangeable “fast start” feature. vLLM, SGLang, llm-d and several checkpoint projects preserve different pieces of startup state, with different portability and lifecycle constraints.

vLLM keeps processed weights alive

The most concrete vLLM mechanism is its merged Fast Start work and documented preload command. A daemon per GPU loads and post-processes a model shard, keeps the resulting weights resident in GPU memory, and exposes them to new engine processes through local CUDA inter-process communication.

According to the vLLM preload documentation, a compatible engine can map those allocations in zero-copy mode rather than rebuilding weights from disk. The engine and daemon compare configuration and model fingerprints before reuse; a mismatch or unavailable daemon can fall back to ordinary loading.

That design narrows a specific bottleneck rather than eliminating startup. The daemon must stay alive, CUDA IPC remains node-local, and the serving engine still has runtime work to perform. vLLM also documents preload-time FlashInfer autotuning so that a later engine can reuse persisted tuning results instead of paying that profiling cost on first start.

Snapshots preserve a different boundary

The CNCF survey separates reusable weights from initialized-engine snapshots. vLLM has an experimental same-machine snapshot path built around CRIU and CUDA checkpoint support, while SGLang has its own weight-cache implementation and snapshot proposals. These paths preserve more process state but impose tighter compatibility requirements around hosts, drivers, runtimes and GPU topology.

The distinction matters operationally. A weight daemon can accelerate a fresh engine while keeping model artifacts outside that engine’s lifetime. A snapshot restores a previously initialized execution state. Sleep-and-wake mechanisms retain a live server and release some GPU memory, which is a third lifecycle rather than another name for either approach.

Platforms still need orchestration contracts

llm-d documents multiple startup paths, including model-file caches, warm instances, peer-to-peer weight transfer and GKE Pod Snapshots. The CNCF post also points to Kubernetes checkpoint-and-restore work at the pod and container-runtime boundaries, but notes that orchestration and runtime implementation work is still in progress.

For platform teams, the useful question is therefore not simply whether a stack supports “fast startup.” It is what state survives, which component owns it, where that state can move, and what initialization remains after reuse. The merged vLLM Fast Start pull request and current documentation show that reusable processed weights are now a concrete mechanism; portable, production-grade restoration across heterogeneous GPU nodes remains a broader systems problem.