LoginSign Up
← Back to all sparks
K

KServe

AI-ASSISTANTS
Velocity2.5

Standardized model inference serving platform on Kubernetes

kserve.github.io/website ↗

KServe is rebuilding its control plane around disaggregated LLM serving.

kubernetesllm-servingdisaggregated-inferenceautoscalingkv-cachemulti-protocol
◆Current state
KServe's v0.18–v0.20 release cycle is a substantial overhaul of its LLM serving layer. The LLMInferenceService (llmisvc) API is now the canonical path for deploying large models on Kubernetes, with disaggregated prefill/decode support, autoscaling via WVA/KEDA/HPA, and native KV cache offloading. The older InferenceService continues in parallel for traditional model serving but is no longer the primary development focus. Multiple protocol compatibility — OpenAI, Anthropic Messages, gRPC — is now part of the default routing surface.
◆Where it's heading
Each release candidate is adding production-grade capabilities to the llmisvc: confidential model serving, LoRA adapter routing, traffic splitting, and multi-tier KV cache storage. The project is converging toward a GA-quality LLM serving platform built for multi-node, multi-GPU Kubernetes deployments. The pace of Envoy AI Gateway upgrades (v0.6 → v1.0) and llm-d component upgrades (v0.6 → v0.8) signals that the underlying infrastructure is stabilizing.
◆Prediction
The v0.21 release will likely deliver CRD stability improvements or a v1 designation for the llmisvc API, as the v0.20 cycle exhausted most of the beta feature surface and the v0.21-rc0 prep commit has already landed.

◆Recent moves

  1. 6d ago

    KServe v0.21.0: first release candidate opens

    An automated release-prep commit opens the v0.21.0 release candidate cycle — no user-visible changes, just the signed release orchestration commit. The candidate continues the llmisvc-heavy trajectory established across v0.18–v0.20.

    View source ↗
  2. 1mo ago

    KServe v0.20.0-rc1: TLS hardening for disaggregated inference

    The second v0.20.0 candidate tightens TLS handling in the disaggregated sidecar (swapping an old flag for the new one), fixes quote escaping in KV transfer config generation, and guards against reconcile panics when the v1alpha2 InferencePool CRD is absent. These are correctness fixes, not new capabilities — the feature surface from rc0 carries forward unchanged.

    View source ↗
  3. 2mo ago

    KServe v0.20.0-rc0: Anthropic API, KV cache tiering, LoRA routing

    ⚡ SPARK

    This release candidate lands the highest concentration of new LLM serving capabilities in the KServe 0.x series: Anthropic Messages API routing, CPU-tiered KV cache offloading, confidential model serving, Managed DRA support, and a traffic splitting API for controlled deployments. vLLM is now a first-class supported runtime, and LoRA affinity scoring auto-enables when adapters are present — routing is no longer manually configured.

    View source ↗
  4. 3mo ago

    KServe v0.19.0-rc0: LLM model caching and autoscaling lands

    ⚡ SPARK

    v0.19.0-rc0 ships LocalModelCache integration for LLMInferenceService — models can now be pre-staged on nodes before inference pods start — alongside WVA/KEDA/HPA autoscaling for LLM workloads. Static LoRA adapter reconciliation, dual-protocol REST/gRPC routing, and OpenAI Responses API support all land in the same candidate, establishing the autoscaling and protocol foundation that v0.20 builds on.

    View source ↗
  5. 4mo ago

    KServe v0.18.0-rc1: OpenAI Responses API and dual-protocol routing

    The second v0.18 candidate adds an /v1/responses HTTPRoute for OpenAI Responses API compatibility, namespace-scoped ModelCache, and scheduler TLS certificate reload — extending the API surface and improving operational stability. This builds on the autoscaling and storage migration foundation laid in rc0, refining the llmisvc before general availability.

    View source ↗
  6. 4mo ago

    KServe v0.18.0-rc0: autoscaling and namespace-scoped model cache

    The v0.18 release candidate introduces WVA/KEDA/HPA autoscaling for LLMInferenceService, namespace-scoped ModelCache, storage migration for llmisvc APIs, and distribution-specific build hooks — a broad capability expansion that sets the architecture KServe's subsequent v0.19 and v0.20 cycles refine. These features were further polished in rc1 before the final release.

    View source ↗