Buoyant, the maintainers of Linkerd, announced the open-source release of Kinference, a Kubernetes control plane designed to manage vLLM inference fleets. Released under an Apache 2.0 license, the project acts as an orchestration layer sitting directly in front of model servers to handle dynamic request routing, per-tenant rate limits, and KV-cache optimization across multi-tenant infrastructure.
Kinference pairs a declarative traffic gateway with an inference handler that interfaces via gRPC to evaluate candidate worker endpoints. By leveraging dynamic cache event metrics and integrating NVIDIA’s dynamo-kv-router crate, the control plane calculates predicted time-to-first-token (TTFT) to balance incoming traffic based on semantic tags, GPU pool states, and queue depths.
In benchmarks conducted on Qwen3.8-27B-FP8, Buoyant reported that Kinference reduced median end-to-end latency by 45% on shared-prefix workloads compared to standard Kubernetes ClusterIP routing. Maximum goodput meeting predefined TTFT service-level objectives increased up to 7.4-fold, prioritizing backend policy governance for enterprise deployments.
Why it matters
Engineers deploying open-weight models at scale get an open-source control plane to significantly reduce TTFT and GPU memory thrashing.
Kinference provides declarative CRD-based governance for multi-tenant inference, allowing precise rate limiting and fallback policies per team.
Optimizing KV-cache awareness directly in Kubernetes infrastructure reduces hardware costs by dramatically boosting request throughput per GPU.
Source: marktechpost.com



