Secure model inference changed Baseten, Oct 5, 2026

Baseten · Oct 5, 2026 · 2 added, 2 removed · seen by the weekly check

Added: KV cache: Cache storage depends on your inference engine and configuration. For example, Engine-Builder-LLM supports offloading KV cache to CPU memory .
Removed: KV cache: The attention KV cache remains in GPU memory during inference. Baseten does not persist it to disk and discards it when a replica restarts or scales down.
Added: Baseten offers self-hosted deployments for organizations that need control over their inference infrastructure and network boundary. The workload plane runs inside your VPC, while Baseten operates the control plane. Inference requests addressed to your own domain route directly to your workload plane within your VPC. To scope self-hosted inference, contact Baseten .
Removed: Baseten offers self-hosted deployments for organizations that need control over their inference infrastructure and network boundary. The workload plane runs inside your VPC, while Baseten operates the control plane. Inference requests route directly to your workload plane without passing through Baseten infrastructure. To scope self-hosted inference, talk to us .

4 paragraphs left out as page noise

Added (spacing or invisible characters only): Framework: the compliance programs your workloads are restricted to.
Added (spacing or invisible characters only): Region: the geographic regions where your workloads can run. The policy can't be overridden: a regional deployment in a region outside the policy fails at deploy time.
Removed (spacing or invisible characters only): Framework : the compliance programs your workloads are restricted to.
Removed (spacing or invisible characters only): Region : the geographic regions where your workloads can run. The policy can't be overridden: a regional deployment in a region outside the policy fails at deploy time.

About this change