A rack of GPUs, a few notebooks, and one engineer’s fine-tuned model running in their own namespace is expensive hardware with ad hoc access on top. It becomes an AI platform when a second team can run a model in production on the same hardware, safely, without a hand-built path. What separates the two states is an infrastructure layer.
This blog post covers what that layer actually needs to do, how the pieces fit together, and where KKP’s AI Kit sits in that stack if you’re running on Kubernetes already.
Gartner expects that by 2030, intelligent agents will take on a growing share of infrastructure orchestration itself, deciding, allocating, and self-optimizing across cloud, edge, and on-premises environments while humans set the intent and guardrails. (Gartner, The Future of I&O 2030: The Impact of AI, cited via Kubermatic)
What is AI reference infrastructure?
AI reference infrastructure is the standard set of platform components between raw compute (GPUs, accelerators, bare metal) and the people who need to run AI workloads on it. Those people are data scientists, ML engineers, and the applications that consume model output.
Each piece on its own (a scheduler, a model server, a gateway, a secrets store) is well understood. Wiring them together correctly, repeatably, and securely is where in-house builds slow down. A reference architecture gives any team a consistent shape to stand up without re-deriving the same decisions from scratch.
At minimum, that shape covers:
- Multi-tenant compute orchestration: Teams and workloads share GPU pools through multi-tenant orchestration, without static carve-outs that leave capacity idle.
- GPU access: GPUs are exposed to pods directly, on bare metal with no hypervisor in between.
- GPU-aware scheduling: Placement decisions that understand accelerator topology, not just CPU and memory requests.
- Model serving and inference: Training, fine-tuning, and inference each get the request pattern they need from the serving layer.
- AI-aware networking: Inference traffic is routed on what is inside the request, such as the model asked for, the tenant’s token budget, and backend health, on top of the usual L4 and L7 rules.
- Credential and secrets management: Model API keys and provider credentials are stored centrally and rotated, so they never live in a values file.
- Self-service access: A data scientist gets a workspace without filing a ticket with the platform team.
Skip any one of these and the platform team does it by hand, per request. Reference infrastructure exists to remove that manual work.
Why this keeps stalling in practice
GPU procurement moves fast and pilots run quickly, but production stalls soon after, and the model is rarely what blocks it.
The idle-capacity numbers point the same way, with two caveats attached. Cast AI’s 2026 State of Kubernetes Optimization Report measured average GPU utilization of 5% across tens of thousands of its customers’ clusters on AWS, Azure, and GCP, in data running to April 2026. Its full-year 2025 figures for CPU and memory were 8% and 20%. Cast AI sells Kubernetes cost optimization, so the numbers come from a vendor’s own telemetry. A VentureBeat Research survey from July 2026 found that 83% of enterprises running their own GPUs report utilization at 50% or below, from a self-selected sample of 107 infrastructure leaders that VentureBeat itself says to read directionally. Gartner estimates that AI infrastructure adds $401 billion in new spending in 2026, so even a fraction of that sitting idle is a large number.
Three things usually break it:
Operational silos. Data science, platform, and security teams work in separate tools and separate environments. GPU capacity sits idle in one team’s allocation while another team queues for access, because there is no shared scheduling layer underneath.
The deployment bottleneck. Moving a model from a training notebook to a served endpoint is a manual handoff, and the environment on the other side of it rarely matches the one the model was trained in. Each handoff is a small project of its own.
That matches what deployment surveys have shown for years: in Algorithmia’s State of Enterprise ML report, 40% of companies said deploying a single model to production takes more than a month, and only 14% could do it in a week or less.
The governance gap. Without centralized policy enforcement, AI workloads spread across environments with no consistent audit trail. That is a compliance risk as well as a cost one.
All three are platform problems, which is why adding GPUs does not fix production AI on its own.
How the pieces fit together
If you’re already running Kubernetes for the rest of your infrastructure, the reference architecture doesn’t require a parallel stack. It extends the same control plane you already operate.
Some clarification on some of these, because “AI-ready” gets used loosely elsewhere:
Disaggregated inference splits an LLM serving workload into its prefill and decode phases and scales each independently. Prefill is compute-heavy and decode is memory-bandwidth-heavy, so pinning them together wastes whichever resource the other phase is not using. Separating them raises throughput on the same hardware.
Topology-aware gang scheduling matters for training jobs specifically. The default Kubernetes scheduler places pods independently, and a distributed training job that needs several GPU nodes to start in lockstep can deadlock while it waits for the last one. A gang scheduler places the whole job as a unit or holds it until it can.
GPU-aware load balancing means the gateway routes inference traffic based on real-time signals like GPU cache saturation and token rate, not just round-robin or connection count. That’s the difference between a network layer that happens to sit in front of AI workloads and one that’s actually AI-aware.
Trade-offs: building this yourself vs. a packaged reference stack
You can build all of this on upstream Kubernetes. Plenty of platform teams do, and it is the right path if you have the headcount and a multi-quarter timeline.
The work is integrating the NVIDIA GPU Operator, standing up and tuning an inference stack (a disaggregated one if your models need it), adopting and operating a gang scheduler such as Kueue or Volcano, running an AI gateway, and wiring credential rotation into the secrets tooling you already have. Each piece is doable. Doing all of them at once, while running the rest of your fleet, is where in-house builds slip or ship narrower than planned.
A packaged stack trades some flexibility for time. You give up choosing every component in exchange for a set that has been wired together and tested as a unit. If your requirements are unusual (a proprietary scheduler, an accelerator vendor with no operator), the DIY path is still the right call. For the common case of “we need this running for more than one team, in production, this quarter,” it usually is not.
Where KKP AI Kit fits
Kubermatic Kubernetes Platform handles the multi-tenant Kubernetes layer, and its Enterprise Edition ships several of those seven layers as applications in its default application catalog. Each one installs from the Applications tab in the KKP UI, at cluster creation or later, with default values you can edit. The NVIDIA GPU Operator covers GPU access, and Kueue queues training jobs and, with its wait-for-pods-ready setting on, admits them all-or-nothing.
Serving and fine-tuning come from AI Kit, which is the upstream AIKit project. Inference runs on LocalAI behind an OpenAI-compatible API, so existing client code needs no changes, and fine-tuning uses Unsloth. It runs on CPU-only and GPU-backed clusters, supports multi-modal models, and works air-gapped. The AIKit catalog page walks through the install.
The AI gateway is KubeLB, which ships agentgateway as an addon, an open-source proxy for LLM, MCP, and agent-to-agent traffic. KubeLB installs and pins it in both editions, and you configure gateways, backends and routes with agentgateway’s own resources. It routes requests by model and provider, applies auth and token rate limits, and can fail over between backends when passive health checking is configured.
Secrets go through SecureGuard, which combines OpenBao with the External Secrets Operator. It handles model and provider API keys alongside the rest of your credentials, with rotation triggered by events.
AI Kit is the piece that turns “a cluster with the GPU Operator installed” into “a served model endpoint.” It is a single-process serving path. It does not provide disaggregated prefill and decode serving, and CPU-only serving is a fallback that will not match GPU throughput beyond light inference. If your model size or latency budget calls for disaggregated serving, add that layer beside KKP and scope a pilot before committing to a serving pattern. The full reference architecture, covering all seven layers, is on the Kubermatic AI page.
FAQ
What’s the difference between AI infrastructure and an AI platform? Infrastructure is the compute, GPUs and network. A platform is the orchestration, scheduling, serving, and governance layer on top that makes that compute usable by more than one team without manual handoffs.
Do I need Kubernetes to use this reference architecture? No. The same layers, shared GPU scheduling, model serving, request-aware routing, and credential rotation, can be built on VMs or bare metal without it. Kubernetes is the most common foundation because multi-tenancy, autoscaling, and scheduling already exist there and do not need to be built from zero.
Does this work air-gapped? Yes. Air-gapped deployment is a documented capability of AI Kit and of KKP, and Kubermatic’s reference architecture deploys entirely on-premises or air-gapped.
What’s the fastest way to see this running? Talk to us about scoping a pilot against your own GPU footprint.
Photo by Steve A Johnson on Unsplash.
