How Kubernetes is finally solving the GPU utilization crisis to save your AI budget

When I started working with Kubernetes over a decade ago, the conversations were about microservices, stateless web applications and horizontal pod autoscaling. Today, the conversation has fundamentally changed. Every architecture review I participate in now centers on one question: how do we orchestrate GPU-accelerated AI workloads at scale without burning through our budget?

The numbers tell a compelling story. According to IDC’s latest findings, global AI infrastructure spending surged 166% year-over-year in the second quarter of 2025, reaching $82 billion in a single quarter. By 2029, that figure is projected to hit $758 billion. At the heart of this infrastructure explosion sits Kubernetes — the orchestration layer that was never originally designed for GPUs but has become indispensable for running them.

Having spent 20 years in the IT industry, I’ve watched Kubernetes evolve from a container scheduler into the operating system of the AI era. But this transformation didn’t happen overnight, and the challenges it addresses are ones I see enterprise teams grapple with daily.

The GPU utilization crisis that Kubernetes is solving

Here’s a reality that doesn’t get enough attention in boardroom presentations: most enterprise GPU clusters are dramatically underutilized. Industry data consistently shows average GPU utilization hovering around 10–30% in many organizations. When you’re paying $2–$15 per GPU-hour in the cloud or investing millions in on-premises H100 clusters, those idle cycles represent an enormous financial drain.

The root cause is structural, not operational. Traditional Kubernetes treats GPUs as atomic resources. When a pod requests a GPU via nvidia.com/gpu:1, the scheduler allocates an entire physical GPU to that pod. There’s no native sharing mechanism — it’s binary. Consider a real production scenario: a quantized large language model running inference on an 80GB A100 might consume only 12GB of GPU memory and operate at 30–35% compute utilization. That’s 65–70% of an expensive accelerator sitting idle, yet Kubernetes considers it fully occupied.

This is the problem the Kubernetes ecosystem has been racing to solve, and 2025 has been a watershed year for progress. Production case studies from CNCF member organizations show that advanced GPU scheduling on Kubernetes can improve utilization from 13% to 37% — nearly tripling efficiency — with some implementations pushing past 80%. For an enterprise running hundreds of GPUs, that improvement can translate to millions of dollars in recaptured value annually.

The strategies making this possible include multi-instance GPU (MIG) for hardware-level partitioning on Ampere and newer architectures, multi-process service (MPS) for software-based sharing among latency-tolerant inference workloads, time-slicing for development environments and bin-packing algorithms that minimize GPU fragmentation across the cluster. The key insight I’ve seen in successful deployments is that organizations need to treat GPUs as a shared, policy-driven resource governed by queues rather than hand-assigning them to individual projects.

How Kubernetes scheduling evolved for AI training and inference

Training and inference represent fundamentally different challenges for Kubernetes, and the platform has had to develop distinct capabilities for each.

Distributed training jobs need what the community calls “gang scheduling” — the ability to launch all pods simultaneously or not at all. A training run using PyTorch distributed data parallel across eight GPUs is useless if only seven pods can be scheduled. The remaining pod blocks progress, and now seven GPUs are burning cycles waiting. The default Kubernetes scheduler was never designed for this all-or-nothing semantic, and it was one of the most painful gaps for early adopters running AI workloads on Kubernetes.

Two projects have emerged as the primary solutions. Kueue, a Kubernetes-native job queuing system, provides cluster-wide queues, tenant quotas with cohort borrowing and atomic admission control. When one team’s workloads are idle, other teams can temporarily consume those unused resources, and the system automatically returns capacity when the original owners need it. High-priority training runs can preempt lower-priority workloads, evicting all pods in a job simultaneously to maintain the gang semantics that AI workloads require.

NVIDIA’s KAI Scheduler, open-sourced under the Apache 2.0 license in 2025, takes this further with fractional GPU allocation, topology-aware scheduling and hierarchical queue management. Originally developed within Run:ai, it supports the entire AI lifecycle within a single cluster — from interactive Jupyter notebooks that need a fraction of a GPU to massive distributed training runs consuming entire racks of accelerators.

On the inference side, the challenges are different but equally consequential. Inference workloads are bursty and latency-sensitive. A recommendation engine might see 10x traffic spikes during peak hours, requiring rapid scaling. Kubernetes’ Horizontal Pod Autoscaler works here, but the traditional approach of allocating whole GPUs to inference pods creates massive waste during off-peak periods. This is where GPU partitioning strategies like MIG become critical — allowing multiple inference models to share a single physical GPU with hardware-level isolation, each getting guaranteed memory and compute slices.

Perhaps the most significant development is Kubernetes Dynamic Resource Allocation (DRA), which graduated to general availability in Kubernetes 1.34. DRA replaces the rigid device plugin model with a flexible framework where workloads can describe their hardware requirements declaratively. Instead of requesting a static count of GPUs, applications can specify the properties they need—GPU type, memory capacity, interconnect topology — and let the scheduler find the optimal placement. This is particularly transformative for environments with heterogeneous GPU fleets spanning multiple generations of hardware across H100, A100, L4 and Blackwell architectures.

Making GPU economics work at enterprise scale

The financial argument for getting Kubernetes GPU orchestration right is staggering. With NVIDIA H100 GPUs commanding $27,000–$40,000 per unit for purchase and $2–$5 per hour for cloud rental, even modest utilization improvements generate significant returns. The AI infrastructure market reached $50 billion in 2024 and is growing at roughly 35% annually, which means the cost of getting GPU management wrong compounds rapidly.

In my experience working with enterprise migration teams, the organizations achieving the best GPU economics share several practices. First, they implement queue-based admission control from day one. Rather than letting individual teams provision and hoard GPU nodes, they establish organizational queues with guaranteed quotas, borrowing policies and fair-share algorithms. This alone can boost effective utilization by 30–50% because idle resources are automatically redistributed.

Second, they match GPU partitioning strategies to workload profiles. Production inference with strict SLAs runs on MIG-partitioned instances for isolation. Development and experimentation use time-sliced GPUs where the cost of occasional latency jitter is acceptable. Large-scale training reserves full GPUs. This tiered approach prevents the common antipattern of every team requesting dedicated A100s for workloads that could run on a MIG slice or even a smaller accelerator.

Third, they embrace spot and preemptible instances for fault-tolerant workloads. Training jobs with proper checkpointing can safely run on spot GPUs at 50–80% discounts. Kubernetes’ taint and toleration mechanisms, combined with Kueue’s preemptible queue configurations, make this operationally manageable. I’ve seen teams cut their training compute costs in half simply by making checkpointing a policy requirement and routing appropriate workloads to spot capacity.

The topology dimension is equally important and often overlooked. Training runs that spread pods across network boundaries when they don’t need to will hit communication bottlenecks that waste GPU cycles waiting for data. Topology-aware scheduling — placing pods on GPUs connected via NVLink or InfiniBand when distributed training requires it — can dramatically reduce training time and improve overall GPU throughput. The KAI Scheduler’s topology-aware capabilities and DRA’s ComputeDomain abstraction for managing Multi-Node NVLink connectivity are direct responses to this challenge.

What comes next

Looking ahead, three trends will shape Kubernetes’ role in AI infrastructure. First, the convergence toward open GPU scheduling standards. Just as networking converged on CNI and storage on CSI, GPU resource management is moving toward standardized interfaces through DRA and the Container Device Interface. This reduces vendor lock-in and lets organizations manage heterogeneous accelerator fleets — including AMD, Intel and custom silicon — through a unified Kubernetes API.

Second, the rise of intelligent resource optimization. Static allocation created the GPU waste problem; dynamic, context-aware decisions will solve it. Production deployments using advanced GPU resource management are already achieving 70–80% utilization compared to the 20–30% baseline, representing 50–70% reductions in infrastructure spend. As these capabilities mature, expect GPU cost optimization to become as automated as CPU autoscaling is today.

Third, the blurring line between training and inference infrastructure. As techniques like continuous fine-tuning, reinforcement learning from human feedback and retrieval-augmented generation become standard, the rigid separation between “training clusters” and “inference clusters” will dissolve. Kubernetes — with its unified API, namespace isolation and policy enforcement — is uniquely positioned to manage this convergence.

For IT leaders navigating this transition, my advice is pragmatic: start with Kueue and two or three queue definitions. Configure the NVIDIA GPU Operator. Set up DCGM monitoring to understand your actual utilization. Watch the metrics for a month before making big architectural decisions. The organizations that are winning with AI at scale didn’t start by over-engineering their GPU infrastructure—they started by making GPU utilization visible and letting the data guide their investments.

Kubernetes wasn’t built for GPUs. But through the collective efforts of the CNCF community, hyperscalers and hardware vendors, it has become the platform that makes GPU-accelerated AI economically viable at enterprise scale. That’s not just a technical achievement — it’s the foundation for every organization’s AI ambitions.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?