| Scheduler / Tool | Best For | Fair-Sharing Model | Maturity |
|---|---|---|---|
| Default Kubernetes scheduler + NVIDIA device plugin | Simple whole-GPU allocation | None; first-come, first-served | Stable, widely used |
| Kueue | Batch job queueing across teams | Quotas and fair-share via ClusterQueues and Cohorts | CNCF project, actively developed |
| Volcano | Gang-scheduled distributed training | Queues with priority and preemption | CNCF graduated-track project |
| NVIDIA MIG | Partitioning one physical GPU into isolated slices | Hardware-level isolation, not a queue | Stable on Ampere and newer GPUs |
| Run:ai / vendor schedulers | Enterprise-wide GPU pooling and fractional sharing | Dynamic fractional allocation and swapping | Commercial, varies by vendor |
Why GPUs Need Their Own Scheduling Layer
Kubernetes was built around a resource model of CPU and memory, both of which are infinitely divisible in practice: a container can request 250 millicores or 2.3 gigabytes and the scheduler bin-packs accordingly. GPUs do not fit that model. Until fairly recently, a GPU was treated as an indivisible resource; a pod either got a whole physical GPU or it got none. That is a problem for AI workloads, because a small inference service or a lightweight fine-tuning job rarely needs an entire high-end accelerator’s memory and compute, and letting it hold one exclusively wastes capacity that could serve other teams.
Three developments have changed this over the past several release cycles: the device plugin framework that lets vendors expose specialized hardware to the scheduler, NVIDIA’s Multi-Instance GPU (MIG) capability that physically partitions a single die into isolated slices, and batch-oriented queueing systems like Kueue and Volcano that sit on top of the scheduler to manage fairness, quotas, and preemption across teams rather than leaving allocation to whichever pod happens to arrive first.
The Device Plugin Framework
Kubernetes exposes GPUs to workloads through the device plugin framework, a mechanism that lets a vendor-supplied daemon advertise specialized hardware as an extended resource, such as nvidia.com/gpu, without patching the Kubernetes API server itself. The NVIDIA device plugin runs as a DaemonSet on every GPU-equipped node, detects the physical devices, and reports them to kubelet so pods can request them in a resource limit just like CPU or memory.
The default behavior is coarse: a pod requesting nvidia.com/gpu: 1 gets exclusive use of one entire physical GPU, and there is no built-in concept of requesting “half a GPU” without additional configuration. That coarseness is intentional for workload isolation but expensive at scale, which is why partitioning and time-slicing features exist on top of the base device plugin.
Time-Slicing and Fractional GPU Access
For workloads that do not need hardware-level isolation, the NVIDIA device plugin supports a time-slicing configuration that lets multiple pods share a single physical GPU by oversubscribing it, with the driver context-switching between workloads. This increases utilization for bursty or low-intensity inference workloads, at the cost of no memory isolation between the sharing tenants, meaning a memory-hungry neighbor can still affect others sharing the same physical device.
Multi-Instance GPU (MIG): Hardware-Level Partitioning
MIG, available on NVIDIA’s Ampere generation and newer data-center GPUs such as the A100 and H100, takes a fundamentally different approach from time-slicing. Instead of software-level context switching, MIG physically partitions a GPU’s streaming multiprocessors and memory controllers into up to seven isolated instances, each behaving like an independent, smaller GPU with its own dedicated memory bandwidth and compute cores. A workload assigned to one MIG slice cannot see or interfere with another slice’s memory, which makes MIG suitable for genuinely multi-tenant environments where isolation guarantees matter, such as running several customers’ inference workloads on shared infrastructure.
The tradeoff is inflexibility: MIG partition sizes must be configured at the node level before pods are scheduled, and reconfiguring partition shapes typically requires draining and reconfiguring the node, so it works best for relatively stable, predictable workload mixes rather than highly dynamic ones.
Figure 1: GPU Sharing Strategies Compared
Exclusive whole-GPU allocation offers full isolation but low utilization for small jobs. Time-slicing improves utilization through software context-switching but shares memory bandwidth. MIG delivers hardware-isolated partitions at the cost of static, node-level configuration.
Queueing and Fair Sharing: Kueue and Volcano
Device plugins and MIG solve the problem of exposing and partitioning hardware, but neither answers a harder organizational question: when ten teams all submit GPU-hungry jobs to a shared cluster at the same time, who gets to run first, and how is capacity shared fairly over a day or a week? That is the job of a batch scheduling and queueing layer.
Kueue
Kueue is a Kubernetes-native job queueing system, incubated as part of the Kubernetes project itself, that introduces ClusterQueue and LocalQueue resources to define capacity pools and quotas per team or namespace. Jobs sit pending until quota and resource availability allow them to be admitted, and Kueue supports Cohorts that let multiple teams borrow idle capacity from each other’s quotas without permanently taking it away, a pattern often called “lending” that keeps utilization high without abandoning fairness.
Volcano
Volcano takes a scheduler-replacement approach rather than sitting alongside the default scheduler, adding gang scheduling (all-or-nothing pod admission so a distributed training job does not start with only half its workers placed), queue-based priorities, and preemption policies. Gang scheduling in particular matters for distributed training jobs using frameworks like PyTorch’s distributed data parallel, where a job that starts with three of eight required workers wastes GPU-hours sitting in a broken, partially initialized state rather than actually training.
| Mechanism | Solves | Layer |
|---|---|---|
| Device plugin | Exposing GPUs as a schedulable resource | Node / kubelet |
| MIG | Hardware-isolated GPU partitioning | Physical GPU firmware |
| Time-slicing | Software-level GPU oversubscription | Driver / device plugin config |
| Kueue | Quota-based fair sharing and job admission | Cluster-wide controller |
| Volcano | Gang scheduling and priority-based preemption | Scheduler replacement |
Common Mistakes in Kubernetes GPU Scheduling
Pitfall
Teams frequently request a whole GPU per pod by default, even for inference workloads using a fraction of available memory, because it is the simplest configuration to write. This quietly wastes a large share of accelerator capacity. Auditing actual GPU memory and compute utilization before committing to whole-GPU requests, and moving steady, predictable inference workloads onto MIG slices or time-sliced pools, recovers meaningful capacity.
Pitfall
Running distributed training jobs on the default Kubernetes scheduler without gang scheduling is a recurring mistake. Pods for a multi-worker job can be admitted piecemeal as capacity frees up, leaving a job half-started and holding GPUs idle while it waits on missing workers. Volcano’s gang scheduling, or Kueue’s equivalent all-or-nothing admission checks, close this gap.
Pitfall
Some platform teams configure MIG partitions once and never revisit them as workload mix shifts, leaving small MIG slices idle while larger jobs queue for full GPUs elsewhere on the same cluster. MIG partition shapes should be reviewed against actual workload demand periodically, not treated as a one-time setup step.
Expert Practice: Building a Fair-Share GPU Platform
Worked Example
A machine learning platform team supporting four research groups on a shared 64-GPU cluster started by installing the NVIDIA device plugin and confirming baseline whole-GPU scheduling worked correctly across all nodes. They then layered Kueue on top, creating one ClusterQueue per research group with a baseline quota proportional to each group’s budget contribution, and grouped all four queues into a single Cohort so idle quota from a group between experiments could be borrowed by others rather than sitting unused. For their distributed training jobs, which typically used eight workers per run, they adopted Volcano’s gang scheduling to guarantee jobs either started fully staffed or stayed queued, eliminating the partially initialized training runs that had previously wasted GPU-hours. Finally, they identified a class of small evaluation and inference jobs that never used more than a quarter of a GPU’s memory, and moved those onto MIG-partitioned nodes, quadrupling the number of concurrent evaluation jobs the same physical hardware could support. Over one quarter, this combination raised measured GPU utilization significantly without any group losing its guaranteed baseline capacity.
How This Fits the Broader AI Infrastructure Picture
GPU scheduling inside Kubernetes is only one layer of a much larger set of decisions about AI infrastructure. Before a platform team even reaches scheduling questions, they typically have to decide whether to buy accelerators outright or rely on renting versus owning GPUs, a decision shaped heavily by current GPU shortage economics and lead times. Many teams choosing not to own hardware turn to neocloud providers that specialize in GPU capacity rather than general-purpose cloud computing. Once hardware is secured, the scheduling layer described in this article determines how efficiently that capacity is actually used, and the choice between training and inference workloads changes the calculus further, since the two have very different utilization patterns as covered in our breakdown of inference versus training economics. Hardware-level constraints like HBM memory bandwidth also shape which jobs can safely share a MIG slice or a time-sliced GPU without starving each other. Teams that already run stateful workloads on the same Kubernetes clusters supporting these AI pipelines will recognize many of the same fairness and isolation tradeoffs discussed here, just applied to accelerators instead of storage.
Glossary
- Device plugin
- A Kubernetes extension mechanism that lets vendors expose specialized hardware, such as GPUs, as a schedulable resource without modifying the core API server.
- MIG (Multi-Instance GPU)
- An NVIDIA hardware feature that physically partitions a supported GPU into multiple isolated instances, each with dedicated compute and memory.
- Gang scheduling
- A scheduling policy that admits all pods of a distributed job together or not at all, preventing partially staffed jobs from holding resources idle.
- ClusterQueue
- A Kueue resource that defines a pool of quota-controlled capacity that jobs are admitted against.
- Time-slicing
- A software technique that lets multiple workloads share one physical GPU by context-switching between them, increasing utilization without hardware isolation.
- Preemption
- The act of evicting a lower-priority job’s pods to free resources for a higher-priority job waiting in queue.
Key Takeaways
- The device plugin framework is what makes GPUs schedulable in Kubernetes at all, exposing them as an extended resource.
- MIG delivers hardware-isolated GPU partitions but requires static, node-level configuration that is not easily changed on demand.
- Time-slicing improves utilization for bursty workloads but offers no memory isolation between sharing tenants.
- Kueue adds quota-based fair sharing and cohort-based capacity borrowing across teams sharing one cluster.
- Volcano’s gang scheduling prevents distributed training jobs from starting partially staffed and wasting GPU-hours.
- Whole-GPU requests by default are the most common source of wasted accelerator capacity in shared clusters.
- GPU scheduling decisions should be revisited periodically as workload mix shifts, not treated as a one-time setup.
FAQs
What is the difference between MIG and time-slicing on Kubernetes?
MIG physically partitions a GPU into isolated instances with dedicated memory and compute, guaranteeing isolation between tenants. Time-slicing shares one physical GPU across pods through software context-switching, improving utilization without hardware isolation, meaning workloads can still contend for the same memory bandwidth even though they take turns using compute.
Do I need Kueue or Volcano if I only run a few GPU jobs?
Probably not. Both tools solve multi-team fairness and queueing problems that matter once several groups compete for shared, limited GPU capacity. A small team running occasional jobs on dedicated GPUs can usually rely on the default scheduler and device plugin without adding a queueing layer.
Can Kubernetes schedule fractional GPUs without MIG?
Yes, through time-slicing configurations in the NVIDIA device plugin, which oversubscribe a physical GPU across multiple pods without hardware partitioning. This works for lightweight or bursty workloads but does not provide the memory isolation that MIG guarantees, so it is unsuitable for strict multi-tenant isolation requirements.
What is gang scheduling and why does distributed training need it?
Gang scheduling admits all pods belonging to a distributed job together, or not at all. Distributed training frameworks typically require every worker to be present before training can start, so admitting workers piecemeal leaves a job stuck in a partially initialized, non-productive state while still holding allocated GPUs.
Is Kueue a replacement for the default Kubernetes scheduler?
No. Kueue works alongside the default scheduler, managing job admission and quota enforcement before pods are handed off to the scheduler for actual placement. Volcano, by contrast, replaces the default scheduler entirely to add gang scheduling and priority-based preemption directly into the placement decision.
Which NVIDIA GPUs support MIG partitioning?
MIG is available starting with NVIDIA’s Ampere generation data-center GPUs, including the A100, and continues on newer generations such as the H100 and beyond. Consumer and older data-center GPU generations do not support MIG, so fractional access on those must rely on time-slicing instead.
How does fair-share quota borrowing work in Kueue?
Kueue groups multiple ClusterQueues into a Cohort, which lets a queue borrow another queue’s unused quota temporarily when that team is not fully using its allocation. Borrowed capacity is preemptible, meaning the lending team can reclaim it when their own jobs need it, keeping overall utilization high without permanently sacrificing guaranteed capacity.
Does GPU scheduling in Kubernetes work the same way across cloud providers?
The core mechanisms, device plugins, MIG, Kueue, and Volcano, are largely provider-agnostic since they are open-source Kubernetes ecosystem projects. What differs across providers is node provisioning speed, available GPU generations, and whether managed Kubernetes offerings pre-install or simplify configuring these components for you.
- Kubernetes Documentation, “Device Plugins,” kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
- NVIDIA, “Kubernetes Device Plugin for GPUs,” github.com/NVIDIA/k8s-device-plugin
- NVIDIA, “Multi-Instance GPU (MIG) User Guide,” docs.nvidia.com/datacenter/tesla/mig-user-guide/
- Kueue Project, “Kueue Documentation,” kueue.sigs.k8s.io
- Volcano Project, “Volcano Documentation,” volcano.sh
- CNCF, “Cloud Native Landscape: Scheduling and Orchestration,” landscape.cncf.io
