Spot Instances Architecture: Interruption-Tolerant Kubernetes, GPU Training and Checkpointing (2026)

Every major cloud will rent you spare compute at a steep discount, on one condition: it can take the machine back. Teams that treat that condition as a rare outage tend to give up on the savings after the first bad week. Teams that treat it as a scheduled, expected input design for it, and routinely run most of their fleet at a fraction of on-demand cost. The difference is not luck or tooling brand. It is a coherent spot instances architecture in which every layer, from the instance request to the training loop, assumes the node can vanish with seconds to minutes of warning.

This matters more in 2026 than it did when spot was a batch-processing curiosity. GPU capacity is the most expensive line item in many cloud bills, Kubernetes has made fleets of short-lived nodes normal, and the economics reward anyone who can absorb interruptions without human intervention.

You will leave with a layered design: how interruption signals differ on AWS, Google Cloud and Azure; how to diversify capacity pools; how Kubernetes drains and replaces nodes; a workload fit matrix; a checkpointing cost model for GPU training; and runnable Karpenter and Python code.

What this covers: interruption signals per cloud, capacity-pool diversification, Kubernetes drain and disruption controls, workload fit, checkpoint math and code, on-demand fallback, failure modes, and observability.

Context and Background

Spot capacity exists because clouds provision for peak demand and sell the idle remainder. AWS describes Spot Instances as spare EC2 capacity offered at up to a 90 percent discount against On-Demand prices on its EC2 Spot page. Google Cloud advertises Spot VMs at up to 91 percent off for many machine types, GPUs, TPUs and Local SSDs, and describes them as the successor to preemptible VMs. Azure sells Spot Virtual Machines out of unused capacity and does not publish a fixed percentage, pointing customers to per-region and per-SKU pricing instead. Treat all headline discounts as ceilings: actual savings vary by instance family, region, zone and hour.

The vocabulary differs by provider but the contract is the same. The provider can reclaim the machine at any time, gives a short warning (sometimes none), and offers no availability SLA. Microsoft states plainly that Azure Spot VMs have no SLA and no high-availability guarantees. Google’s original preemptible instances were capped at 24 hours of runtime; Google now recommends Spot VMs, which have no maximum runtime unless you set one, and which share the same pricing model.

Pricing mechanics have also changed. AWS moved years ago from a bid model to a smoothed market price, so you no longer win or lose capacity by bidding higher; you simply accept the current price and the interruption risk. Azure still lets you set a maximum price in dollars per hour, and setting it to -1 means the VM is never evicted for price, only for capacity. The practical consequence is that cost is rarely what interrupts you any more. Capacity is.

For Kubernetes users the incumbents are the Cluster Autoscaler with managed node groups, and Karpenter, which provisions instances directly against the EC2 fleet APIs. We compared them in Karpenter vs Cluster Autoscaler for GPU node scaling; this post focuses on what changes once the nodes you scale are spot. For the wider cost picture, including right-sizing GPUs before you discount them, see Kubernetes cost optimization and GPU right-sizing.

The central thesis of this article is simple and, in my experience, under-appreciated: spot is a scheduling problem, not a pricing trick. The discount is the reward for giving the provider flexibility. You earn it by giving flexibility back to your own platform: flexibility in instance type, zone, time and state. Each section below converts one rigid assumption into a flexible one.

The Reference Architecture: Five Layers of Interruption Tolerance

A resilient spot design has five layers, and failing at any one of them negates the others. Capacity is requested broadly (layer one). Interruption signals are captured and normalised (layer two). The orchestrator drains and replaces nodes (layer three). Workloads are shaped to be restartable (layer four). A guaranteed baseline absorbs the worst case (layer five).

Figure 1: The five layers of a spot instances architecture. Signals flow upward from the cloud; replacement capacity flows downward from the provisioner.

Figure 1 shows the flow. The cloud emits an interruption signal. A handler converts the provider-specific signal into a Kubernetes action: cordon, drain, and request a replacement. Pods with graceful-shutdown logic checkpoint or hand off work. If replacement spot capacity is unavailable, the provisioner falls back to on-demand, and a small on-demand baseline keeps critical services alive during the gap.

Interruption signals differ, and the differences shape your design

On AWS, EC2 issues an interruption notice two minutes before it stops or terminates a Spot Instance, on a best-effort basis. The notice appears in instance metadata at /latest/meta-data/spot/instance-action, which returns HTTP 404 until an action is scheduled and then returns JSON such as {"action": "terminate", "time": "2017-09-18T08:22:00Z"} (the timestamp in AWS’s own example). AWS recommends polling every five seconds. The same notice is published to Amazon EventBridge as an event with detail-type EC2 Spot Instance Interruption Warning. One exception matters: for hibernation, the instance hibernates immediately and there is no two-minute lead time.

AWS also emits an earlier, softer signal called a rebalance recommendation, readable at /latest/meta-data/events/recommendations/rebalance and published to EventBridge as EC2 Instance Rebalance Recommendation. It indicates elevated interruption risk. Crucially, AWS notes it may arrive earlier than the two-minute notice but can also arrive together with it, so your shutdown path must work when both land at once. Karpenter’s documentation states that it does record rebalance events but does not taint, drain or terminate nodes in response to them, so if you want proactive migration you must build it or use Auto Scaling Capacity Rebalancing.

On Google Cloud, Spot VMs can be preempted at any time, and Compute Engine stops (the default) or deletes the VM according to the termination action you configure. Google’s documented model gives a short soft-off window, historically 30 seconds, delivered as an ACPI G2 soft-off signal that your operating system turns into a shutdown; a shutdown script can use that window. I could not re-confirm the exact figure on the live documentation during this run, so check the current Spot VM preemption section before you size your drain logic around it.

On Azure, an eviction can come with as little as 30 seconds of notice. You opt in to Azure Scheduled Events to receive an in-VM notification, delivered best-effort up to 30 seconds before eviction. Eviction is either capacity-driven (Azure needs the hardware back) or price-driven (the current price exceeds your maximum). The eviction policy you pick at creation matters: Deallocate, the default, keeps the VM and its disks, still counts against quota, and keeps billing for the disks; Delete removes the VM and its disks.

Cloud	Typical warning	Delivery channel	Notes
AWS	2 minutes (best effort)	IMDS, EventBridge	Earlier rebalance recommendation is optional and best effort
Google Cloud	about 30 seconds (verify current docs)	ACPI soft-off, shutdown script	Spot VMs have no 24-hour cap; legacy preemptible VMs do
Azure	up to 30 seconds (best effort)	Scheduled Events	Capacity or price eviction; Deallocate or Delete policy

The design implication is stark. A 120-second window allows a meaningful drain and a small checkpoint. A 30-second window does not. If you target Google Cloud or Azure for stateful or GPU work, you cannot rely on the warning to save state; you must save state continuously in the background and use the warning only to stop accepting new work and flush what is cheap to flush.

Capacity pools: the unit of risk is not the instance type

Spot capacity is organised into pools: a specific instance type in a specific availability zone (and, on some clouds, region and OS). Interruptions are not random per instance; they are driven by the shortfall in a pool. When the provider needs capacity back in a pool, it reclaims many instances from that pool at roughly the same time. This is the origin of the correlated-reclaim failure mode discussed later.

Diversification is therefore the first-order control. If you request only p4d.24xlarge in one zone, you have one pool and one correlated risk. If you allow dozens of instance types across three zones, you have dozens of pools whose reclaim events are mostly independent. AWS publishes guidance on being flexible across instance types, sizes, generations and zones, and the Karpenter pattern below turns that into a few lines of requirements. For a quick illustration (illustrative numbers, not measured), suppose a single pool has a 5 percent chance of a reclaim event in a given hour. With one pool, you face a 5 percent chance of losing everything. With ten independent pools and workload spread evenly, the chance of losing everything at once is vanishingly small, and the expected fraction lost in an event is only 10 percent of the fleet. Real pools are not fully independent, because families share underlying hardware and demand patterns, so diversify across families and generations, not just sizes.

The price-capacity-optimized lesson

Selecting the cheapest pool is a trap, because the cheapest pool is often the one the provider has the least spare capacity in, and therefore the one most likely to be reclaimed. AWS recommends the price-capacity-optimized allocation strategy for Spot Fleet, EC2 Fleet and Auto Scaling groups, which prefers pools with the deepest available capacity among low-priced options. Karpenter selects launch candidates with the same strategy against the EC2 fleet APIs. The takeaway is to optimise for expected cost including interruptions, not the sticker price. A pool that is 10 percent cheaper but gets reclaimed twice as often loses money once you count restart overhead.

Deeper Analysis: Kubernetes Spot Nodes, Workload Fit and GPU Checkpointing

How Kubernetes spot nodes handle an interruption

The sequence matters, so walk through it. The cloud emits the notice. Something has to hear it: either a per-node daemon such as the AWS Node Termination Handler in IMDS mode, or a queue-based controller. Karpenter takes the queue route. You point it at an Amazon SQS queue with the --interruption-queue argument, and EventBridge rules forward spot interruption warnings, scheduled maintenance events, instance terminating or stopping events and instance status-check failures to that queue. When a spot warning arrives, Karpenter cordons and drains the node while it launches a replacement in parallel, so replacement capacity is booting during the same two minutes the old pods are shutting down.

Figure 2: Sequence of a spot interruption on Kubernetes. Replacement launch and pod eviction run in parallel inside the two-minute window.

Draining uses the Kubernetes eviction API, which honours PodDisruptionBudgets (PDBs) and each pod’s terminationGracePeriodSeconds. Here is the first trap: a PDB protects against voluntary disruption, but a cloud reclaim is effectively involuntary. The eviction API will still refuse an eviction that violates a PDB, yet the instance will be terminated at the deadline regardless. A PDB cannot hold back a spot reclaim; it only shapes the order and pace of the drain before the deadline. If your PDB allows one pod at a time to leave and the node hosts five replicas of the same service, only a few will drain gracefully and the remainder will be killed hard when the instance disappears. The cure is topology spread, not a stricter PDB.

Use topologySpreadConstraints with topologyKey: kubernetes.io/hostname and zone keys so that replicas of one service never concentrate on one node, and keep maxSkew at 1. Add pod anti-affinity only for the strictest cases; spread constraints scale better. Combined with a PDB of maxUnavailable: 1 (or a percentage), reclaiming a single node removes at most one replica and the service keeps serving.

Priority classes complete the picture. When replacement capacity is slow, the scheduler must decide who gets the new nodes first. Give revenue-critical and stateful workloads a higher PriorityClass than batch and best-effort jobs, so that preemption inside the cluster and scheduling order both favour them. Lower-priority batch pods can be evicted to make room, which turns spot churn into an acceptable cost for them rather than an incident for everyone else.

Karpenter’s own disruption machinery adds a second source of node churn on top of cloud reclaim. Its consolidation feature replaces and removes nodes to reduce cost, expireAfter caps node lifetime (the documented default is 720 hours), and budgets limit how many nodes can be disrupted at once. These apply only to voluntary disruption: the karpenter.sh/do-not-disrupt pod annotation blocks consolidation and drift but, per the documentation, does not block expiration, interruption or manual deletion. That detail is the heart of the spot lesson: you can protect a long-running pod from your own autoscaler, but not from the cloud. Set terminationGracePeriod on the NodePool to bound how long a blocked drain can hang; after it expires, remaining pods are force-deleted and the instance is terminated.

A Karpenter NodePool for spot with on-demand fallback

The pattern below uses two NodePools. The spot pool has a higher weight, so Karpenter tries it first. The on-demand pool is a fallback with a lower weight and a hard limit so a spot shortage cannot silently turn into an uncapped on-demand bill. Karpenter’s documentation notes that when a NodePool allows both capacity types it prefers spot, but separate pools give you explicit control of limits, taints and weights. Instance families here are examples; adapt them to your region.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-general
spec:
  weight: 100
  template:
    metadata:
      labels:
        capacity: spot
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      taints:
        - key: capacity
          value: spot
          effect: PreferNoSchedule
      expireAfter: 168h
      terminationGracePeriod: 10m
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
    budgets:
      - nodes: "20%"
  limits:
    cpu: "2000"
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ondemand-fallback
spec:
  weight: 10
  template:
    metadata:
      labels:
        capacity: on-demand
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 5m
  limits:
    cpu: "200"

Three details deserve explanation. First, the broad requirements (three categories, two architectures, three zones, generations above four) let Karpenter choose from a large set of pools, which is diversification expressed as configuration. Arm64 only works if your images are multi-architecture, so drop it otherwise. Second, the on-demand pool’s limits.cpu of 200 against 2000 for spot is a deliberate circuit breaker: the fallback is a safety net with a ceiling, so a prolonged spot drought pages a human instead of burning budget. Third, a spot taint of PreferNoSchedule steers workloads that tolerate it toward spot while letting everything else spill over if needed. Use NoSchedule for strictness, at the cost of pending pods during spot shortages.

Note the version: the manifest uses the karpenter.sh/v1 API. Older tutorials use v1beta1 with disruption.expireAfter in a different place; if you copy a snippet, check the API version against your installed Karpenter.

Workload fit matrix

Not every workload belongs on spot. Fit is determined by three properties: how much state is lost on an abrupt kill, how long a restart takes, and whether the interruption window is enough to hand off. The matrix below summarises a conservative stance.

Workload class	Spot fit	Condition for safety	Primary tactic
Stateless web and API tiers	Good	3+ replicas across zones, PDB, readiness gates	Topology spread plus small on-demand baseline
Queue workers and batch ETL	Excellent	Idempotent tasks, visibility timeouts	Let the queue redeliver; no checkpoint needed
CI runners, build farms	Excellent	Ephemeral, cacheable	Retry on node loss
Stream processors	Moderate	Checkpointed state to durable store	Keep state backends off the node
Databases and stateful sets	Poor	Replication quorum survives loss	On-demand for quorum members; spot for read replicas only
Distributed GPU training	Good, with discipline	Frequent async checkpoints, elastic restart	Sections below
Latency-critical inference	Moderate	Warm spare capacity, fast model load	Mixed fleet with on-demand floor
Single-pet services, license-bound apps	Avoid	n/a	On-demand or Savings Plans

The important idea is that spot suitability is a property you can engineer. Queue-based workers are naturally safe because the queue already treats workers as disposable. A database becomes spot-tolerant only when replication means any single node is replaceable, and even then you rarely want a majority of the quorum on a single pool. Inference sits in the middle: model load times of tens of seconds to minutes make cold replacements expensive, so keep warm headroom.

Checkpoint economics: how often is often enough

For GPU training, the spot question is: how much compute do you re-do after an interruption, and how much do you spend saving state to avoid it? Two costs trade off. Checkpoint more often and you pay more write overhead; checkpoint less often and each interruption loses more progress.

The classical answer is the Young-Daly approximation. If the time to write one checkpoint (the cost the training loop actually pays) is C seconds and the mean time between interruptions affecting the job is M seconds, the checkpoint interval that minimises expected waste is approximately the square root of 2 times C times M. Treat this as a first-order guide; it assumes exponentially distributed failures and constant checkpoint cost, neither of which holds exactly for spot fleets.

Worked example (illustrative numbers). A job runs on 64 GPUs across eight spot nodes. Suppose each node is interrupted on average once per 24 hours, so with eight nodes the job sees an interruption roughly every 3 hours, or M of about 10,800 seconds, assuming independence. Suppose a synchronous checkpoint stalls training for C = 90 seconds. The optimal interval is the square root of 2 x 90 x 10,800, which is about 1,394 seconds, a little over 23 minutes. Expected waste has two parts: checkpoint overhead of C divided by the interval, about 6.5 percent, and expected lost work of half the interval divided by M, about 6.5 percent. Total waste is about 13 percent of GPU time.

Now apply asynchronous checkpointing so the training loop only stalls for a short device-to-host copy, say C = 8 seconds, with the rest overlapped. The optimal interval drops to the square root of 2 x 8 x 10,800, about 416 seconds (roughly 7 minutes), and waste falls to about 1.9 percent for overhead plus about 1.9 percent for lost work, near 4 percent in total. This is the real payoff of async checkpoints: not that they are faster to write, but that a small stall lets you checkpoint often enough to shrink the loss window dramatically.

Add the restart cost. Every interruption also costs a reschedule, image pull, data loader warmup and checkpoint load; assume 6 minutes (360 seconds, illustrative). With one interruption every 3 hours, that adds about 3.3 percent. Total waste is then roughly 7 to 8 percent in the async case versus about 16 percent synchronous. Against a 65 to 70 percent price discount (illustrative; yours will vary), effective cost is about 0.32 to 0.38 of on-demand in the async case, a saving in the region of 60 to 68 percent. Notice how sensitive the result is to restart cost: halving restart time is worth more than most checkpoint tuning.

Figure 3: GPU spot training checkpointing loop. A background writer persists sharded state; a commit marker makes the checkpoint atomic; the SIGTERM handler triggers a final flush within the notice window.

A Python checkpoint and resume loop that reacts to SIGTERM

PyTorch’s distributed checkpoint package (torch.distributed.checkpoint) saves sharded state in parallel from every rank and supports loading into a different world size, which is exactly what you need when a spot fleet shrinks and grows. It provides save, load and an asynchronous async_save, and torch.distributed.checkpoint.state_dict provides get_state_dict and set_state_dict helpers for model and optimizer state. API details change between releases, so check the documentation for your installed version. The loop below follows the pattern; the atomic-commit and signal logic is the part that matters.

import os, signal, time, glob, shutil
import torch
import torch.distributed as dist
import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict, set_state_dict

CKPT_ROOT = os.environ.get("CKPT_ROOT", "/mnt/shared/ckpt")
SAVE_EVERY = int(os.environ.get("SAVE_EVERY", "400"))   # steps
KEEP = 3

stop_requested = False
def _on_term(signum, frame):
    global stop_requested
    stop_requested = True            # never do heavy work inside the handler
signal.signal(signal.SIGTERM, _on_term)

def step_dir(step): return os.path.join(CKPT_ROOT, f"step_{step:09d}")

def latest_committed():
    # only directories containing the COMMIT marker are trusted
    done = [d for d in sorted(glob.glob(os.path.join(CKPT_ROOT, "step_*")))
            if os.path.exists(os.path.join(d, "COMMIT"))]
    return done[-1] if done else None

def save(model, optim, step, pending=None):
    if pending is not None:
        pending.result()             # wait for the previous async save to finish
    d = step_dir(step)
    model_sd, optim_sd = get_state_dict(model, optim)
    state = {"model": model_sd, "optim": optim_sd, "meta": {"step": step}}
    fut = dcp.async_save(state, checkpoint_id=d)   # returns a future
    return fut, d

def commit(d):
    dist.barrier()                   # every rank finished writing
    if dist.get_rank() == 0:
        open(os.path.join(d, "COMMIT"), "w").close()
        for old in sorted(glob.glob(os.path.join(CKPT_ROOT, "step_*")))[:-KEEP]:
            shutil.rmtree(old, ignore_errors=True)
    dist.barrier()

def resume(model, optim):
    d = latest_committed()
    if d is None:
        return 0
    model_sd, optim_sd = get_state_dict(model, optim)
    state = {"model": model_sd, "optim": optim_sd, "meta": {"step": 0}}
    dcp.load(state, checkpoint_id=d)
    set_state_dict(model, optim, model_state_dict=state["model"],
                   optim_state_dict=state["optim"])
    return int(state["meta"]["step"])

def train(model, optim, loader, total_steps):
    dist.init_process_group("nccl")
    step = resume(model, optim)
    pending, pending_dir = None, None
    it = iter(loader)
    while step < total_steps:
        batch = next(it)
        loss = model(batch).mean()
        loss.backward(); optim.step(); optim.zero_grad(set_to_none=True)
        step += 1

        # Agree on the stop flag across ranks, otherwise ranks diverge.
        flag = torch.tensor([1 if stop_requested else 0], device="cuda")
        dist.all_reduce(flag, op=dist.ReduceOp.MAX)
        must_stop = bool(flag.item())

        if step % SAVE_EVERY == 0 or must_stop:
            if pending is not None:
                pending.result(); commit(pending_dir)
            pending, pending_dir = save(model, optim, step)
            if must_stop:
                pending.result(); commit(pending_dir)
                dist.destroy_process_group()
                raise SystemExit(0)   # exit cleanly so the scheduler restarts us

Several design choices are deliberate. The signal handler only sets a flag, because doing I/O or collectives inside a signal handler is unsafe. Ranks agree on the flag with an all-reduce, because SIGTERM may reach nodes at different instants and a rank that exits alone would hang the others in a collective. Checkpoints become valid only after a COMMIT marker is written, which is the defence against half-written checkpoints discussed in the failure modes. Retention keeps the last three committed checkpoints, so a corrupted newest one still leaves a fallback. On a 30-second warning, the final synchronous flush may not finish; that is acceptable only because periodic async checkpoints already bound the loss.

For the checkpoint destination, local NVMe on the spot node is fast but dies with the node. Use shared storage (a network filesystem, or object storage through a fast client) for the committed copy, and treat local disk only as a staging tier.

Elastic training with torchrun

Checkpointing handles state; elasticity handles membership. PyTorch’s torchrun supports elastic launches where --nnodes takes a minimum and maximum, for example --nnodes=4:8, together with a rendezvous backend such as c10d and a --max-restarts budget. When a node leaves or joins, the workers are restarted from the last checkpoint with the new world size, which is why your resume path must tolerate a changed number of ranks (distributed checkpoint’s resharding on load is designed for this). I could not retrieve the current torchrun page body this run, so confirm flag names and semantics against the documentation for your PyTorch version.

torchrun --nnodes=4:8 --nproc-per-node=8 --max-restarts=20 \
  --rdzv-backend=c10d --rdzv-endpoint=$HEAD_ADDR:29400 --rdzv-id=job42 \
  train.py

Set --max-restarts high enough to survive a day of churn, but not unlimited; a crash loop on a genuine bug should fail the job, not burn GPUs forever. Distinguish interruption restarts from application errors in your logs so you can tell them apart in the retrospective.

Elasticity changes the maths. With fewer nodes the global batch changes, so either hold the global batch constant through gradient accumulation or adjust the learning rate schedule. A resumed run on 4 nodes instead of 8 must not silently train with half the effective batch. Record the data-loader position in the checkpoint so samples are neither repeated nor skipped; for deterministic sampling, store the sampler epoch and offset.

Mixing spot with on-demand and commitments

The cheapest architecture is rarely all-spot. A blended fleet uses three tiers: a committed baseline (Savings Plans or reserved capacity) sized for the always-on minimum, an on-demand layer for burst that cannot tolerate interruption, and spot for everything elastic. A useful rule of thumb is to commit to the 20th percentile of your hourly usage, run spot above that, and keep an on-demand floor large enough that losing every spot node in one pool still leaves the service within its latency objective.

Illustrative example: a service needs 100 vCPU-equivalents at peak and 40 at trough. Commit 35, hold 10 on-demand as the interruption floor, and run the rest on spot. At a hypothetical 65 percent spot discount and 30 percent commitment discount, blended cost is roughly 35 x 0.70 + 10 x 1.00 + 55 x 0.35 = 24.5 + 10 + 19.25 = 53.75 units against 100 for pure on-demand, a 46 percent saving at the peak with a floor that survives a pool reclaim. The all-spot version looks cheaper on paper (about 35 units) but offers no protection when capacity disappears. Pair this view with the right-sizing guidance in Kubernetes cost optimization and GPU right-sizing, because discounting an oversized GPU just makes waste cheaper.

Trade-offs, Gotchas, and What Goes Wrong

Spot architectures fail in a small number of repeatable ways. Knowing them in advance is most of the defence.

Correlated reclaim. The most damaging failure is many nodes disappearing together because they sat in the same pool. The notice windows are per instance, so a hundred simultaneous warnings means a hundred drains competing for the same eviction budget, the same registry bandwidth and the same replacement capacity. Mitigation is structural: spread across families, sizes and zones; cap per-pool concentration with topology spread; and keep the on-demand floor. Do not rely on the average interruption rate in provider dashboards, which describes a fleet-wide mean and hides pool-level bursts.

Replacement scarcity. The pool that was just reclaimed is likely to be short for a while, and your replacement request lands in the same neighbourhood. With broad requirements the provisioner can move to other pools; with narrow requirements it retries the same exhausted pool and pods sit pending. Watch pending-pod age, not just node count.

Checkpoint corruption. A node dying mid-write leaves a partial directory that looks like a checkpoint. A loader that picks the newest directory by name will load garbage or crash, and then every restart crashes the same way. The remedy is the commit marker pattern in the code above: write to a temporary or uncommitted location, verify, then publish atomically, and keep several generations. Checksum large shards if your storage layer does not, and test restores regularly; a checkpoint you have never restored is a hypothesis.

Thundering herd on restart. When a whole job restarts, every rank pulls the same container image, the same dataset shards and the same checkpoint at once. Object storage and registries throttle, so a 5-minute restart becomes a 25-minute one. Pre-pull images with a node image cache or a pull-through registry, stagger dataset reads, and consider local caching for checkpoint shards that many ranks need.

Stuck drains and the PDB deadlock. A tight PDB or a do-not-disrupt annotation can block your own voluntary drains indefinitely, leaving old nodes alive, while involuntary reclaims ignore them. Use terminationGracePeriod on the NodePool and an expireAfter so nothing lives forever, and reserve the strongest protections for pods that genuinely cannot restart.

Warning-window mismatch. Teams port a two-minute AWS drain design to a 30-second environment and discover their cleanup never completes. Design for the shortest window across clouds you use, and make everything idempotent so a hard kill is harmless.

Silent savings erosion. If the fallback pool has no limit, a prolonged spot shortage converts quietly to on-demand spend. The savings you advertised internally evaporate without a single alert. Limits, budgets and a dashboard for the spot ratio close that gap.

Figure 4: Failure modes in a spot fleet mapped to the signal that detects each one and the mitigation that contains it.

Observability: what to measure

Instrument four families of signals. First, supply health: interruption count per pool per hour, rebalance recommendations, provisioning latency from pending pod to ready node, and insufficient-capacity errors from the provisioner. Second, workload health: pods evicted per interruption, pods that exceeded their grace period and were killed hard, and requests failed during drains. Third, training health: time since the last committed checkpoint, checkpoint duration, restart count, and goodput, meaning useful steps per paid GPU-hour. Fourth, financial health: the ratio of spot to on-demand hours, realised discount against on-demand list price, and cost per useful training step.

Goodput is the one metric that keeps the whole exercise honest. A job that runs at 70 percent of ideal throughput on a 65 percent discount is still a good deal; a job that spends half its time restarting is not, whatever the invoice says. Alert when time since last committed checkpoint exceeds twice your target interval, because that indicates a stuck or failing writer long before an interruption exposes it.

Practical Recommendations

Start with the workloads where spot is nearly free: queue workers, CI runners and stateless services with several replicas. Prove the plumbing there, because every lesson you learn (signal handling, drain timing, replacement latency) transfers to harder workloads at low risk.

Then build the layers in order. Broaden your instance requirements before anything else, since diversification delivers more resilience per hour of effort than any other change. Add an interruption signal path (the SQS queue and EventBridge rules for Karpenter, or a node termination handler elsewhere). Set topology spread and PDBs on every service you move. Add the on-demand fallback pool with a limit. Only then move stateful and GPU workloads, with checkpoints and restore tests in place first.

For GPU training, decide the checkpoint interval from measured numbers rather than habit. Measure how long the training loop is actually stalled by a checkpoint, estimate your fleet’s interruption interval from a week of data, compute the Young-Daly interval as a starting point, then tune against goodput. Use asynchronous saves, shared storage, atomic commits and a restart budget. Rehearse a forced interruption monthly, including a simultaneous loss of several nodes.

Keep the cost view broader than the discount. Commit to a baseline with Savings Plans or reserved capacity, hold a small on-demand floor, and run elastic demand on spot. Revisit the mix quarterly. For carbon-aware or time-shifted scheduling that pairs naturally with interruptible compute, see FinOps and GreenOps carbon-aware scheduling.

A checklist to adopt:

[ ] Instance requirements span at least 10 instance types across 3 zones, with generations and families broadened.
[ ] Interruption events reach the cluster (SQS plus EventBridge for Karpenter, or a termination handler) and are tested by simulating an interruption.
[ ] Every spot-hosted service has topology spread, a PDB and a graceful shutdown path shorter than the shortest cloud warning.
[ ] An on-demand fallback NodePool exists with a hard limit, a weight below spot and an alert on usage.
[ ] Training jobs use async, sharded checkpoints, an atomic commit marker, retention of at least three generations and a restore test.
[ ] Elastic restart is configured with a bounded restart budget, and global batch size is preserved across world-size changes.
[ ] Dashboards track interruption rate per pool, pending-pod age, time since last checkpoint, goodput and realised discount.
[ ] A game-day simulating correlated reclaim runs at least quarterly.

Frequently Asked Questions

How much notice do I get before a spot instance is interrupted?

It depends on the cloud. AWS provides a best-effort two-minute interruption notice through instance metadata and EventBridge, with an optional earlier rebalance recommendation. Azure can evict with as little as 30 seconds of notice, delivered through Scheduled Events if you opt in. Google Cloud’s documentation describes a short soft-off window, historically 30 seconds. Because notices are best effort, design workloads to survive an abrupt kill with no warning at all.

Are preemptible instances the same as spot instances?

Not exactly. Preemptible VMs are Google Cloud’s older product, limited to 24 hours of runtime, and Google now recommends Spot VMs instead, which have no maximum runtime unless you set one and share the same pricing model. In general conversation, people use preemptible, spot and low-priority interchangeably for interruptible discounted compute. Check the specific provider’s terms for runtime caps, notice periods and termination actions before assuming equivalence.

Is Kubernetes on spot nodes safe for production?

Yes, for the right workloads and with the right controls. Stateless services with multiple replicas spread across nodes and zones, protected by PodDisruptionBudgets and backed by a small on-demand floor, run well on spot. Databases and single-replica services do not. The safety comes from diversification, graceful drain, topology spread and fallback capacity, not from the discount itself. Test with simulated interruptions before moving anything critical.

How often should GPU training checkpoint on spot instances?

Use the Young-Daly approximation as a starting point: the interval is about the square root of two times the checkpoint stall time times the mean time between interruptions. With a stall of 8 seconds and an interruption every three hours, that is roughly 7 minutes (illustrative). Asynchronous checkpointing shrinks the stall and therefore the optimal interval. Then tune against measured goodput, including restart time, which is often the larger cost.

What happens to Karpenter nodes when a spot interruption arrives?

If the interruption queue is configured, Karpenter receives the warning through SQS, cordons and drains the node, and launches a replacement in parallel so new capacity boots while pods shut down. Rebalance recommendations are recorded as events but do not trigger a drain. The do-not-disrupt annotation does not stop an interruption, because it only blocks voluntary disruption such as consolidation and drift. Pods that miss their grace period are killed when the instance terminates.

Should I use Savings Plans or spot for GPU workloads?

Use both for different layers. Commit to the always-on baseline you are confident about, where a commitment discount applies with no interruption risk. Run elastic, restartable work such as training and batch on spot, where discounts can be much deeper. Keep a limited on-demand floor for anything that cannot tolerate loss. GPU spot availability varies widely by region and instance type, so verify capacity in your target pools before committing to a spot-only plan.

Spot and Preemptible Instance Architecture: Interruption-Tolerant Kubernetes, GPU Training and Checkpointing (2026)