Multi-LoRA Serving: Architecture for Thousands of Adapters

Give every customer their own fine-tuned model and the economics collapse almost immediately. A thousand tenants means a thousand fine-tunes, and if each one is a full copy of an 8B model you are staring at sixteen terabytes of weights and a fleet of idle GPUs. Multi-LoRA serving is the architecture that makes this tractable: instead of a thousand models, you host one shared base model plus a thousand tiny low-rank adapters, each a few tens of megabytes, and you swap them in and out of a single GPU on demand. The hard part is not storing the adapters — it is serving a batch where every request needs a different one without collapsing to batch-size-one. That single constraint drives the whole design, from custom CUDA kernels to unified memory paging.

What this covers: the LoRA math and why adapters are small, why naive per-adapter hosting fails, how heterogeneous batching and unified paging work, the S-LoRA and Punica kernels behind it, adapter lifecycle and routing, and the multi-tenant cost trade-offs that decide when to merge instead.

Context and Background

Low-Rank Adaptation, introduced by Hu et al. in 2021, freezes the pretrained weights and learns a small update. For a weight matrix W0 of shape d×k, LoRA represents the update as a product of two thin matrices: the effective weight becomes W = W0 + B·A, where B is d×r, A is r×k, and the rank r is far smaller than d or k. Only A and B are trained. The base model never changes.

The parameter count of one adapted matrix is therefore r·(d+k), not d·k. That ratio is the entire reason multi-tenant serving is possible. Fine-tuning stops producing a new multi-gigabyte checkpoint and starts producing a delta measured in megabytes. You can read the original argument in the LoRA paper; the practical upshot is that an adapter is a rounding error next to the base weights.

That smallness invites an obvious but wrong deployment: give each fine-tune its own model server. Merge the adapter into the base weights, load the merged checkpoint, and serve it like any other model. This works for one or ten adapters. It fails at a thousand, because each merged model is again a full copy of the base — you have thrown away the size advantage the moment you merged.

The alternative — keep adapters separate and share one base — runs into the batching wall. Modern LLM serving lives and dies on batching, the same principle behind continuous batching for LLM inference: amortize one expensive weight load across many requests. But if request one needs adapter A and request two needs adapter B, a naive implementation must run them separately, and batching evaporates. Solving that contradiction is what a real multi-LoRA serving system does.

How multi-LoRA serving works

Multi-LoRA serving runs one shared base model in GPU memory and keeps a pool of small LoRA adapters that are loaded, batched, and evicted on demand. A router maps each incoming request to its adapter, a scheduler groups mixed-adapter requests into a single batch, and custom kernels apply the correct low-rank delta per request so the whole batch runs in one forward pass instead of many.

The base model computation is shared and unchanged. Every request in a batch flows through the same frozen W0 for attention and MLP projections. The only per-request difference is the additive term B·A, and that term is cheap. So the architecture splits each linear layer into two paths: the big shared base matmul that every request uses identically, and a small per-adapter LoRA matmul that differs by request.

Figure 1: Multi-LoRA serving request flow. A mixed batch hits the adapter router, which looks up each request’s adapter in the registry, then the batch runs through the shared base weights and the per-request adapters in one fused forward pass. The diagram shows requests fanning into a router, the registry resolving adapter identities, and the base-plus-adapter compute converging into a single batched output stream.

Request routing and the adapter registry

Before any batching happens, each request has to be resolved to an adapter. That is the registry’s job. Every adapter is enrolled under a stable identifier — a name plus a globally unique id — and the request carries that id, not the weights.

The router looks up the id, checks whether the adapter is already resident in HBM, and hands the scheduler an adapter index for the batched kernel. If the adapter is warm, this is a pointer lookup. If it is cold, the router triggers a fetch from DRAM or storage and the request waits behind that transfer.

Routing also shapes batching quality. A scheduler that is adapter-aware can co-schedule requests that share an adapter, raising the kernel’s operational intensity, or deliberately spread distinct adapters across a batch up to the max_loras limit. This is where multi-tenant fairness lives: a single tenant flooding requests should not evict every other tenant’s warm adapter, so production routers add per-adapter admission and quota controls on top of the raw lookup.

Why naive batching collapses to batch-size-one

Picture the honest implementation. The batch has eight requests using eight different adapters. For the base matmul, you stack all eight hidden states and do one big matrix multiply — fine, the base is shared. But the LoRA term needs a different B·A per request.

If you loop over the batch and apply each adapter separately, you have serialized the LoRA path. Eight small matmuls, eight kernel launches, no batching. On a GPU that hates small ops, this is death by a thousand launches. The base matmul is efficient and the adapter path is a stall.

The fix is a batched gather-matmul: a single kernel that, given a batch of hidden states and a per-request adapter index, gathers the right B and A for each row and computes all the low-rank deltas together. This is the core kernel innovation, and it is exactly what Punica’s SGMV and S-LoRA’s MBGMM/MBGMV provide.

Heterogeneous batching and rank

Real adapter fleets are not uniform. One tenant trained a rank-8 adapter, another rank-16, another rank-64. The batching kernel has to cope with adapters of different ranks in the same batch without padding everything to the maximum rank, which would waste both memory and compute.

S-LoRA handles this by storing adapter weights in a non-contiguous, ragged layout and letting the kernel index into it per request. Ranks can differ across the batch; the kernel gathers the correct slice for each row. This heterogeneous batching is what lets a single server host adapters trained by different teams with different hyperparameters, all live at once.

Unified paging of adapters and KV cache

The second big idea is memory management. GPU HBM is scarce and shared between three hungry consumers: the base weights, the KV cache for in-flight sequences, and the active adapters. These pools grow and shrink dynamically — sequences finish, new adapters get requested — and static allocation fragments the memory badly.

Figure 2: Unified paging tiers adapters between host DRAM and GPU HBM. All adapters live in host memory; active ones are prefetched into a unified HBM pool shared with the KV cache, and idle adapter pages are evicted back to DRAM. The figure traces an adapter from DRAM residence, through prefetch into the shared pool alongside KV-cache pages, into batched compute, and back out on eviction.

S-LoRA’s answer is Unified Paging: manage adapter weights and KV-cache tensors in one shared, paged memory pool rather than two fixed arenas. Because both are dynamic and both have variable sizes — adapters vary by rank, KV cache varies by sequence length — pooling them cuts fragmentation and lets one pressure absorb the other’s slack. It is the same paging insight behind PagedAttention, generalized to adapter weights.

The tiering underneath is host-DRAM-to-GPU-HBM. All adapters live in main memory, which is plentiful and cheap. The system fetches the adapters that active queries need into HBM and evicts the ones that go cold. A thousand adapters at 32MB each is 32GB of host DRAM — trivial — while only the handful currently in flight occupy precious HBM. That asymmetry is what makes serving thousands of adapters on one GPU real rather than aspirational.

A worked memory example

Numbers make the case concrete. Take a Llama-3-8B-class base with hidden dimension d = 4096 and 32 transformer layers. Apply LoRA at rank r = 16 to the four attention projections — query, key, value, output — each a 4096×4096 matrix.

Per projection, the adapter parameter count is r·(d+k) = 16·(4096+4096) = 131,072. Four projections per layer gives 524,288 parameters. Across 32 layers that is 16,777,216 parameters — call it 16.8M. Stored in fp16 at two bytes each, the adapter is 33.5MB.

Now scale it. One thousand such adapters occupy about 33GB in host DRAM. Contrast the naive alternative: a thousand full 8B models at roughly 16GB each in fp16 is 16TB. The adapter approach is three orders of magnitude smaller in aggregate, and only a few adapters sit in HBM at any instant. Extend LoRA to every linear layer including the MLP and the per-adapter size climbs to roughly 90MB, still negligible against the 16GB base. This is the arithmetic that turns per-tenant fine-tuning from a fantasy into a line item.

S-LoRA, Punica and the batched kernels

Two 2023 systems defined this space, and both center on a custom GPU kernel that batches many different adapters into one operation. The base model is trivially shared; the research problem was making the per-adapter path efficient when every request in the batch wants a different adapter.

Punica introduced SGMV — Segmented Gather Matrix-Vector multiplication. The Punica paper frames the goal precisely: run a batch where each request uses a different LoRA on a single shared base with negligible overhead versus running the base alone. SGMV parallelizes the feature-weight multiply across the batch and groups requests that share an adapter to raise operational intensity and hit the Tensor Cores.

The striking Punica result is that batching different adapters costs almost the same as batching the same adapter. The kernel gathers each request’s adapter weights by a segment index, so a batch of sixteen requests spanning sixteen adapters runs in one launch, not sixteen. Punica reports up to roughly 12x throughput over systems that lacked this multi-tenant batching; treat that as a paper-reported figure against its specific baselines, not a universal constant.

Figure 3: SGMV batches many different adapters in one kernel. Hidden states are segmented by adapter id, each segment gathers its own low-rank weights, and a single gather-matmul produces the combined delta added back to the shared base output. The figure shows a batch split into per-adapter segments feeding one SGMV op, whose low-rank deltas merge into a single batched output.

S-LoRA’s MBGMM and MBGMV kernels

S-LoRA generalizes the kernel idea across both inference phases. The S-LoRA paper uses Multi-size Batched Gather Matrix-Matrix multiplication (MBGMM) in the prefill stage, where many tokens are processed at once, and Multi-size Batched Gather Matrix-Vector multiplication (MBGMV) in the decode stage, where one token per sequence is generated.

The “multi-size” qualifier is the heterogeneous-rank capability: the kernels batch LoRA computations with varying ranks and sequence lengths in a non-contiguous layout, so you never pad to a common rank. Prefill and decode have different arithmetic shapes — matrix-matrix versus matrix-vector — which is why S-LoRA ships two kernels rather than one.

S-LoRA reports serving thousands of adapters on a single GPU, and up to roughly 4x throughput over strong baselines like HuggingFace PEFT and vLLM’s earlier LoRA path, while increasing the number of servable adapters by orders of magnitude. The LMSYS write-up on S-LoRA is a readable companion to the paper. Again, read those multipliers as reported ranges tied to specific hardware and baselines.

There is a scheduling layer above the kernels that matters just as much. S-LoRA clusters requests by adapter and prefetches adapter weights while the current batch is still computing, so the host-to-device transfer overlaps with useful GPU work instead of stalling in front of it. This adapter clustering raises the odds that a fetched adapter serves many queued requests before it is evicted, amortizing the transfer cost. It also has a tensor-parallelism dimension: the paper introduces a parallelism scheme for the added LoRA computation that keeps communication overhead small when the base model is already sharded across GPUs. The kernels make a mixed-adapter batch cheap; the scheduler and prefetcher keep the adapters that batch needs from ever being cold.

How vLLM, TGI and TensorRT-LLM expose it

These research kernels are now production features. vLLM exposes multi-LoRA through a small set of knobs: enable_lora=True turns it on, max_loras caps how many distinct adapters can be active in a single batch, max_lora_rank sets the maximum rank the server will accept, and max_cpu_loras sizes the host-side LRU cache of swapped-out adapters. Requests carry a LoRARequest naming the adapter, and the vLLM LoRA documentation is the reference.

The relationship between those flags encodes the whole architecture. max_loras is the HBM working set — how many adapters batch together at once. max_cpu_loras is the DRAM tier — how many stay warm in host memory before eviction. The constraint that max_cpu_loras exceed max_loras is just the tiering made explicit. Hugging Face TGI exposes a similar model through a LORA_ADAPTERS list, and NVIDIA TensorRT-LLM supports multi-LoRA with its own batched kernels and adapter cache.

S-LoRA vs Punica vs a merged model

The three approaches trade differently across memory, batching, and multi-tenancy. The table below is a decision matrix, not a benchmark; the throughput column reflects the papers’ own reported ranges against their baselines.

Dimension	S-LoRA	Punica	Merged single model
Base copies in HBM	One shared	One shared	One per adapter
Adapter batching	MBGMM prefill, MBGMV decode	SGMV single kernel	Not applicable
Heterogeneous ranks	Yes, non-contiguous layout	Yes, segmented gather	Baked in, fixed
Memory tiering	Unified paging, DRAM to HBM	On-demand adapter load	Whole model resident
Multi-tenancy	Thousands of adapters	Many concurrent adapters	Breaks — one tenant per model
Reported throughput	Up to ~4x vs PEFT and vLLM baseline	Up to ~12x vs non-batched baselines	Fast per model, no sharing
Best fit	Large, dynamic adapter fleets	High-concurrency multi-tenant	One or few hot adapters

The merged-model column is the honest baseline. Merging B·A into W0 produces standard weights with zero per-request overhead — the LoRA term vanishes because it is folded in. That is the fastest possible path for a single adapter. But merging destroys sharing: every merged model is a full base copy, so the merged column is exactly the naive approach that multi-LoRA serving exists to avoid. Merge when you have one dominant adapter and latency is everything; keep adapters separate when you have many.

Trade-offs, Gotchas, and What Goes Wrong

The first failure mode is cold-start latency. An adapter not in HBM must be fetched from host DRAM, and a truly cold one may come from object storage or disk. That transfer sits on the critical path of the first request, so a tenant whose adapter was just evicted eats a latency spike. LRU eviction is a heuristic, and adversarial or bursty traffic across many rare adapters can thrash the cache, turning a serving system into a paging system.

The size of that spike depends on which tier the adapter falls to. A 32MB adapter promoted from DRAM over a fast PCIe or NVLink path is a sub-millisecond-to-few-millisecond transfer — usually hidden if the scheduler prefetches while the previous batch runs. The same adapter fetched cold from object storage is tens to hundreds of milliseconds of network plus deserialization, which dominates the request’s time-to-first-token. This is why the DRAM tier matters so much: keeping the long tail warm in host memory converts a storage round-trip into a cheap bus copy. Under-sizing max_cpu_loras is the single most common cause of surprise tail latency in a multi-tenant fine-tuned deployment.

Figure 4: Adapter lifecycle. An adapter is registered and resident in DRAM, loaded into the HBM pool on request, active while batched, then LRU-evicted back to DRAM when idle — with a cache-hit fast path that skips the load. The figure shows the state machine including the cache-hit shortcut and the eviction loop back to DRAM residence.

Rank heterogeneity is the second gotcha. Batching a rank-8 adapter with a rank-64 one is supported, but the kernel efficiency and the memory footprint skew toward the largest rank present. A single high-rank adapter in an otherwise low-rank batch can drag down the batch’s arithmetic intensity, so mixing wildly different ranks in the same server is a real capacity-planning variable, not a free feature.

Tail latency is the third. Multi-LoRA serving optimizes aggregate throughput across a fleet, but per-tenant p99 depends on whether that tenant’s adapter is warm. A popular adapter stays in HBM and is fast; a long-tail adapter requested once an hour is perpetually cold. This produces a bimodal latency distribution that a single average hides, and it is why per-adapter latency SLOs are harder to hit than per-model ones.

The throughput-versus-latency picture also inverts with load. A merged model wins single-request latency because it has no LoRA term at all, but it cannot share a GPU with other tenants’ fine-tunes, so its GPU sits idle whenever that one tenant is quiet. Shared multi-LoRA serving accepts a small per-request overhead in exchange for packing many tenants’ traffic onto one accelerator, which is the throughput and cost-per-tenant win. At high aggregate load across many adapters, the shared server keeps the GPU busy; at low load on a single hot adapter, the merged replica is both faster and simpler. The crossover is a utilization question, not a correctness one.

When should you merge instead? When an adapter is hot enough to justify a dedicated replica, merging removes all per-request LoRA overhead and gives you a plain, maximally-optimized model — the same logic that governs fine-tuning versus RAG versus long context trade-offs. The rule of thumb: shared multi-LoRA for the long tail of many low-traffic adapters, merged dedicated models for the few high-traffic ones. Mixing both tiers in one deployment is common and correct.

Practical Recommendations

Start by classifying your adapter fleet by traffic. A small number of hot adapters and a long tail of cold ones is the typical shape, and it maps cleanly onto a two-tier deployment: merged dedicated replicas for the hot few, a shared multi-LoRA server for the tail. Do not force everything into one mode.

Size the two memory tiers deliberately. max_loras (or the equivalent) governs the HBM working set and interacts directly with KV-cache capacity, so raising it steals memory from batch size and sequence length. max_cpu_loras governs how many adapters stay warm in DRAM; set it generously because host memory is cheap and it directly reduces cold-start misses.

Constrain rank at intake. Allowing arbitrary ranks complicates batching and capacity planning, so most production fleets standardize on one or two ranks — commonly 8, 16, or 32 — and reject or re-train outliers. Uniform rank makes the batched kernels predictable.

Instrument the things that actually break: per-adapter cache hit rate, cold-start latency, and the distribution of active adapters per batch. A falling hit rate or a widening p99 signals cache thrash before users complain.

Checklist before you ship multi-LoRA serving:

Split the fleet into hot (merge) and long-tail (shared) tiers.
Standardize adapter rank at intake; reject unbounded ranks.
Size max_loras against KV-cache needs, not in isolation.
Set max_cpu_loras large to keep the tail warm in DRAM.
Measure per-adapter p99, not just aggregate throughput.
Alert on cache hit rate and cold-start latency, per adapter.
Load-test with a realistic long-tail adapter distribution, not one hot adapter.

Frequently Asked Questions

What is multi-LoRA serving?

Multi-LoRA serving is an inference architecture that runs many fine-tuned LoRA adapters on top of a single shared base model. Instead of hosting a full model copy per fine-tune, the server keeps one frozen base in GPU memory and swaps small low-rank adapters in and out, batching requests that use different adapters into one forward pass. Systems like S-LoRA and Punica make this efficient with custom batched kernels.

How many LoRA adapters can one GPU serve?

S-LoRA reports serving thousands of concurrent adapters on a single GPU, because only the base model and the currently active adapters occupy scarce HBM while the full fleet resides in cheaper host DRAM. The practical ceiling depends on base model size, KV-cache demand, and how many adapters must be active in one batch. The bound is usually the HBM working set of active adapters, not the total adapter count in DRAM.

Why not just merge the LoRA adapter into the base model?

Merging folds B·A into the base weights, giving zero per-request overhead — ideal for a single hot adapter. But a merged model is a full base copy, so merging a thousand adapters means a thousand full models and terabytes of weights. Merging destroys the memory sharing that multi-tenant serving depends on. Merge the few hot adapters; keep the long tail shared and unmerged.

What is SGMV in Punica?

SGMV is Segmented Gather Matrix-Vector multiplication, the CUDA kernel at the heart of Punica. It batches the low-rank LoRA computation for many different adapters into one GPU operation by segmenting the batch by adapter id and gathering each request’s adapter weights. Crucially, batching different adapters costs almost the same as batching the same adapter, which is what makes multi-tenant LoRA serving efficient rather than serialized.

How does S-LoRA manage GPU memory for adapters?

S-LoRA uses Unified Paging, a shared paged memory pool that holds both dynamic adapter weights and KV-cache tensors instead of two fixed arenas. All adapters live in host DRAM; active ones are prefetched into HBM and idle ones evicted, using an LRU-style policy. Pooling adapters and KV cache together reduces fragmentation because both are variable-sized and dynamic, so one pool absorbs the other’s pressure.

Does multi-LoRA serving slow down inference versus a single model?

There is a small per-request overhead from the extra low-rank matmul, but the batched kernels keep it close to negligible — Punica and S-LoRA are designed so a mixed-adapter batch runs almost as fast as the base alone. The real cost is cold-start latency when an adapter must be fetched into HBM, and a bimodal tail where rare adapters are slower. For hot single adapters, a merged model is still faster.

Multi-LoRA Serving: Architecture for Thousands of Adapters (2026)