Multi-LoRA Serving Architecture: Serving Hundreds of Fine-Tuned Adapters on One GPU with vLLM and S-LoRA (2026)

Fine-tuning one model per customer is easy to promise and ruinous to operate. If every tenant gets a full copy of an 8-billion-parameter model, you pay for roughly 16 GB of weights per tenant in half precision, and most of those GPUs sit idle because any single tenant sends a trickle of traffic. Multi-LoRA serving removes that waste by keeping one copy of the base model in GPU memory and treating each tenant’s fine-tune as a small low-rank adapter that is swapped, cached and batched on demand. The adapters are tens or hundreds of megabytes, not gigabytes, and a well-built server can mix requests for different adapters inside the same forward pass.

This is no longer research. The S-LoRA paper laid out the design, and vLLM now exposes it through flags and an OpenAI-compatible API. But the details decide whether you get a 10x cost win or a latency disaster: memory fragmentation, cold-load stalls, rank mismatches, eviction churn and noisy neighbours.

What this covers: the LoRA math you need, why merging per tenant fails, adapter memory arithmetic, the S-LoRA and Punica designs, vLLM configuration, routing and caching, a runnable LRU simulator, isolation limits, and a cost model against dedicated deployments.

Context and Background

Low-Rank Adaptation, introduced by Hu et al. in 2021, freezes a pretrained weight matrix and learns a small additive update. For a frozen weight W of shape d_out x d_in, LoRA learns two matrices B (d_out x r) and A (r x d_in) with rank r far smaller than either dimension. The adapted layer computes y = W x + (alpha / r) * B A x, where alpha is a scaling hyperparameter. Only A and B are trained and stored per task. We covered when this beats QLoRA, full fine-tuning and distillation in LoRA vs QLoRA vs full fine-tuning vs distillation; this post assumes you already chose adapters and now have to serve them.

The serving problem appears the moment you have more than a handful of adapters. A SaaS product that offers per-customer tuning on support tickets, a platform that lets developers upload fine-tunes, or an enterprise with a model per business unit all end up with tens to thousands of adapters. The traffic distribution is nearly always heavy-tailed: a few adapters are hot, and a long tail is touched a few times per hour. That shape is exactly what dedicated deployments handle worst, because each replica pays full price for weights whether or not anyone is calling it.

The incumbents for single-model serving are covered in vLLM vs SGLang vs TensorRT-LLM. What they share, and what makes multi-LoRA possible, is continuous batching: the scheduler admits and retires requests at every decoding step rather than waiting for a whole batch to finish. If you need a refresher on that mechanism, see continuous batching in LLM inference. Multi-LoRA serving extends that scheduler with one more dimension: each request in the running batch may belong to a different adapter.

Two research systems defined the approach. Punica (Chen et al., arXiv 2310.18547) introduced a CUDA kernel design that batches GPU operations across different LoRA models so that only one copy of the base model is needed, and reports roughly 12x the throughput of the state-of-the-art serving systems it compared against, with about 2 ms of added latency per token. S-LoRA (Sheng et al., arXiv 2311.03285) added memory management and scheduling: it keeps adapters in host memory, moves the ones needed by running requests onto the GPU, and reports up to 4x throughput over HuggingFace PEFT and vLLM with basic LoRA support, while serving thousands of adapters. Those numbers come from the authors’ own setups and baselines of 2023; treat them as evidence the design works, not as a prediction for your hardware.

The Reference Architecture for Multi-LoRA Serving

A multi-LoRA server shares one base model across all tenants and stores each tenant’s low-rank adapter separately. A router names the adapter per request, the scheduler batches mixed-adapter requests together, and a tiered cache moves adapters between object storage, host RAM and GPU memory. The base matmul runs once per batch; the adapter math is a small batched side computation.

Figure 1: Reference multi-LoRA serving architecture. One base model, a bounded set of GPU adapter slots, a larger CPU cache, and an object store holding every adapter.

Figure 1 shows four layers. The request layer carries an adapter identifier, which in the vLLM API is simply the model field. The router and registry map that identifier to a replica, a storage location and metadata such as rank and target modules. The scheduler composes each decoding step from requests that may reference different adapters. The memory hierarchy holds the base weights permanently, a limited number of adapters in GPU slots, a larger set in pinned host memory, and the full catalogue in object storage.

Why merging per tenant does not scale

The first instinct is to merge. LoRA is algebraically W' = W + (alpha / r) * B A, so you can fold each adapter into the base weights and serve a normal model with zero extra inference cost. For a single adapter this is the right call: no extra kernels, no extra memory, identical latency to the base model.

For many adapters it collapses on three counts. Each merged model is a full copy of the weights, so memory scales linearly with tenants: 100 tenants on an 8B model in 16-bit precision need roughly 1.6 TB, which is 20 GPUs of 80 GB before any KV cache. Switching between merged variants means rewriting every weight matrix in GPU memory, which takes seconds and blocks the batch. And merging into a quantized base is lossy, because adding a full-precision delta into 4-bit weights forces a re-quantization that discards part of the adapter signal.

The unmerged path keeps W untouched and computes the adapter contribution as a side branch. That adds work per token, but the work is tiny: two skinny matrix multiplications through a rank of 8 to 64. The art is making that small work efficient when every row of the batch wants a different A and B.

The memory arithmetic that makes it work

Here is a worked example. The numbers below are illustrative, using the published Llama-3-8B-style shape: hidden size 4096, 32 layers, 8 key-value heads of dimension 128 (so a 1024-wide K and V projection), and an MLP intermediate size of 14336.

A LoRA on a projection with input width d_in and output width d_out costs r * (d_in + d_out) parameters. Summing across the seven linear projections in one transformer layer: query 4096+4096 = 8192, key 4096+1024 = 5120, value 5120, output 8192, gate 4096+14336 = 18432, up 18432, down 18432. The total is 81,920 per unit of rank. At rank 16 that is 1,310,720 parameters per layer, and across 32 layers about 41.9 million parameters. In 16-bit precision that is roughly 84 MB. At rank 64 it is about 336 MB. If you target only the attention projections, rank 16 drops to roughly 13.6 million parameters, about 27 MB.

Compare that with the 16 GB base. A rank-16 all-linear adapter is about 0.5 percent of the base size. One hundred such adapters occupy about 8.4 GB. That is a real number worth budgeting for, but it fits next to the base model on a single 80 GB card with room left for cache.

The competing consumer of GPU memory is the KV cache. For the same shape, each token stores 2 (K and V) x 32 layers x 1024 x 2 bytes = 131,072 bytes, which is 128 KiB per token in 16-bit precision. Eight gigabytes of adapters therefore costs you roughly 65,000 tokens of KV capacity. On an 80 GB card running at a 90 percent memory budget, after 16 GB of weights you have around 56 GB for everything else, or about 450,000 tokens of KV if adapters took nothing. Every adapter slot you reserve trades directly against concurrent context. This tension is the central design problem, and it is what S-LoRA’s unified paging addresses.

Heterogeneous batching: the core compute trick

Naively, a batch of 32 requests drawing on 32 different adapters would need 32 separate small matmul launches per layer per projection. That is launch-overhead death. The fix, shared by Punica and S-LoRA, is to batch the base computation densely and handle all adapters in one grouped kernel call.

Figure 2: The compute path. The base GEMM runs once on the whole batch; a gather-based shrink and expand pair applies each token’s own A and B, and the results are summed.

As Figure 2 shows, the activations for the whole mixed batch go through the base x W multiplication as one dense matrix operation, which is efficient and unchanged from single-model serving. In parallel, each token’s adapter index is used to gather its A matrix and project down to rank r (the shrink step), then its B matrix to project back up (the expand step), scaled by alpha / r. The two branches are added. Because r is small, the shrink output is a narrow matrix, and the kernel is memory-bound rather than compute-bound: its cost is dominated by reading adapter weights. Punica’s kernels are commonly described as BGMV (batched gather matrix-vector, aimed at decode where each request contributes one token) and SGMV (segmented gather matrix-multiply, aimed at prefill where a request contributes many tokens). The Punica abstract itself describes the kernel only as one that batches operations across different LoRA models, so treat the BGMV and SGMV names as terminology from the project’s code and later literature rather than from the abstract.

The practical consequence is that adapter overhead grows with the number of distinct adapters and their rank in a batch, not with the batch size alone. A batch of 64 requests all using one adapter reads that adapter’s weights once; a batch of 64 requests on 64 adapters reads 64 sets. This is why the adapter-diversity of your traffic matters for throughput, and why a single max-loras number hides a lot of behaviour.

Deeper Analysis: Unified Paging, vLLM Configuration, and the Cold-Load Path

S-LoRA unified paging

Adapters have an awkward property: they vary in size. A rank-8 adapter and a rank-64 adapter on the same base differ by 8x in footprint, and the KV cache of each running request grows by one token at a time. If you carve GPU memory into a fixed adapter region and a fixed KV region, you will be wrong in one direction or the other as traffic shifts. If you allocate adapters as contiguous tensors of varying size next to growing KV tensors, you fragment memory and eventually fail an allocation with plenty of total free space.

S-LoRA’s answer, called Unified Paging in the paper, extends the paged KV-cache idea to adapters. It manages adapter weights and KV cache in one memory pool made of fixed-size pages. An adapter is stored across pages; a request’s KV cache is stored across pages; both are allocated from the same free list. When a rank-64 adapter is evicted, its pages become available to any request that needs more KV space, and vice versa. Fragmentation is bounded because everything is page-sized, and the split between “adapter memory” and “cache memory” adapts to the workload instead of being tuned by hand.

Figure 3: Unified paging. Adapter weights and per-request KV cache are carved from the same page pool, so evicting an adapter frees pages that KV growth can immediately use.

The second piece of the S-LoRA design is its placement strategy. All adapters live in host main memory. Only the adapters needed by the requests currently in the running batch must be on the GPU, and the scheduler can prefetch adapters for queued requests while the current step executes, overlapping the PCIe transfer with compute. The paper also describes a tensor-parallel scheme for partitioning the LoRA computation across multiple GPUs in a way that adds little communication on top of the base model’s own tensor parallelism, plus custom kernels that read adapter weights directly from the non-contiguous paged layout. A key caveat: the S-LoRA reference implementation was built on a research serving stack. What most teams run in production is the idea reimplemented inside vLLM, whose memory model is a fixed set of GPU LoRA slots rather than a fully unified pool. Know which one you are deploying when you read S-LoRA’s numbers.

What vLLM actually gives you

vLLM’s LoRA documentation (verified against the current docs for this article) describes the main controls. You enable the feature with --enable-lora. You register adapters at startup with --lora-modules name=path, or the JSON form that also carries a base_model_name. You cap the maximum adapter rank with --max-lora-rank, and the docs advise setting it to the highest rank among your adapters rather than something much larger, because the slots are sized to that maximum. The engine also accepts max_loras (how many distinct adapters can be active in one batch, i.e. GPU slots) and max_cpu_loras (the host-side cache size); the CLI spellings are --max-loras and --max-cpu-loras in the versions I am familiar with, but the docs page I fetched only listed the parameter names, so confirm with vllm serve --help for your release. Clients pick an adapter by putting its registered name in the model field of an OpenAI-compatible request, and base-model requests run on the same server.

Two operational features matter at scale. First, runtime loading: with the environment variable VLLM_ALLOW_RUNTIME_LORA_UPDATING set to true, you can POST to /v1/load_lora_adapter with a lora_name and lora_path, and to /v1/unload_lora_adapter to remove one. The docs describe a load_inplace option to replace weights under the same name without interrupting inference. The docs explicitly warn that runtime updating carries security risk and should not be used in production unless the environment is isolated and trusted, which is a real constraint for multi-tenant platforms where tenants upload weights. Second, LoRA resolver plugins: when a request names a model that is not loaded, a resolver can fetch it on demand. A filesystem resolver (lora_filesystem_resolver in VLLM_PLUGINS, with VLLM_LORA_RESOLVER_CACHE_DIR) looks for a directory named after the requested adapter; a Hugging Face Hub resolver exists but the docs say remote downloads are not intended for production; and you can implement the LoRAResolver interface and register it for something like an S3 bucket. Resolvers also require runtime updating to be enabled.

The memory implication of max-loras is the one people miss. GPU slots are provisioned at the maximum rank, so setting --max-lora-rank 64 and --max-loras 32 reserves 32 slots sized for rank 64 even if every real adapter is rank 8. Using the illustrative Llama-3-8B-style numbers above, that is roughly 32 x 336 MB = 10.7 GB of slot memory, versus about 1.3 GB if the rank cap were 8. Mixed-rank fleets should either standardize rank or run separate pools per rank class.

Launching the server and calling a per-request adapter

The following is a runnable sketch. Treat model names and paths as placeholders; the flags are the ones documented above. Substitute your own adapter directories (each must contain the PEFT-format adapter_config.json and weights).

# Start vLLM with two adapters preloaded and room for 8 active LoRAs per batch
export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True   # only in an isolated, trusted environment
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --max-lora-rank 16 \
  --max-loras 8 \
  --max-cpu-loras 64 \
  --lora-modules tenant-a=/adapters/tenant-a tenant-b=/adapters/tenant-b \
  --gpu-memory-utilization 0.90

# client.py - per-request adapter selection via the OpenAI-compatible API
import requests, time

BASE = "http://localhost:8000"

def chat(adapter: str, prompt: str) -> str:
    t0 = time.perf_counter()
    r = requests.post(f"{BASE}/v1/chat/completions", json={
        "model": adapter,              # adapter name, or the base model id
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 64, "temperature": 0.0,
    }, timeout=120)
    r.raise_for_status()
    dt = (time.perf_counter() - t0) * 1000
    print(f"{adapter:10s} {dt:7.0f} ms")
    return r.json()["choices"][0]["message"]["content"]

def load_adapter(name: str, path: str) -> None:
    r = requests.post(f"{BASE}/v1/load_lora_adapter",
                      json={"lora_name": name, "lora_path": path}, timeout=60)
    r.raise_for_status()

if __name__ == "__main__":
    load_adapter("tenant-c", "/adapters/tenant-c")   # hot-load at runtime
    for a in ["tenant-a", "tenant-b", "tenant-c", "meta-llama/Llama-3.1-8B-Instruct"]:
        print(chat(a, "Summarize our refund policy in one sentence."))

Run the client from a concurrent pool rather than sequentially if you want to see heterogeneous batching in action: send 32 requests across 8 adapters at once and watch the server log for running-request and LoRA counts. Measure your own p50 and p99 time-to-first-token with the adapter warm versus cold; those two numbers are the main outputs of this experiment.

The cold-load path and routing

A request for an adapter in a GPU slot costs nothing extra beyond the adapter kernels. A request for an adapter in the host cache needs a host-to-device copy before it can join a batch; an adapter of 84 MB over a PCIe link that sustains on the order of 20 to 25 GB/s (an illustrative figure for PCIe Gen4 x16 in practice) takes a few milliseconds, which is small against typical time-to-first-token. A true cold miss is different: fetch from object storage, deserialize safetensors, possibly validate, stage to host, then copy. That can take hundreds of milliseconds to seconds depending on the store, the file size and whether the replica has to evict something first.

Figure 4: Three request paths in a multi-LoRA deployment. The router prefers replicas that already hold the adapter; the vLLM replica falls back to its CPU cache and then to the object store.

This is why the router matters as much as the engine. A naive round-robin spreads each tenant’s traffic across every replica, so every replica needs every hot adapter and your CPU caches thrash. Affinity routing does better: hash the adapter identifier to a small set of replicas (rendezvous or consistent hashing with two or three candidates), prefer a replica that reported the adapter as resident, and fall back to the least-loaded candidate when the preferred one is saturated. Maintain an adapter registry that records storage URI, checksum, rank, target modules, base model version, tenant owner and an active/disabled flag. The base-model version field is critical: an adapter trained against one base checkpoint will silently produce garbage on another, with no error from the loader if shapes match.

Warm-up policy is the other lever. Pre-load the top-N adapters by recent traffic at replica start, keep them pinned if your engine supports it, and let the long tail pay cold-start costs. Tell your product teams plainly that the first request from an idle tenant may be slower; some platforms surface that as an expected property of cheap tiers.

An adapter-cache LRU simulator

Before buying GPUs, simulate your cache. The following script models adapter popularity as a Zipf distribution, runs an LRU cache of a given slot count, and reports the hit rate. It uses only the standard library.

import random, bisect, collections

def zipf_weights(n, s):
    w = [1 / (i ** s) for i in range(1, n + 1)]
    t = sum(w)
    return [x / t for x in w]

def simulate(n_adapters=500, slots=32, s=1.1, requests=200_000, seed=7):
    rng = random.Random(seed)
    cum, acc = [], 0.0
    for x in zipf_weights(n_adapters, s):
        acc += x
        cum.append(acc)
    lru, hits = collections.OrderedDict(), 0
    for _ in range(requests):
        a = bisect.bisect_left(cum, rng.random() * cum[-1])
        if a in lru:
            hits += 1
            lru.move_to_end(a)
        else:
            if len(lru) >= slots:
                lru.popitem(last=False)
            lru[a] = True
    return hits / requests

if __name__ == "__main__":
    for s in (0.8, 1.1, 1.4):
        for slots in (8, 32, 128):
            print(f"zipf_s={s} slots={slots} "
                  f"hit_rate={simulate(slots=slots, s=s):.3f}")

I ran this with 500 adapters, 200,000 requests and seed 7. These are simulation outputs on a synthetic Zipf workload, not measurements of a real service. With a fairly flat popularity curve (s = 0.8) and 32 slots the hit rate was about 27 percent; with a moderately skewed curve (s = 1.1) it was about 55 percent; with a strongly skewed curve (s = 1.4) about 79 percent. Growing the cache to 128 slots lifted those to roughly 57, 78 and 92 percent. Two lessons follow. First, the skew of your real traffic dominates everything, so measure it from logs before choosing max-loras. Second, a request-level LRU is pessimistic for a batching engine, because slots are occupied by adapters in the running batch rather than by independent requests; the GPU slot count behaves more like a working-set limit, and the CPU cache absorbs the long tail. Use the simulator for the CPU tier and for sizing a fleet-wide warm set, and use live metrics for the GPU tier.

Quantized base models plus LoRA

Running adapters on a quantized base is the natural next step, because it cuts the 16 GB base to roughly 4 to 5 GB at 4-bit and frees memory for adapters and KV cache. The design rule is to keep the adapter branch in higher precision than the base and never merge. QLoRA, which trains adapters over a 4-bit base, is the usual source of such adapters; the unmerged serving path preserves what training saw. Two cautions apply. Support is not uniform: the vLLM LoRA page I fetched contained no quantization notes, so check the current compatibility matrix for your specific quantization format and kernel backend before committing. And an adapter trained against the full-precision base but served over a quantized one meets a slightly different function than it was tuned for; evaluate that exact combination, not the two parts separately.

Trade-offs, Gotchas, and What Goes Wrong

Throughput is not free. The adapter branch adds memory traffic proportional to the number of distinct adapters and their rank in each step. Punica reports about 2 ms of added latency per token in its setup, and S-LoRA claims small overhead at scale, but those are best-case figures on specific hardware. In practice, a batch dominated by many distinct high-rank adapters decodes visibly slower than a batch dominated by one. Benchmark with your own adapter-diversity distribution, not a single-adapter load test, which flatters the system.

Rank heterogeneity wastes slots. As computed earlier, slots are sized to the maximum rank. One tenant with a rank-128 adapter can force every slot to be 16x larger than the typical rank-8 adapter needs. Cap rank at registration time, or segregate high-rank tenants onto their own pool.

Cold starts are a tail-latency problem, not a mean problem. Mean latency looks fine because most requests hit warm adapters. The p99 is where a cold object-store fetch lands. Set a latency budget for cold loads, pre-warm for scheduled traffic spikes such as a customer’s morning shift, and expose a timeout so a stalled fetch cannot hold a request slot indefinitely.

Eviction churn can collapse the system. If the active working set exceeds the GPU slot count, adapters get evicted and reloaded every few steps. The engine spends time copying instead of computing, and throughput falls off a cliff rather than degrading smoothly. Watch the adapter load rate per second as a first-class metric and alarm when it rises.

Quality limits are real. A LoRA adapter is a low-rank update; it excels at style, format, domain vocabulary and narrow task behaviour, and is weaker at injecting large amounts of new factual knowledge or changing deep capabilities. Serving hundreds of adapters makes it tempting to treat them as interchangeable, but each one needs its own evaluation against its own held-out set. There is no interference between adapters at inference time in the unmerged design, since each token only sees its own adapter, but there can be interference in a different sense: tenants who all fine-tune on overlapping data can overfit in correlated ways, and a base-model upgrade invalidates every adapter at once. Plan adapter retraining as a fleet operation.

Isolation is logical, not physical. All tenants share one process, one base model and one GPU memory space. A correctness bug in an adapter kernel or in the indexing that maps tokens to adapters is a cross-tenant data-leak class of bug, not just a wrong-answer bug. Shared KV caching adds another risk: automatic prefix caching must be keyed by adapter, or two tenants with different adapters could reuse each other’s cached prefix states and produce wrong outputs or leak content. Verify that your engine version keys prefix-cache entries on the adapter identity, and treat runtime upload of untrusted weights as a security boundary. The vLLM docs’ own warning about runtime updating is the starting point, not an exaggeration. Safetensors loading avoids arbitrary code execution from pickle files, which is the minimum bar; you should also validate shapes, rank and target modules before admission.

Noisy neighbours. One tenant sending a burst of long prompts consumes batch tokens and KV pages that other tenants need. Continuous batching shares the GPU fairly in the throughput sense but not in the latency sense. Apply per-tenant concurrency limits and token-rate limits at the router, and consider priority classes so a paying tier is not queued behind free-tier prefill storms. If one tenant’s contract demands guaranteed latency, give them a dedicated replica: the architecture allows mixing pooled and dedicated deployments behind the same router.

Alternatives exist. LoRAX, an open-source project from Predibase, targets the same problem; its public description centres on dynamic adapter loading, tiered weight caching and heterogeneous continuous batching. I have not re-verified its current feature set or release state for this article, so evaluate it against vLLM on your workload rather than on either project’s marketing. The decision usually comes down to which engine already serves your base model, and what operational tooling you need around it.

Cost Model: Pooled Multi-LoRA Versus Dedicated Deployments

Consider an illustrative scenario with entirely hypothetical numbers: 200 tenants, each averaging 2 requests per minute during business hours, 500 input tokens and 150 output tokens per request, on an 8B-class base model. Assume a GPU costs $2.50 per hour. The dedicated approach runs one replica per tenant for availability and isolation: 200 GPUs, or $500 per hour, regardless of traffic. Even if you run each on a smaller shared card, the idle fraction is enormous, because 2 requests per minute is about 400 tokens of output per minute against a GPU capable of thousands of tokens per second.

Pooled: total load is 400 requests per minute, about 60,000 output tokens per minute or 1,000 tokens per second, plus prefill. A single modern 80 GB-class GPU running an 8B model can plausibly sustain that order of throughput, though you should measure your own number. With headroom for peaks, redundancy and a 2x safety factor, say 4 GPUs behind affinity routing: $10 per hour. The pooled design is cheaper by a factor of roughly 50 in this toy scenario, driven almost entirely by utilization. The factor shrinks fast as per-tenant traffic grows. If each tenant needs sustained 20 requests per second, each tenant fills a GPU on its own and pooling saves little; the crossover is where a tenant’s steady demand approaches one GPU’s capacity. Add the cost of the extra engineering: registry, router, adapter pipeline, evaluation and monitoring. For fewer than about ten adapters with steady traffic, static merges on dedicated replicas are often simpler and faster.

A useful rule of thumb that follows from the arithmetic: pooled multi-LoRA wins when you have many adapters, each with low or bursty utilization, on a shared base model. It loses when adapters target different base models, need guaranteed isolation, or each carry steady high load.

Practical Recommendations

Start by asking whether you actually need hundreds of adapters. If you have two or three, merge them and deploy normal replicas; the unmerged path costs throughput you do not need to spend. If you have dozens to thousands with a long-tail traffic shape, adopt multi-LoRA serving and design around the cache, not around the kernels.

Standardize adapter rank and target modules across tenants so slots are uniformly sized, and fix the base checkpoint version in the registry. Measure real traffic skew from logs, feed it into the simulator above, and size the CPU cache to hold the warm set with margin. Put an affinity router in front of replicas, and report adapter residency from each replica so the router can prefer warm placements. Keep the number of distinct adapters per batch within max-loras, and treat the adapter load rate as an alarm metric.

Treat tenant weights as untrusted input. Admit adapters through a pipeline that checks format, shape, rank, base version and an evaluation score before they go live, and keep runtime hot-loading restricted to an isolated control path. Re-run per-tenant evaluations after every base or engine upgrade.

• Confirm adapters share one base checkpoint and a bounded rank, and record both in the registry.
• Set --max-lora-rank to the true maximum, not a comfortable guess.
• Size GPU slots from the working set and CPU cache from the Zipf-style simulation.
• Route by adapter affinity with fallback to least-loaded replicas.
• Pin or pre-warm the top adapters by recent traffic.
• Verify prefix caching is keyed by adapter identity.
• Rate-limit per tenant, and give latency-critical tenants dedicated replicas.
• Alarm on adapter load rate, cold-load p99 and eviction churn.
• Load-test with realistic adapter diversity, not a single adapter.
• Plan a fleet-wide retrain path for base model upgrades.

Frequently Asked Questions

What is multi-LoRA serving?

Multi-LoRA serving is an inference design where one copy of a base language model stays in GPU memory and many small LoRA adapters, one per task or tenant, are loaded, cached and applied on demand. Requests for different adapters can share a batch. It replaces one full model per customer with one base plus lightweight low-rank deltas, which cuts GPU cost sharply when traffic per adapter is low or bursty.

How many LoRA adapters can one GPU serve?

It depends on rank, targeted modules, base size and cache design. By the illustrative arithmetic in this article, a rank-16 all-linear adapter on an 8B-class model is about 84 MB, so a hundred adapters is around 8 GB. The S-LoRA paper reports serving thousands of adapters on one GPU by keeping them in host memory and moving only active ones to the GPU. Practical limits are active adapters per batch and cold-load latency, not raw storage.

Does vLLM support multiple LoRA adapters per request batch?

Yes. vLLM supports LoRA with the --enable-lora flag, registers adapters with --lora-modules, and lets clients choose one through the model field. The engine can apply different adapters to different requests in the same batch, bounded by the maximum number of active LoRAs configured. Runtime loading and unloading is available through dedicated endpoints behind an environment flag, which the documentation says to avoid in production unless the environment is isolated and trusted.

What is the difference between S-LoRA and Punica?

Punica contributed a CUDA kernel design that batches LoRA computation across different adapters so one base model copy suffices, plus a cluster scheduler. S-LoRA built a fuller serving system: Unified Paging for adapter weights and KV cache in one pool, host-memory adapter storage with on-demand fetch, a tensor-parallel scheme for adapters, and custom kernels. S-LoRA reports up to 4x throughput over HuggingFace PEFT and vLLM with basic LoRA support in its comparison.

Should I merge LoRA weights or serve them unmerged?

Merge when you serve one or a few adapters on dedicated replicas: you pay no extra inference cost and the model behaves like any other. Serve unmerged when you have many adapters sharing a base, because merging multiplies memory by the number of tenants and makes switching slow. Merging into a quantized base also loses adapter precision through re-quantization, which is another reason to keep the adapter branch separate.

How do I isolate tenants in a multi-LoRA deployment?

Isolation is logical rather than physical. Enforce per-tenant rate and concurrency limits at the router, key any prefix caching by adapter identity, validate uploaded adapters for format, shape and base version before admission, and keep runtime loading on a trusted control plane. Give tenants with strict latency or compliance needs dedicated replicas. Remember that all tenants share a process and GPU memory, so kernel or indexing bugs carry cross-tenant risk.

Further Reading

• LoRA vs QLoRA vs full fine-tuning vs distillation for choosing the adaptation method before you serve it.
• vLLM vs SGLang vs TensorRT-LLM for engine selection and the baseline serving trade-offs.
• Continuous batching in LLM inference for the scheduler mechanics multi-LoRA builds on.
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters (arXiv 2311.03285)
• Punica: Multi-Tenant LoRA Serving (arXiv 2310.18547)
• vLLM documentation: LoRA adapters

By Riju — about