LLM Gateway Comparison: LiteLLM vs Portkey vs Kong AI Gateway vs Cloudflare AI Gateway (2026)

The first time a team ships an LLM feature, it calls one provider directly with one API key. The second time, finance asks who spent $9,000 last Tuesday, security asks where the key lives, and an upstream outage takes the feature down for ninety minutes. At that point the question stops being “which model” and becomes “which control point sits between my code and every model.” That control point is the LLM gateway.

Four products dominate the shortlist in late 2026: the LiteLLM proxy, Portkey, Kong AI Gateway, and Cloudflare AI Gateway. They overlap on the headline features, yet they come from very different lineages: a Python SDK, an observability startup, an API-management incumbent, and a CDN. Those lineages decide what each does well, what is gated behind a license, and where it hurts in production.

This guide compares them on the dimensions that matter after the demo: OpenAI compatibility, routing and fallbacks, budgets, rate limits, caching, guardrail hooks, telemetry, licensing, security, and latency overhead. It ends with a decision matrix and minimal configs you can adapt.

What this covers: the shared reference architecture, how each product implements routing and budgets, an honest licensing and security review (including a 2026 supply-chain incident), a comparison matrix, a decision flow, and a practical checklist.

Context and Background

An LLM gateway is a reverse proxy that speaks an LLM API dialect, usually the OpenAI chat completions shape, and adds policy: authentication, quotas, retries, caching, logging, and routing across providers. The idea is old; API gateways have done auth, rate limiting, and observability for REST traffic for a decade. What is new is that the payload is expensive, slow, non-deterministic, and billed per token rather than per request.

That last point reshapes every classic gateway feature. A request-per-minute limit is nearly useless when one request can cost a fraction of a cent and the next can cost several dollars. Latency is measured in seconds, so a proxy that adds 20 milliseconds is noise, while one that buffers a streamed response is a user-visible regression. Retries are dangerous because a retried request is billed twice if the first one succeeded upstream but timed out locally.

The market also moved. A year ago most teams used a gateway as a convenience layer for swapping models. In 2026, with agents calling tools in loops and spend measured in the tens of thousands of dollars per month, the gateway has become a governance layer: who may call which model, with what budget, with which data allowed to leave. For the architectural pattern in depth, see our LLM gateway architecture guide; this post focuses on picking an implementation.

A useful frame is the split between the data plane, which forwards every request and must be fast and boring, and the control plane, which stores keys, budgets, and policies and can be slower. LiteLLM and Portkey ship both in one codebase. Kong has always separated them (Kong Gateway nodes versus Konnect or a database). Cloudflare hides both behind a managed service. Where the planes live determines your failure modes, which we return to later.

The standard external reference point is the OpenAI API specification, which became the de facto dialect; see the OpenAI API reference for the chat completions contract that all four gateways emulate. If your traffic is plain REST and you are weighing Kong against other API gateways first, our Kong vs Envoy Gateway vs Apache APISIX comparison covers that layer.

The Reference Architecture Every LLM Gateway Implements

Whatever the brand, every LLM gateway runs the same pipeline on each request: authenticate the caller, check limits and budget, run input guardrails, look in the cache, pick a target, call the provider with retries and fallbacks, run output guardrails, record usage, and emit telemetry. The products differ in which stages are built in, which are plugins, and which cost money.

Figure 1: The common LLM gateway request pipeline. Telemetry is a side channel from both the entry and the router.

The diagram shows the order that matters. Authentication and budget checks run before anything expensive, so a revoked key or an exhausted team budget fails in microseconds without touching a provider. Guardrails sit before the cache so that a poisoned prompt cannot be answered from a cached response meant for a different tenant, and again after the router so that provider output is inspected before it reaches the client. The router comes last because it is the only stage that has to know provider-specific facts: endpoints, auth schemes, model names, and which error codes are retryable.

Stage one: identity, keys, and the virtual key abstraction

The single most valuable feature of a gateway is the virtual key: a gateway-issued credential that maps to a team, a project, or an end user, plus limits. The real provider keys stay in the gateway. Developers, notebooks, and CI jobs only hold virtual keys, which can be revoked, capped, or rotated without touching the provider account.

LiteLLM issues virtual keys from its proxy, backed by Postgres, and supports users, teams, and per-key budgets in the open-source build. Portkey uses “virtual keys” and, in newer releases, a model catalog and integrations that reference provider credentials by ID. Kong delegates identity to its existing authentication plugins (key-auth, OIDC, JWT) and attaches AI policy to the consumer or route. Cloudflare offers a bring-your-own-keys store for provider credentials and gateway-level authentication tokens; its documentation lists 20-plus providers for stored keys and labels the feature beta at the time of writing.

Stage two: the data plane versus the control plane

LiteLLM’s proxy reads keys, budgets, and spend from a database on the request path (with in-memory caching), so Postgres availability is a real dependency. Kong data-plane nodes cache configuration and continue to forward traffic if the control plane is unreachable, a property inherited from its API-management design. Cloudflare’s gateway runs on its edge network, so there is nothing for you to operate, and nothing for you to debug when it misbehaves. Portkey’s open-source gateway is a lightweight stateless process configured per request or via config objects; the hosted platform adds the persistent control plane.

This is the first place to spend thought. A gateway that is a hard dependency of every AI call is, by construction, your new tier-zero service. Treat its availability budget accordingly: multiple replicas, health checks, and a documented bypass path.

Stage three: how a request actually becomes spend

Cost control is the reason most finance-driven gateway projects start. The mechanism is simple: the provider response includes token usage, the gateway multiplies it by a per-model price table, and it increments counters keyed by virtual key, team, and tag. Budgets are enforced by comparing the counter with a limit before the next request.

The subtlety is that the check happens before the request while the cost is known only after. Concurrent requests can therefore overshoot a budget by the cost of whatever was in flight. Anyone selling “hard” budgets is selling a hard-ish budget with a bounded overshoot; ask how it behaves under concurrency and with streaming responses where usage arrives in the last chunk. Price tables also go stale: a new model released this week may be unpriced in the gateway’s map, in which case spend is recorded as zero. Good gateways let you override prices per deployment, which matters for fine-tuned or self-hosted models.

Routing, Fallbacks, and Load Balancing

Routing is where the four products diverge most. All of them can send a request for a logical model name to one of several concrete deployments, and all can fall back when a call fails. The differences lie in strategy richness, the failure signals that trigger a switch, and whether circuit-breaking exists.

Direct answer: LiteLLM offers the broadest open-source routing menu (weighted, latency-based, cost- and usage-based, with cooldowns and ordered fallbacks). Portkey expresses routing as declarative config objects with fallback and load-balance strategies. Kong’s AI Proxy Advanced adds enterprise balancers including lowest-latency, semantic, and priority tiers. Cloudflare keeps routing simple, with dynamic routing and automatic fallback.

Figure 2: Generic LLM routing and fallback logic. Retry on the same target first, open a cooldown if it keeps failing, then walk the fallback chain.

LiteLLM: model groups, cooldowns, and ordered fallbacks

LiteLLM’s router works on model groups: several deployments share a model_name, and the router picks among them. Its reliability documentation shows num_retries, request_timeout, an ordered fallbacks list, and router settings such as allowed_fails and cooldown_time. The semantics are worth stating exactly: retries happen against the model group first, fallbacks start only after retries are exhausted, fallbacks are tried in order, and a deployment that fails more than allowed_fails times inside the cooldown window is removed from rotation for cooldown_time seconds. That last behavior is a lightweight circuit breaker.

Two details trip people up. First, enable_pre_call_checks: true is required for context-window-aware routing; without it a long prompt is happily sent to a deployment whose window is too small and fails there. Second, the routing strategy list (simple shuffle, latency-based, usage-based, cost-based, and others) lives on a separate documentation page, and defaults have changed between releases, so pin a version and test your chosen strategy.

Portkey: configs as the unit of routing

Portkey’s design centers on a JSON config: a declarative object with a strategy of fallback or loadbalance, a list of targets, and optional retry, cache, and timeout settings. A config can be attached to a request by header or saved in the platform and referenced by ID. Because configs nest, you can express “load balance 70/30 across two OpenAI keys, and if both fail fall back to an Anthropic target” as one object. The strength is that routing logic travels with the request or with a named config rather than being baked into a deployment file; the weakness is that behavior depends on which config a given caller attaches, so governance requires locking configs server-side.

Kong AI Gateway: balancers, failover criteria, and a circuit breaker

Kong’s ai-proxy-advanced plugin documents the richest balancer list of the four: round-robin with weights, consistent hashing, least-connections, lowest-latency using EWMA (exponentially weighted moving average) scoring, lowest-usage by tokens or cost, semantic routing by prompt similarity, and priority tiers for failover. Failover is controlled by config.balancer.failover_criteria; client 4xx errors do not trigger failover by default, and you opt in for codes such as 429 or 502. Per the plugin documentation, cross-format fallback (for example OpenAI to Mistral) arrived in version 3.10 and circuit-breaker fields max_fails and fail_timeout in 3.13.

The catch is licensing: the same page states the plugin is available only with AI Gateway Enterprise, requires an AI license, and needs at least Kong Gateway 3.8. The simpler ai-proxy plugin handles provider translation and a single target, and is the entry point for open-source users, but without the balancer logic.

Cloudflare: dynamic routing, minimal knobs

Cloudflare’s documentation describes fallbacks configured through Dynamic Routing, plus an Auto Router that falls back automatically when a model is unavailable. Dynamic routing is a visual and JSON flow editor where you branch on metadata, apply percentage splits, and attach rate limits. It is intentionally less programmable than LiteLLM or Kong. You trade fine-grained control for zero infrastructure.

Load balancing is not the same as capacity pooling

A frequent misunderstanding: load balancing across two API keys for the same provider mostly pools rate limits, not cost. Providers throttle per organization or per key tier, so spreading across keys raises your ceiling, but a 429 on one still needs a retry on another. When you load balance across providers you also change output quality, so route only among models you have evaluated for the task. Our semantic caching deep-dive covers the adjacent question of when not to call a model at all.

Budgets, Rate Limits, Caching, Guardrails, and Telemetry

This is the governance half of the comparison. The routing features above keep requests succeeding; the features below keep them affordable, safe, and explainable.

Figure 3: Budget enforcement with a virtual key. The key resolves to limits, the ledger is checked before the call, and cost is recorded after the response returns usage.

Virtual keys and budgets

In LiteLLM’s open-source build, you get virtual keys, users, teams, spend tracking, and budgets. The enterprise tier adds organizations with delegated admins, tag-based budgets, per-model budgets per key, temporary budget increases, soft budget alerts, and spend reports, according to LiteLLM’s enterprise documentation. A practical consequence: if your finance team wants cost allocation by cost center tag, you will likely need the paid license, while per-team caps are free.

Portkey provides budget limits and rate limits on its platform, set per API key or workspace, and its March 2026 open-sourcing announcement lists usage-policy controls among the open components. I could not confirm from a primary source exactly which policy features are in the open build and which remain hosted-only, so verify that against the repository before designing around it.

Kong approaches cost control through plugins: the AI Rate Limiting Advanced plugin limits by tokens rather than requests, and the Prometheus and logging plugins export usage. In Kong you compose budget behavior from rate-limit windows keyed on consumer, which is flexible but means “monthly budget in dollars” is something you build, not something you toggle. Cloudflare added Spend Limits as a beta feature that sets cost-based budgets at the gateway level; the features page flags it as beta.

Rate limits: requests, tokens, and concurrency

Three kinds of limit matter. Requests per minute protects the gateway and downstream providers from bursts. Tokens per minute protects spend and provider quotas. Max parallel requests protects your own latency when a slow model stalls. Most gateways expose RPM easily; TPM is where products differ. LiteLLM supports RPM and TPM per key, team, and deployment. Kong’s token-aware limiter is the most mature because it reuses its rate-limiting engine, including Redis-backed counters for multi-node clusters. Cloudflare supports fixed and sliding windows per gateway or per request, counted in requests.

The failure mode to design for is the cluster-wide counter. With multiple gateway replicas, a per-replica in-memory counter permits N times the intended limit. Either use a shared store such as Redis or accept sticky routing. Shared counters add a network hop on every request, so measure their cost.

Caching: exact match, semantic, and provider prompt caching

Three different things hide under “caching.” Exact-match caching returns a stored response for an identical request; it is safe, cheap, and surprisingly effective for repeated system-health prompts, classification of fixed inputs, and evaluation reruns. Semantic caching returns a stored response for a similar prompt based on embedding distance, which saves more but risks serving a wrong answer, so it needs a similarity threshold, tenant isolation, and a per-route opt-in. Provider prompt caching is not the gateway’s cache at all; it is the provider discounting repeated prefixes, and the gateway’s only job is to keep prefixes stable and pass through cache-control fields.

Cloudflare’s cache is exact match with a configurable TTL and per-request control, served from its global network. LiteLLM supports Redis and in-memory caches and semantic caching through Redis or Qdrant backends. Kong’s AI Semantic Cache plugin is an enterprise plugin that needs a vector store; the Kong documentation lists Redis and PostgreSQL with pgvector for the balancer’s vector database. Portkey offers simple and semantic cache modes on its platform. In every case, key the cache on the full effective request, including model, temperature, tools, and tenant. A cache that ignores the tenant is a data leak with extra steps.

Guardrail hooks

All four can call a moderation or PII service before and after the model, but they differ in whether the guardrail is built in or a hook you wire to your own service. LiteLLM’s open-source tier includes custom guardrails and Presidio-based PII masking; per its enterprise page, built-ins such as LLM Guard, LlamaGuard, secret hiding, OpenAI moderation, Google text moderation, and Lakera and Aporia prompt-injection integrations need an enterprise license, as do key- and team-scoped guardrails. Portkey ships a large catalog of input and output guardrails and lets you chain them in a config with deny or fallback actions. Kong has prompt-guard, prompt-template, and decorator plugins in its AI family, plus an AI Semantic Prompt Guard and integrations with cloud moderation services in the paid tier. Cloudflare’s Guardrails (beta) evaluates prompts and responses with a Llama Guard model and configurable categories, and its DLP (beta) scans for PII and other sensitive data with flag or block actions.

Two engineering points. Guardrails add latency, because each synchronous check is another model call or HTTP round trip; run the cheap deterministic checks (regex, allow lists) inline and the model-based checks asynchronously where policy permits. And Cloudflare states that Guardrails are billed as Workers AI inference proportional to prompt and response length, so a “free” gateway can still produce a bill when you turn them on.

Observability and OpenTelemetry

The right target state is gateway spans that link into your application traces: a trace ID from the caller propagates through the gateway to the provider call, so one flame graph shows queue time, gateway overhead, provider latency, and streaming time to first token. The OpenTelemetry GenAI semantic conventions define attribute names such as the requested model and token usage, and they are still evolving, so expect attribute churn between collector versions; see the OpenTelemetry GenAI semantic conventions for the current state.

LiteLLM has an OpenTelemetry callback and also ships integrations for Langfuse, Prometheus, and others; per-key or per-team routing of logs to different tools is an enterprise feature. Kong emits OpenTelemetry through its OpenTelemetry plugin and logs AI-specific fields in its analytics. Portkey’s open gateway reports metrics and its platform stores request logs with traces. Cloudflare provides request and response logs with metadata, retention controls, and Logpush export; its feature page lists OpenTelemetry only as a navigation entry without detail, so confirm exact OTel support in the current docs before relying on it.

One caution on logging: a gateway that stores full prompts and responses becomes a repository of your most sensitive text. Decide retention and redaction up front. LiteLLM’s enterprise tier can disable logging per team; Cloudflare offers configurable log retention.

Licensing, Self-Host Versus Managed, and Latency Overhead

Licensing is where “open source” labels mislead. Each product is open in a different place.

The licensing split, product by product

LiteLLM is MIT-licensed at its core, per the project footer. The proxy server includes an enterprise/ portion governed by a commercial license, activated with a LITELLM_LICENSE key. The open build includes the OpenAI-compatible gateway, virtual keys, spend tracking, budgets, fallbacks, and logging. Enterprise adds SSO with SCIM, OIDC and JWT auth, audit logs, RBAC, IP allow lists, tag-based budgets, and per-key guardrails. Single sign-on for the admin UI is free up to five users, according to the enterprise docs. Pricing is by quote.

Portkey announced in March 2026 that its AI Gateway and MCP Gateway are fully open source. Earlier it had open-sourced a lighter gateway under the MIT license, which is the license shown in the repository listing I checked; the news coverage of the 2026 release does not name a license, so read the current LICENSE file before assuming. The company’s stated model is a free foundation with commercial value above it (managed control plane, enterprise governance, support). Company-reported traffic figures, such as two trillion tokens in a day, are marketing numbers and I did not independently verify them.

Kong AI Gateway is built on Kong Gateway, whose open-source core is Apache 2.0. The ai-proxy plugin handles provider abstraction. The ai-proxy-advanced plugin, with its balancers, semantic routing, and failover criteria, is explicitly Enterprise, and it needs an AI license. In practice, if your requirement is multi-model balancing with circuit breaking, plan for a paid license.

Cloudflare AI Gateway is a proprietary managed service. Cloudflare states that core features (dashboard analytics, caching, rate limiting) are free on all plans. Its pricing page distinguishes legacy logs for gateways created before September 24, 2026 (persistent storage, with 100,000 logs on the Workers Free plan and 10 million per gateway on Workers Paid) from Workers Logs pricing and retention for gateways created on or after that date. Unified Billing, where Cloudflare invoices provider usage, carries a 5% fee on credits purchased. This is a recent change worth noticing: a gateway created last month and one created last year may log differently.

Self-host versus managed

Self-hosting buys data residency, no per-request fees, and control over upgrades. It costs you on-call ownership, patching, and the engineering to make a stateful proxy highly available. Managed buys the opposite. For regulated workloads where prompts may include personal or operational-technology data, the deciding question is often not features but whether the prompt may leave your network before reaching the provider anyway. If it will, a managed gateway adds one more processor to your data-protection assessment; if it will not (you call a self-hosted vLLM), a self-hosted gateway is the only coherent choice.

Latency overhead: what to measure, and what not to claim

Vendors publish overhead figures, but they are measured on their own hardware, payloads, and concurrency, and I will not repeat them as fact. The structure of the overhead is more useful. A gateway adds: TLS termination and parsing of the request body; a key lookup (memory hit versus database round trip); limit checks (memory versus Redis); guardrail calls; cache lookup (a vector search for semantic caching can cost tens of milliseconds); and, on the way back, usage accounting. For streaming, the critical metric is added time to first token and whether the gateway re-chunks Server-Sent Events.

Against a model call that takes 800 milliseconds to several seconds, a well-built proxy adding single-digit to low tens of milliseconds is rarely the bottleneck; Python-based proxies under heavy concurrency can show tail-latency variance from the interpreter and garbage collection, which is why teams run more replicas or move hot paths to a compiled data plane. Kong, built on NGINX and Lua, and Cloudflare, on its edge runtime, are structurally better placed for high request rates. These are architectural expectations, not benchmarks. Run your own test: fixed prompt set, realistic concurrency, streaming on, with and without the gateway, and compare p50 and p99 time to first token. The illustrative harness below measures that.

import asyncio, time, statistics, httpx

async def ttft(client, url, key, model):
    t0 = time.perf_counter()
    async with client.stream("POST", url, headers={"Authorization": f"Bearer {key}"},
        json={"model": model, "stream": True,
              "messages": [{"role": "user", "content": "Say hi"}]}) as r:
        async for line in r.aiter_lines():
            if line.startswith("data:"):
                return (time.perf_counter() - t0) * 1000

async def run(url, key, model, n=200, conc=20):
    sem = asyncio.Semaphore(conc)
    async with httpx.AsyncClient(timeout=60) as c:
        async def one():
            async with sem:
                return await ttft(c, url, key, model)
        xs = sorted(await asyncio.gather(*[one() for _ in range(n)]))
    print(f"p50={statistics.median(xs):.0f}ms p99={xs[int(.99*len(xs))-1]:.0f}ms")

# Compare: direct provider URL vs gateway URL, same model and prompt set.

Run it against the provider directly and against the gateway, alternating, and subtract. Use a mock upstream with a fixed delay if you want to isolate gateway overhead from provider variance.

Minimal Configuration Examples

Concrete configs make the differences visible. Each example does the same job: expose one logical model, send traffic to a primary provider, and fall back to a second. Treat model names as placeholders and check each project’s current schema before copying; these formats evolve.

LiteLLM proxy (config.yaml):

model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-4o-mini        # placeholder model id
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-backup
    litellm_params:
      model: anthropic/claude-3-5-haiku-latest   # placeholder
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  num_retries: 2
  request_timeout: 30
  fallbacks: [{"chat-default": ["chat-backup"]}]

router_settings:
  routing_strategy: latency-based-routing
  allowed_fails: 3
  cooldown_time: 30

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Then mint a capped virtual key for a team:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"models":["chat-default"],"max_budget":50,"budget_duration":"30d",
       "rpm_limit":60,"metadata":{"team":"forecasting"}}'

Portkey gateway config (passed in a header or saved by ID):

{
  "strategy": { "mode": "fallback" },
  "targets": [
    { "provider": "openai", "api_key": "sk-...", "override_params": { "model": "gpt-4o-mini" } },
    { "provider": "anthropic", "api_key": "sk-ant-...", "override_params": { "model": "claude-3-5-haiku-latest" } }
  ],
  "retry": { "attempts": 2 },
  "cache": { "mode": "simple", "max_age": 600 }
}

Kong AI Gateway (declarative, open-source ai-proxy on one route):

_format_version: "3.0"
services:
  - name: llm
    url: https://api.openai.com
    routes:
      - name: chat
        paths: ["/v1/chat"]
        plugins:
          - name: key-auth
          - name: ai-proxy
            config:
              route_type: llm/v1/chat
              auth:
                header_name: Authorization
                header_value: "Bearer ${{ env 'OPENAI_KEY' }}"
              model:
                provider: openai
                name: gpt-4o-mini

For multi-target balancing and failover, swap ai-proxy for ai-proxy-advanced with a targets list and balancer.algorithm, which requires the Enterprise license noted earlier. Field names differ slightly between the two plugins and across versions, so confirm on the plugin page for your Kong release.

Cloudflare AI Gateway (point the OpenAI SDK at the gateway):

from openai import OpenAI

client = OpenAI(
    api_key="PROVIDER_KEY_OR_BYOK_PLACEHOLDER",
    base_url="https://gateway.ai.cloudflare.com/v1/<ACCOUNT_ID>/<GATEWAY_ID>/openai",
)
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
)

The Cloudflare path is the shortest: change a base URL, then configure caching, rate limits, and fallbacks in the dashboard. That brevity is the product’s selling point and its limit; there is no file to review in a pull request unless you drive the API or Terraform provider.

Comparison Matrix

The matrix below summarizes the findings as of November 2026, based on vendor documentation fetched during research. “Enterprise” means a paid license is required for the feature in a self-hosted deployment; verify before purchase because tiers move.

Dimension	LiteLLM proxy	Portkey	Kong AI Gateway	Cloudflare AI Gateway
Delivery	Self-host (Docker, pip), Python	OSS gateway plus hosted platform	Self-host or Konnect, NGINX/Lua	Managed edge service only
OpenAI-compatible API	Yes, core design	Yes	Yes, default format; native formats since 3.10	Yes, plus per-provider endpoints
Core license	MIT, with commercial enterprise folder	Open source (MIT shown on repo listing)	Apache 2.0 core, enterprise AI plugins	Proprietary
Routing	Retries, cooldowns, ordered fallbacks, several strategies	Config strategies: fallback, load balance	Many balancers, semantic, priority (Enterprise)	Dynamic routing, auto fallback
Virtual keys and budgets	Free: keys, teams, budgets; paid: tags, per-model	Platform budgets and limits	Compose from consumers and rate limits	Spend limits (beta), gateway auth
Rate limits	RPM and TPM per key, team	Per key, per workspace	Token-aware via plugin	Requests, fixed or sliding window
Caching	Exact and semantic	Simple and semantic	Semantic cache (Enterprise)	Exact match, TTL
Guardrails	Custom and Presidio free; vendor built-ins paid	Large catalog, chainable	Prompt guard plugins; more in paid tier	Guardrails and DLP, both beta
OTel and logs	OTel callback, Prometheus, Langfuse	Metrics and platform logs	OpenTelemetry plugin	Logs, Logpush; OTel unclear
Ops burden	You run Postgres, Redis, replicas	Low (managed) to medium (self-host)	Medium, familiar to Kong teams	None

No row here is a quality ranking. A free feature that your team will not operate is worth less than a paid one that works at 3 a.m. without paging anyone.

Choosing with a decision flow

Figure 4: Decision flow. Network boundary and existing platform investment decide most cases before feature lists do.

The flow encodes four opinions. If prompts must stay inside your network, Cloudflare and hosted Portkey are out. If you already operate Kong for API traffic, extending it with AI plugins reuses your authentication, observability, and on-call muscle, so the marginal cost is low even if the license is not. If you are Python-centric, need many providers quickly, and want key and budget management without buying a platform, LiteLLM fits. If you are already on Cloudflare, or you want the lowest operational burden for a modest feature set, the managed route wins.

Security Considerations

A gateway concentrates secrets. Every provider key, every virtual key hash, and often every prompt flows through it, which makes it a high-value target and a single point where one mistake spreads widely.

The LiteLLM supply-chain incident of March 2026

On March 24, 2026, security researchers reported that LiteLLM versions 1.82.7 and 1.82.8 on PyPI were malicious. According to NetSPI’s write-up, a maintainer’s GitHub account was compromised and the release pipeline used to publish the tainted packages; a group calling itself TeamPCP claimed responsibility. Version 1.82.8 carried a .pth file that executes at Python startup, with no import required. The payload harvested environment variables, cloud and Kubernetes credentials, SSH keys, and CI secrets, and sent them to a lookalike domain. Both versions were quarantined on PyPI, and a maintainer statement said the official Docker proxy image was not affected.

The lesson is not specific to LiteLLM. Any Python-installed gateway runs inside a dependency graph you do not control, and the gateway process holds precisely the secrets attackers want. Mitigations are ordinary but frequently skipped: pin exact versions with hash checking, prefer signed container images pulled by digest, build in a clean pipeline, keep provider keys in a secret manager rather than environment variables where feasible, restrict egress from the gateway to known provider hostnames, and apply short-lived credentials. If you ran either affected version, treat all reachable credentials as compromised and rotate them.

Other risks to model

Prompt and response logs. They contain personal data, source code, and secrets users paste. Redact before storage and set retention explicitly.
Virtual key sprawl. Keys without expiry or budgets are a persistent liability; require both.
Admin UI exposure. LiteLLM and Portkey both ship dashboards; put them behind SSO and a private network, not the public internet.
Server-side request forgery. Gateways that accept an api_base from the caller can be pointed at internal addresses. Disable caller-supplied base URLs.
Prompt injection through tool outputs. A gateway guardrail sees text, not intent; it reduces risk but does not replace least-privilege tool design. The OWASP project catalogs these in its Top 10 for LLM applications.
Cache poisoning and cross-tenant leakage. Include tenant identifiers in cache keys and avoid semantic caching for personalized or regulated content.

Trade-offs, Gotchas, and What Goes Wrong

The gateway is a new outage domain. If the gateway or its database fails, every model call fails, including calls that would have succeeded directly. Provide a tested bypass (a feature flag that points the SDK at the provider), run at least two replicas, and monitor the gateway’s own error rate separately from upstream errors. Teams discover this in their first incident, not their first demo.

Fallbacks can mask quality drops. A silent switch from a flagship model to a smaller backup keeps uptime green while answer quality falls. Emit the actual served model in a response header and a metric, alert on fallback rate, and restrict fallback chains to models that passed the same evaluation set.

Retries multiply cost and load. A retry storm during a provider brownout can triple traffic and trip rate limits on every key. Use exponential backoff with jitter, cap total attempts across retry and fallback layers, and prefer cooldown over blind retry. Non-idempotent tool calls should never retry automatically.

OpenAI-compatible is a spectrum. Providers differ on tool calling, structured output, reasoning parameters, streaming chunk shapes, and usage reporting. Gateways translate the common subset well and the edges poorly; a parameter silently dropped by the translation layer changes behavior without an error. Test the exact features you use, per provider, through the gateway.

Budgets are approximate. Streaming responses report usage at the end, concurrent requests overshoot, and unpriced models record zero. Treat budgets as guardrails for runaway loops and not as accounting-grade billing, and reconcile against the provider invoice monthly.

Semantic caching returns confident wrong answers. Two questions that embed close together can need different answers (“renew” versus “cancel” a contract). Start with exact-match caching, add semantic caching on narrow, high-repeat routes, and log cache hits for review.

Feature gating changes under you. Open-source projects shift lines between free and paid. LiteLLM’s guardrail built-ins, Kong’s advanced balancers, and Cloudflare’s log storage model all illustrate that today’s free feature may not be tomorrow’s. Record which features you depend on and re-check licenses at each upgrade.

Practical Recommendations

Start from constraints, not features. Write down where prompts may travel, who must be able to see spend by team, what your availability target is, and who will be paged. Those four answers eliminate most options.

For a small team that wants one control point quickly and is comfortable running Postgres, deploy the LiteLLM proxy from the official container image pinned by digest, enable virtual keys with budgets from day one, and add enterprise licensing only when you hit SSO, audit, or tag-budget needs. For an organization already standardized on Kong, enable ai-proxy first, prove the pattern on one route, and buy the AI license when you need balancing, token-aware limits, or semantic features. For teams on Cloudflare with moderate needs, turn on the managed gateway, caching, and rate limits in an afternoon and revisit if you outgrow the routing model. For teams that want a managed governance experience with an open-source escape hatch, evaluate Portkey’s hosted platform alongside its open gateway.

Whichever you pick, keep your application code on the standard OpenAI SDK pointed at a configurable base URL. That one decision makes the gateway replaceable and turns a migration from a rewrite into a configuration change.

Checklist before production:

Gateway runs with at least two replicas, health checks, and a tested bypass flag.
Every consumer uses a virtual key with an expiry, a budget, and RPM/TPM limits.
Provider keys live in a secret manager; egress is restricted to provider hosts.
Image or package is pinned by digest or hash; upgrades go through staging.
Fallback chains contain only evaluated models; served model and fallback rate are metrics.
Cache keys include tenant, model, and parameters; semantic caching is opt-in per route.
Traces propagate from the application through the gateway; log retention and redaction are documented.
Latency was measured with streaming on, at realistic concurrency, with and without the gateway.
Monthly spend is reconciled against provider invoices.

Frequently Asked Questions

What is an LLM gateway and do I need one?

An LLM gateway is a proxy between your applications and model providers that standardizes the API, injects policy (authentication, budgets, rate limits, guardrails), and records usage. You likely need one once more than one team or provider is involved, once spend needs attribution, or once uptime matters enough that a single provider outage is unacceptable. For a single prototype calling one model, it is optional and adds a dependency without much benefit.

Is LiteLLM free for production use?

The core proxy is MIT-licensed and includes OpenAI-compatible routing, virtual keys, teams, budgets, fallbacks, and logging, so many teams run it in production without paying. Enterprise features such as SSO beyond five users, SCIM, audit logs, tag-based budgets, and several built-in guardrail integrations require a commercial license activated by a license key. Always check the current enterprise documentation, because the boundary moves.

LiteLLM vs Portkey: which should I choose?

Choose LiteLLM if you want a self-hosted Python proxy with broad provider coverage and strong key and budget management in the free tier. Choose Portkey if you value config-driven routing, a large guardrail catalog, and a managed platform with an open-source gateway as a fallback. Portkey open-sourced its gateway in March 2026, narrowing the gap, so compare the current repositories and hosted pricing directly.

Can Kong AI Gateway do load balancing without an enterprise license?

The open-source ai-proxy plugin provides provider translation for a single target. Kong’s documentation says the ai-proxy-advanced plugin, which adds multi-target balancing, lowest-latency and semantic algorithms, and failover criteria, is available only with AI Gateway Enterprise and requires Kong Gateway 3.8 or later. If you need weighted balancing or circuit breaking, budget for the license or build routing outside Kong.

Is Cloudflare AI Gateway free?

Cloudflare states that core features, including dashboard analytics, caching, and rate limiting, are available free on all plans. Costs can arise elsewhere: guardrails are billed as Workers AI inference, Logpush exports follow Logpush pricing, gateways created on or after September 24, 2026 follow Workers Logs pricing, and Unified Billing adds a 5% fee on purchased credits. Provider token costs are separate and passed through at provider rates.

How much latency does an LLM gateway add?

It depends on deployment, payload, concurrency, caching, and which guardrails run inline, so no honest universal number exists. Structurally, a lean proxy adds small overhead relative to multi-second model calls, but synchronous guardrails and semantic cache lookups can add tens of milliseconds or more. Measure time to first token with streaming enabled, with and without the gateway, at your real concurrency and compare p50 and p99.