AI Gateway for Production: Do You Need One, and Which to Use

An AI gateway is a proxy that sits between your app and every model provider, so routing, fallback, rate limits, caching, spend, and logs live in one place instead of scattered through your code. You need one the moment a second model, a second provider, or a second engineer touches your LLM calls. Below that line it is a dependency you do not need yet.

This is the read an engineer needs before adding one: what it actually does, the single test for whether you need it, and a real-pricing comparison of the four that matter today, Vercel AI Gateway, Portkey, Cloudflare AI Gateway, and LiteLLM, including the production trade-off none of their landing pages will tell you.

What an AI gateway actually does

Strip the marketing and a gateway is one thing: a single endpoint your app calls instead of calling OpenAI, Anthropic, and Google directly. That one hop buys you the operational layer every production LLM app eventually needs:

• Routing and fallback. Send a request to a primary model, and when it errors or times out, retry on a fallback provider automatically instead of surfacing a 500 to your user.
• Rate limiting and budgets. Cap spend and requests per key, team, or route, so one runaway loop cannot burn your monthly budget in an afternoon.
• Caching. Return a stored response for a repeated prompt instead of paying for the same completion twice.
• Observability. Log every request, token count, latency, and dollar cost in one place, keyed by user or feature.
• A provider-agnostic interface. Swap gpt-5 for claude-opus-4-8 by changing a string, not rewriting your client code.

A gateway does not make your model smarter. It makes your model calls governable. That distinction is the whole reason to adopt one, and the reason to wait if you do not need it yet.

Do you need one yet?

Here is the honest test, and it is not "are you in production."

Add a gateway when any one of these becomes true: a second model, a second provider, or a second engineer touches your LLM calls. A single feature calling a single model through the provider SDK does not need a gateway. Wrapping that in a proxy adds a network hop, a dependency, and a new failure mode for zero benefit. Ship it direct.

The moment you add a second provider (say, Anthropic as a fallback for OpenAI), or a second engineer starts making calls from another service, the things a gateway centralizes stop being optional. Now you have retry logic duplicated in two codebases, spend you cannot attribute, and no single place to see what broke. That is the point where the gateway earns its hop.

The one axis that separates the four: managed vs self-hosted

Every other feature difference is downstream of a single choice: does the gateway run as someone else's service, or inside your own infrastructure?

Managed (Vercel, Portkey Cloud, Cloudflare) means one line of code, no infra to run, and the vendor owns uptime and patching. You trade control and data residency for speed.

Self-hosted (LiteLLM, Portkey's open-source gateway) means the proxy runs in your VPC, your keys never leave your network, and you can shape it however you want. You own the deployment, the scaling, and, as we will get to, the security patching.

Pick the axis first. It eliminates half the options before you compare a single feature.

The four, compared

Now the case for each.

Cloudflare AI Gateway: the fastest free start

Cloudflare gives away the core. Analytics, caching, and rate limiting are free on every plan, and it connects with one line of code in front of OpenAI, Anthropic, Google, and more. Persistent logs are capped at 100,000 total across all gateways on the Workers Free plan and 10,000,000 per gateway on Workers Paid, which is generous for early production.

If you want visibility and basic control today and do not want a bill or a deployment, this is the lowest-friction option on the list. Its ceiling is depth: it is a control plane for observability and caching, not a governance suite with per-team budgets and guardrails.

Vercel AI Gateway: zero markup, if you live on Vercel

Vercel's pitch is pricing honesty: you pay provider list rates with zero markup on tokens, even when you bring your own key. Every team gets a $5 monthly credit and a free tier limited to a subset of models with lower rate limits; exceed a per-model limit and you get a 429 to retry. Moving to the paid tier (buy credits, no commitment) opens all models, custom rate limits, and BYOK.

The appeal is that a swap to a fallback model is a string change across a unified API. The constraint is gravity: it is a managed edge service that makes the most sense when your app already runs on Vercel.

Portkey: managed governance with a compliance path

Portkey is the option that treats the gateway as a governance product. Fallbacks, load balancing, retries, and routing are on every plan, including the free Developer tier, but Portkey itself labels that tier "not suitable for production workloads": it records only 10,000 logs a month with 3-day log retention.

Production is $49/month for 100,000 recorded logs (then $9 per additional 100,000), 30-day log retention, guardrails, role-based access control, service-account keys, and semantic caching. The regulated-industry controls, SOC2 Type 2, HIPAA, VPC hosting, custom BAAs, live on Enterprise with custom pricing. Portkey also publishes an open-source gateway, so you can start self-hosted and move to their cloud, or the reverse.

Choose Portkey when you need managed convenience and the audit trail, access controls, and guardrails that a real compliance review will ask for.

LiteLLM: full control, and the bill for it

LiteLLM is the open-source default for teams that want the gateway inside their own perimeter. It runs on-prem, multi-cloud, Kubernetes, or Helm; the vendor cites 240M+ Docker pulls, 1B+ requests served, and 1,005+ contributors. Every core feature is in the open-source build: cost tracking, budgets, rate limiting, guardrails, RBAC and usage tracking by key/team/org, an MCP gateway, and logging to Datadog and OpenTelemetry. Your provider keys never leave your network.

The bill for that control is not dollars, it is operational ownership, which the next section is about.

What breaks in production

The demo works. Here is what a gateway has to survive once real traffic hits it.

Fallback that actually fails over

The headline feature is provider fallback, and it is also where teams get a false sense of safety. A fallback only works if the gateway's health check is honest: a provider returning 200 with a truncated or malformed body is a "success" to a naive check but a failure to your user. Configure fallback on real signals (timeouts, 5xx, 429, and schema validation of the response), not just connection errors, and test it by actually blackholing the primary in staging. A fallback path you have never triggered is a config file, not a safety net.

Cost telemetry you can attribute

Aggregate spend is useless when the number spikes. What you need is spend keyed by feature, by customer, and by model, so a 3x jump on Tuesday points at which route caused it. Every gateway here logs token counts and cost; the ones that let you tag requests with your own metadata (customer ID, feature flag, environment) are the ones that turn a scary bill into a one-line root cause. Wire that tagging on day one, because you cannot backfill attribution onto logs you already dropped. If you are also caching to cut that bill, the same discipline applies to prompt caching in production AI apps: measure the hit rate, do not assume it.

Semantic caching, with a correctness caveat

Exact-match caching is safe. Semantic caching (returning a stored answer for a similar prompt) can cut cost hard, but it will happily serve a stale or wrong answer when two prompts are close in embedding space but different in intent ("cancel my order" vs "can I cancel my order"). Turn it on for stable, read-only queries; keep it off for anything transactional or personalized until you have measured the false-hit rate against a real query set.

The observability you build once

The single best reason to adopt a gateway is that it becomes the one place every LLM request is logged, so you do not instrument observability in five services. Make that log line carry request and response payloads, model, latency, token counts, cost, and your metadata tags, and you have the raw material for evals and incident triage. This is the same discipline as logging for AI agent observability: the gateway is where you enforce it for model calls specifically.

The security line no landing page mentions

Self-hosting means you own the patch cycle, and this is not hypothetical. In 2026, LiteLLM disclosed CVE-2026-42271 (advisory GHSA-v4p8-mg3p-g94g): two MCP-preview endpoints accepted a full stdio server config and spawned the supplied command as a subprocess on the proxy host, gated only by a valid API key with no role check. Any authenticated user, including a holder of a low-privilege internal-user key, could run arbitrary commands on the host. NVD scores it 8.8 HIGH (CVSS 3.1). It affects versions 1.74.2 up to the fix in 1.83.7.

The lesson is not "LiteLLM is unsafe," every gateway will ship a CVE eventually. The lesson is the ownership split. When a managed gateway is affected, the vendor patches and you may never notice. When your self-hosted gateway is affected, you are on the clock to upgrade to 1.83.7 and rotate every master and provider key it held, on the disclosure's timeline, not yours. That is the real cost of self-hosting, and it belongs in the decision.

The decision rule

Reduce it to three moves.

1. 1

   Confirm you need one

   A second model, provider, or engineer is touching LLM calls. If not, ship direct and revisit later.

2. 2

   Pick the axis: managed or self-hosted

   Regulated data or a hard no on keys leaving your VPC pushes you to self-hosted (LiteLLM, or Portkey OSS). Otherwise start managed.

3. 3

   Match the tool to the need

   Fast free visibility: Cloudflare. Already on Vercel and want zero markup: Vercel. Managed governance, RBAC, and a compliance path: Portkey. Full control inside your perimeter, with the patch-ownership that implies: LiteLLM.

Whichever you choose, the gateway is where you enforce the production controls that keep an AI system honest: fallback that fails over, spend you can attribute, caching you have validated, and logs you can run evals against. Gartner projects that over 40% of agentic-AI projects will be cancelled by end of 2027 on cost, unclear value, and weak controls. The gateway is one of the cheapest places to install the controls that keep a project on the right side of that number.

Do I need an AI gateway?

Not for a single feature calling a single model, that just adds a hop and a dependency. Add one when a second model, a second provider, or a second engineer touches your LLM calls, because that is when centralized routing, spend, and logging stop being optional.

What is the difference between an AI firewall and an AI gateway?

A firewall inspects traffic for threats like prompt injection and data exfiltration. A gateway manages the operational flow: routing, fallback, rate limiting, caching, and spend. Some products bundle both; confirm what a "security" claim actually inspects.

What is the best AI gateway?

There is no single best; it depends on the managed-vs-self-hosted axis. Cloudflare for the fastest free start, Vercel for zero-markup pricing if you are on Vercel, Portkey for managed governance and compliance, LiteLLM for full self-hosted control.

Is an AI gateway a single point of failure?

Yes, every request flows through it. That is a reason to engineer around it, not to skip it: use health-check-driven fallback, and keep a documented bypass path that calls providers directly so a gateway outage degrades your app instead of downing it.

Is a self-hosted gateway safe?

It can be, but you own the security. It holds every provider key, so a vulnerability like LiteLLM's CVE-2026-42271 (fixed in 1.83.7) is your emergency to patch and your keys to rotate. Pin versions, watch advisories, and rehearse key rotation.

Book the Agentic Readiness Audit

Five working days to a clear read on your AI stack: where a gateway belongs, what to log, where costs leak, and which controls a production launch actually needs.