LLM Gateway Reference Architecture

Why every serious AI platform needs a gateway

Without a gateway you have: scattered provider keys, no cost attribution, no vendor portability, no shared guardrails, no shared cache, and no shared observability. Each product team will solve these badly, in incompatible ways, and your CFO will find out at the worst possible moment.

A gateway centralises these concerns and turns them into a platform feature. Done right, it is invisible to product teams; done wrong, it is a bottleneck.

The request flow at a glance

Every request follows the same pipeline: authenticate → resolve tenant → pick model → consult cache → enforce guardrails (input) → call provider with retries → enforce guardrails (output) → coerce structure → meter and trace → return. Each stage is a pluggable middleware so policies can be added without forking the core.

Cost-aware routing

Routing decides which model serves a request based on a deterministic, auditable policy: workload class, latency budget, customer tier, content sensitivity, and current per-tenant spend. Routing is not a/b testing - that lives in the experimentation layer.

• Workload class: simple Q&A → small/cheap; multi-step reasoning → frontier.
• Latency budget: streaming UX → fastest stable provider; batch → cheapest.
• Sensitivity: PHI/PII flagged → on-prem model only.
• Spend: hard ceiling per tenant per day with graceful degradation to a smaller model.

Fallback and provider redundancy

Frontier providers do go down. Plan for it: at least one secondary provider per workload class with deterministic, observable fallback. The fallback path is not "best effort" - it is documented, evaluated and tested in the same eval pipeline as the primary.

Caching: prompt, semantic and embedding

Three cache layers, in order of effort: exact-prompt cache (free win), embedding cache (cheap, big win for RAG), semantic cache (more complex, reserve for high-volume Q&A). Every cache hit is observable; every cache write respects tenant isolation.

BYOK and key rotation

Customers should never hand you a raw provider key as a config string. Bring-your-own-key uses a customer-controlled vault binding; rotation is automated; usage is attributed per key. Hold-your-own-key (HYOK) is required for sovereign tiers - the platform never sees the key in cleartext.

Structured outputs and function calling

Different providers expose function calling and structured outputs differently. The gateway normalises the surface so product teams write to one schema; the gateway translates per provider, falls back to constrained decoding when a provider lacks JSON mode, and validates against the requested schema before returning.

Observability and metering

Every request emits an OTel-compliant span with prompt, response (sampled), tokens, latency, cost, refusal class and trace context. Metering aggregates spend per tenant, per feature, per model, per env - exportable hourly to your finance system. observability docs documents the schema.

Failure modes catalogue

A short, honest list: provider outage; provider degradation (latency without errors); rate-limit storms; partial structured-output failures; cache poisoning; key compromise; runaway tenant spend. Each failure mode has a documented detection signal, runbook and customer comms template.

Capacity planning rules of thumb

Plan for: 3× your forecast peak QPS for streaming workloads (latency-sensitive); 1.5× for batch; reserve provider quotas with at least one secondary per primary; cache hit-rate targets of 35%+ for embedding, 10%+ for semantic. Track actual vs plan weekly.

How Bhogar AI compares to LiteLLM, Portkey and Helicone

LiteLLM is a great library. Portkey is a hosted gateway. Helicone is a great observability tool. Bhogar AI bundles a gateway, KBs, agents, workflows, evals, observability and governance. If you want a single component, the open-source / SaaS options are excellent. If you want one platform that ships every concern in this whitepaper, Bhogar AI Platform is built for that.