Platform
Model Gateway & LLM providers
The gateway sits between your workloads and every provider you use. Route by cost and capability, fall back when a provider degrades, cache what repeats, cap what it may spend - and attribute every call to the run and team that caused it.
How it works
Configuring providers and routes
Set up capacity once, then treat model strategy as something you revisit with data rather than something frozen at build time.
Register providers
Add commercial providers, your own keys, and self-hosted endpoints as available capacity.
Capabilities
Gateway capabilities
A gateway earns its place through resilience and unit economics - not through abstraction for its own sake.
Policy-based routing
Route each call by capability, cost, latency, and provider health instead of hardcoding a model name into application logic.
Portal - Models → Gateway
Ordered fallback
On error, timeout, or rate limit, the gateway retries down a defined chain - so a single provider incident does not take your processes offline.
per-route fallback order
Caching
Exact and semantic cache lookups return known answers without provider spend, which matters most on high-volume repetitive traffic.
exact + semantic cache
Budgets and cost controls
Ceilings per organization, workspace, and route, with alerts before limits and enforcement at them.
budget guards - quotas
Rate limiting and queueing
Protect shared provider quota so a batch job cannot starve interactive traffic during business hours.
per-route rate limits
Bring your own keys
Use your own provider accounts and commitments, keeping your negotiated rates and your relationship with the provider.
BYOK - per-org credentials
Self-hosted and BYO models
Route residency-constrained or high-volume work to open-weight or self-hosted deployments alongside commercial providers.
open weights - private endpoints
Guardrails in the path
Because every call goes through the gateway, policy, PII handling, and spend authority apply uniformly - including to direct API traffic.
policy evaluated per call
Uniform capability surface
Streaming, function calling, and structured output are normalised across providers, so switching a model does not mean rewriting agents.
streaming - tools - JSON mode
Design decisions
Design decisions
The model market moves faster than most release cycles. These positions are how the platform keeps that from becoming your problem.
- Model choice is infrastructure, not application code
- When a provider name is embedded in agents and services, every price change or capability shift becomes a release. Putting routing behind a gateway makes model strategy a configuration decision you can revisit weekly.
- Not every step needs a frontier model
- Classification, extraction, and summarisation often run acceptably on a mid-tier or open-weight model. Route-level policy is how you capture that saving without degrading the steps that genuinely need the strongest reasoning.
- Fallback is a resilience requirement
- Provider incidents and rate limits are normal operating conditions. An ordered fallback chain per route turns a provider outage into slightly different latency instead of a stalled business process.
- Cache before you optimise prompts
- On repetitive traffic, cache hits are the cheapest possible response. Measuring hit rate per route usually finds more savings faster than another round of prompt tuning.
- Cost must be attributable to be controllable
- Provider invoices cannot tell you which team or workflow caused the spend. Recording cost per call at the gateway is what makes budgets, chargeback, and cost-per-outcome possible.
- Independence is a negotiating position
- Being able to move traffic between providers without a code change protects you commercially and technically - and lets residency requirements be satisfied per route rather than per platform.
Workload routing
Routing by workload
A single default model is either too expensive for simple work or too weak for hard work. Routing by workload class is how both problems go away.
Right-sizing the model per route is usually the single largest AI cost lever.
Cost per 1k requests
gateway cost per route vs volume
Repeated questions answered without provider spend.
Cache hit rate
cache hits vs total calls per route
Business processes keep running through a provider incident.
Availability
fallback engagements and success rate
Routing by latency keeps interactive surfaces usable under load.
Latency p95
per-route duration percentiles
Spend lands inside plan because limits enforce rather than warn.
Budget variance
budget guard events per workspace
Adopting a new model is a config change rather than an engineering project.
Migration cost
routes repointed without code release
FAQ
Frequently asked questions
Which providers are supported?
Can we use our own provider accounts?
What happens if a provider goes down mid-run?
How is semantic caching kept safe?
Does routing hurt answer quality?
Related capabilities
Explore the rest of the platform
Reasoning
Agents
Autonomous agents that retrieve company context, call tools, and resolve work end to end.
Action
Workflows
Durable DAG execution for multi-step business processes, with branching, retries, and human approvals.
Judgment
Guardrails & Governance
Policy enforcement, PII handling, RBAC, tenant isolation, and audit trails on every request and action.
See Bhogar on your own data
Book a 45-minute working session. We connect one of your sources, build one agent, run one governed workflow, and review the trace together.