engineering · 1 min read
KEDA Autoscaling for AI Workloads
Why CPU-based autoscaling is wrong for AI services and how Bhogar AI uses KEDA on token throughput, queue depth and concurrent agents.
BABhogar AI TeamProduct & Engineering
AI services do not behave like CRUD apps. CPU-based autoscaling either scales too late (latency spikes) or too early (cost waste). Token-based scaling is the right answer.
Why it matters
Bhogar AI uses KEDA scalers on token throughput, queue depth and active-agent counts - signals that actually correlate with load on AI services.
How Bhogar AI approaches it
Per-service KEDA ScaledObjects map AI-specific signals to replica counts, with cooldown windows tuned for the long-tail nature of AI requests.
- Token-throughput-based scaling for LLM proxies
- Queue-depth scaling for agent runtimes
- Active-agent count for stateful agents
- Cooldown tuned for AI long tail
- Cost dashboards per scaling decision
What you get
AI services scale promptly under load and scale down fast when idle, with measured cost and latency wins.