Auto Scaling Protocol for Gen AI Agents

Abstract

This dissertation addresses the core operational challenge of scaling Generative AI (Gen AI) agents in production environments, where traditional cloud autoscaling mechanisms fail to accommodate the stochastic, stateful, and computationally intensive nature of agentic workloads. The study proposes the Semantic‑Aware Autoscaling Framework (SAASF), a predictive, semantics‑driven scaling approach designed to overcome the limitations of reactive, resource‑based, and request‑based autoscaling used in platforms such as Kubernetes HPA and serverless architectures.
The research analyzes more than 15 million production trace records, revealing that agentic workloads display high variance, non‑stationary arrival patterns, and heavy‑tailed inference times—properties that violate classical M/M/c queue assumptions and contribute to performance degradation under conventional autoscaling. A four‑class Agentic Workload Taxonomy is developed using hierarchical clustering, identifying distinct workload categories that differ significantly in latency sensitivity, semantic depth, and data dependency.
A predictive regression model is then constructed to infer inference time from semantic features—including token volume, reasoning complexity, and context dependency—achieving an R² of 0.93 and substantially outperforming token‑only baselines. This enables proactive scaling behavior by estimating computational demand before inference begins.
The SAASF architecture integrates this predictive intelligence into a Kubernetes custom controller featuring context‑aware routing for KV‑cache locality, class‑specific scaling policies, and a model‑predictive control loop. The controller translates semantic predictions into real‑time scaling decisions while minimizing cold‑start penalties and preventing resource thrashing.
Simulation experiments benchmark SAASF against standard reactive autoscaling and static over‑provisioning. Results show that SAASF reduces SLO violation rates to 0.12%, achieves 40.3% lower resource over‑provisioning, and delivers 42% faster scale‑up responsiveness, while maintaining high system stability with minimal pod oscillation.
The study concludes that semantics‑aware predictive scaling is essential for the industrial viability of Gen AI agents. By aligning resource orchestration with cognitive workload characteristics, SAASF resolves the fundamental discontinuities in queueing, control, and state management that hinder current systems, establishing a scalable, cost‑efficient, and performance‑reliable paradigm for next‑generation AI infrastructures.