跳到主要内容

Deploy and Configure the AI Gateway

Implementation Effort: High – Requires provisioning Azure API Management infrastructure, defining API policies for token rate limiting and content safety, configuring networking, and integrating with identity providers and backend AI services.
User Impact: Medium – Agent developers must route their LLM calls through the gateway, which changes how they connect to model endpoints and may affect response latency when gateway-level safety evaluation is active.

Overview​

Azure API Management (APIM) provides a centralized gateway specifically designed to manage generative AI workloads. Deploying and configuring APIM as the AI gateway means that all agent-to-model traffic — whether agents call Azure OpenAI, Azure AI Foundry models, or other LLM endpoints — routes through a single control point where the organization enforces security policies, token-based rate limits, content safety inspection, and observability. Without this gateway, each agent connects directly to its model endpoint, and the organization has no centralized mechanism to enforce consistent controls across the agent fleet.

The GenAI gateway capabilities in APIM are purpose-built for language model workloads. They include LLM-aware token counting, load balancing across multiple model deployments, semantic caching to reduce redundant calls, and built-in policies for content safety and prompt inspection. These are not generic API management features repurposed for AI — they are capabilities that understand the token-based economics and request/response patterns unique to large language models.

Gateway configuration includes two critical policy areas. First, token-based rate limiting and quotas must be configured to control resource consumption. Standard API rate limiting counts HTTP requests, but language model workloads consume resources proportional to token count, not request count. The LLM token limit policy counts both prompt and completion tokens and enforces limits per agent or consumer group. Without token-based rate limiting, a single agent with a bug or a compromised orchestration loop can monopolize model capacity, causing latency spikes for the rest of the fleet and generating significant uncontrolled costs. Second, the LLM content safety policy should be configured to forward prompts and responses to the Azure AI Content Safety resource for evaluation at the gateway level. This provides defense-in-depth on top of the SDK-level content safety checks that individual agents perform — if an agent's internal safety controls are misconfigured, skipped, or compromised, the gateway catches the gap because it evaluates traffic independently of what the agent's code does.

This task supports Verify Explicitly by providing a central point where every agent request is authenticated, authorized, and inspected before reaching the model endpoint. Agents must present valid credentials to the gateway, and the gateway validates those credentials against Entra ID before forwarding the request. This ensures that no agent can reach a model endpoint without first passing through identity verification. The task also supports Use Least Privilege Access applied to compute resources — token quotas enforce that each agent consumes only the model capacity it needs. It supports Assume Breach because if an individual agent is compromised, the gateway provides an independent control layer that detects anomalous request patterns, enforces rate limits to constrain how much data a threat actor can extract, and blocks requests that violate content safety policies. The compromised agent cannot bypass these controls because they are enforced at the infrastructure layer, not in the agent's own code.

The AI gateway also serves as a prerequisite for Microsoft Foundry Control Plane. Foundry Control Plane is a unified management interface that provides fleet-wide visibility, compliance enforcement, and security monitoring for all AI agents, models, and tools across a subscription. It requires an AI gateway to be configured before advanced governance features — including token quota enforcement from the Foundry portal, custom agent registration, and MCP tool governance — become available. Without the gateway in place, Foundry Control Plane cannot intercept or govern agent-to-model traffic, and the organization loses access to centralized fleet management capabilities that depend on the gateway as the traffic routing layer.

Organizations that allow agents to connect directly to model endpoints lose visibility and control over their AI traffic. They cannot enforce consistent rate limits, cannot inspect prompts and responses for safety compliance at the infrastructure level, and cannot aggregate telemetry across agents for security monitoring. Each development team makes independent decisions about how to connect to models, leading to inconsistent security posture and no centralized audit trail.

Reference​