If you’ve shipped anything with AI agents recently, you’ve probably wired an agent to a tool with the Model Context Protocol (MCP), called more than one model provider, and maybe had two agents talk to each other. Each of those connections is a piece of network traffic, and right now most teams manage each one by hand, in application code.
An agent gateway gives all of that traffic a single place to be managed.
What is an agent gateway?
An agent gateway is a data plane for agent traffic. It’s a single proxy that sits between your agents and everything they talk to, and it applies routing, security, and observability consistently across all of it.
A traditional API gateway handles north-south HTTP traffic: authentication, rate limiting, routing to backends. An agent gateway does the same job for the traffic patterns agentic systems generate:
- MCP: the calls an agent makes to discover and invoke tools.
- Agent-to-Agent (A2A): messages between agents, often across frameworks (LangChain, CrewAI, ADK).
- LLM inference: requests to model providers (OpenAI, Anthropic, Bedrock, Gemini) or to self-hosted models on your own GPUs (vLLM, TGI, Triton).
- Regular service traffic: the plain HTTP, gRPC, and REST APIs your agents still call.
One gateway handles those concerns once, so no agent has to reimplement auth, retries, logging, and policy for every protocol. agentgateway, recently donated to the Linux Foundation’s Agentic AI Foundation, describes itself as “one high-performance gateway for service, LLM, and MCP traffic.”
Why does agentic AI need one?
Agents need a gateway for the same reasons microservices did, plus a few sharper ones. Four stand out:
1. The same problems microservices had. Agentic systems need authentication, authorization, rate limiting, retries, and observability, exactly like microservices did. Without a gateway, every agent and every tool integration solves them again, inconsistently. One agent logs every tool call; the next logs nothing.
2. Agents carry real authority. When an agent invokes a tool, it’s taking an action: writing to a database, sending an email, moving money. The blast radius of a bad call is larger than a normal API request, which is why you want a single enforcement point for which agent may call which tool, with which arguments.
3. Models and tools change constantly. Teams switch model providers for cost or quality, add new MCP tools weekly, and run some inference locally and some in the cloud. A gateway lets you route, fail over, and budget across providers without rewriting the agent.
4. Governance depends on visibility. Compliance, cost control, and debugging all depend on observability. A gateway that emits metrics, traces, and audit logs for every tool call and inference request turns “what did the agent do?” from a guess into a query.
The agent gateway is becoming the control point for agentic systems, the layer where governance, security, and observability are enforced.
Main use cases
MCP gateway. Expose a curated set of MCP tools through one endpoint with discovery, RBAC, and audit logging. Each agent works from one governed catalog, with no direct, ungoverned connections to the underlying systems.
LLM gateway. Put one endpoint in front of multiple model providers, with token budgets, semantic caching, and failover. A provider outage or a price change is handled in config, with no agent code to rewrite.
Inference routing. Route inference across self-hosted model servers (vLLM, TGI, Triton) with latency- and cost-aware policies, so expensive GPU capacity is used deliberately.
Agent-to-agent communication. Bridge agents built in different frameworks over A2A, with the same auth and observability you apply everywhere else.
Unified policy and observability. Apply one set of security controls (mTLS, authn/authz, policy-as-code) and one observability pipeline (OpenTelemetry) across agent and traditional traffic, so the AI parts run through the same stack as everything else.
Where Kubernetes fits
Most of these gateways are designed to run on Kubernetes. Take agentgateway: it deploys via Helm and the Gateway API, and it’s Envoy-compatible. The agentic data plane behaves like the rest of your platform. You deploy it declaratively, scale it horizontally, govern it with policy-as-code, and connect it to the service networking you already run. Running it on the same Kubernetes substrate as your other infrastructure means one operational model, one security posture, and one place to reason about where your agents’ most sensitive traffic flows.
If you’re already running platforms on Kubernetes, the agentic data plane is just another workload on infrastructure you already operate, ideally one you control.
Takeaways
- An agent gateway is a data plane for agent traffic (MCP, A2A, LLM inference, and regular services) with consistent routing, security, and observability.
- Agentic systems need it for the same reasons microservices needed an API gateway: shared cross-cutting concerns, plus the higher stakes of agents that act.
- The main use cases are governed MCP tool access, multi-provider LLM routing, GPU-aware inference routing, A2A bridging, and unified policy and observability.
- Running it on Kubernetes keeps the agentic control point on infrastructure you operate and control.
If you already run Kubermatic’s KubeLB, you have this today. Its AI & MCP Gateway uses agentgateway as the data plane and federates MCP servers behind a single endpoint via the AgentgatewayBackend CRD. And if you want to go hands-on from the ground up, the new AI Infrastructure track in Kubermatic Learn walks through deploying and configuring an agent gateway on Kubernetes step by step.
