Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma
This video presents a comprehensive framework for transitioning AI agents from experimental stages to production environments, emphasizing governed enterprise AI agents. The core idea revolves around building a robust agent infrastructure, including MCP servers and a gateway, complemented by frameworks for agent memory management, tracking and evaluation, PII trimming, and context optimization to ensure security, cost-efficiency, and reliable performance. The key takeaway is to prioritize a holistic framework design over isolated agent development, ensuring gateway-first security, human-in-the-loop control, and observability for high-risk actions and cost management.
read more
Harish Gaggar's presentation, "Paving the Road to Production: Governed Enterprise AI Agents," addresses the challenges of deploying AI agents into production by advocating for a comprehensive framework approach rather than focusing solely on individual agents.
The presentation identifies several key challenges: lack of proper authentication/authorization for agents communicating with backend services, absence of sufficient tools for agents to execute queries, inadequate tracking and evaluation systems, and inefficient memory and context management leading to high token costs and potential PII leakage.
To overcome these, Gaggar proposes a design framework that includes eight crucial components:
1. Agent Framework: Utilizing solutions like LangGraph (or LangChain) to build task-specific nodes and easily integrate skills, prompts, or marketplace plugins. This ensures efficient agent responses based on queries.
2. OAuth before every call: Implementing a robust MCP Gateway to ensure that every call made by an agent or service account is securely authenticated with the correct auth providers. This gateway then redirects to the appropriate MCP servers.
3. Customized MCP (Microservice Control Plane) Server: Building custom MCP servers with the necessary tools for agents to execute queries. These tools must undergo internal security practices and reviews to ensure only authorized tools are used in production.
4. Continuous Audit: Implementing a system to log all operations at the MCP gateway level, providing comprehensive visibility and accountability.
5. PII/secrets stripped: A critical component to scrub sensitive data (like API keys, SSN, credit card numbers, passwords, etc.) before it reaches the LLMs, memory, or tracking systems. Severity levels are defined for different data types, with high-severity data being blocked or redacted immediately.
6. Tracking and Eval: A system for per-query tracking, recording metrics like model used, tokens consumed, latency, and response quality. This data is crucial for continuous evaluation and feedback to improve agent performance.
7. Memory Management: Implementing a mechanism to save and restore the state of an agent's workflow over time (including human-in-the-loop actions). This ensures durability and seamless recovery across runs. The concept of Run_ID and checkpoints within a thread-based architecture is introduced, where each query or follow-up action within a thread creates a run ID, and each node execution within a LangGraph flow saves a checkpoint. This data can be stored in a Memory DB like SQLite locally or Cloud Spanner in production.
8. Context Optimizer: A framework to forecast token usage and prune unnecessary context before it leads to excessive costs. This involves: selecting relevant parts of the context based on token limits, reranking information for relevance, clustering similar pieces of information, and compressing the final payload before sending it to the LLM. This significantly reduces token consumption and cost, with reported savings of 70-90% in some cases.
The Agent Infrastructure diagram illustrates a typical deployment flow: local development involves agent registration via a Dev Portal, which generates a repository containing the agent framework and local resources (memory, tracking & eval). Once the agent is built as a Docker image and passes security scans in a Deployment Pipeline, it's registered with a LangFuse URL for tracing and deployed to a GKE Cluster. The cluster provides Agent Workload Identity and dependent resources (Agent Memory, Track & Eval tables, Agent Secret management). Finally, the agent interacts with Shared Services like LangFuse, an AI Gateway (for LLM services), MCP Servers (for tools), and API Endpoints (for external services).
The MCP Gateway acts as a centralized authentication and routing hub, ensuring secure connections. It involves a one-time OAuth authentication, namespace lookup, JWE unwrapping, and user token attachment, with session states stored in Redis. All traffic is routed through the gateway, and direct access to backend MCP servers is prevented.
For MCP Gateway Monitoring & Compliance, an OpenTelemetry Collector gathers traces and metrics from the gateway. Traces are stored in Tempo, and metrics in Prometheus, both visualized through Grafana dashboards. Operational logs are sent to a Log Store (e.g., ELK, Loki, CloudWatch). This provides real-time visibility into system performance, error rates (like 401s), and token refresh failures.
Ultimately, the speaker emphasizes that building an enterprise framework, not just an AI agent, is paramount. Gateway-first security is foundational, MCP transforms APIs into safe tools for agents, human oversight is crucial for high-risk actions, trust is built through observability, privacy, and persistence, and context optimization is essential for cost management. This holistic approach ensures secure, efficient, and governable AI agent deployments in enterprise settings.