Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

AI Engineer · 2026-10-14

This video presents a comprehensive framework for transitioning AI agents from experimental stages to production environments, emphasizing governed enterprise AI agents. The core idea revolves around building a robust agent infrastructure, including MCP servers and a gateway, complemented by frameworks for agent memory management, tracking and evaluation, PII trimming, and context optimization to ensure security, cost-efficiency, and reliable performance. The key takeaway is to prioritize a holistic framework design over isolated agent development, ensuring gateway-first security, human-in-the-loop control, and observability for high-risk actions and cost management.

read more

Harish Gaggar's presentation, "Paving the Road to Production: Governed Enterprise AI Agents," addresses the challenges of deploying AI agents into production by advocating for a comprehensive framework approach rather than focusing solely on individual agents.

The presentation identifies several key challenges: lack of proper authentication/authorization for agents communicating with backend services, absence of sufficient tools for agents to execute queries, inadequate tracking and evaluation systems, and inefficient memory and context management leading to high token costs and potential PII leakage.

To overcome these, Gaggar proposes a design framework that includes eight crucial components:

1. Agent Framework: Utilizing solutions like LangGraph (or LangChain) to build task-specific nodes and easily integrate skills, prompts, or marketplace plugins. This ensures efficient agent responses based on queries.

2. OAuth before every call: Implementing a robust MCP Gateway to ensure that every call made by an agent or service account is securely authenticated with the correct auth providers. This gateway then redirects to the appropriate MCP servers.

3. Customized MCP (Microservice Control Plane) Server: Building custom MCP servers with the necessary tools for agents to execute queries. These tools must undergo internal security practices and reviews to ensure only authorized tools are used in production.

4. Continuous Audit: Implementing a system to log all operations at the MCP gateway level, providing comprehensive visibility and accountability.

5. PII/secrets stripped: A critical component to scrub sensitive data (like API keys, SSN, credit card numbers, passwords, etc.) before it reaches the LLMs, memory, or tracking systems. Severity levels are defined for different data types, with high-severity data being blocked or redacted immediately.

6. Tracking and Eval: A system for per-query tracking, recording metrics like model used, tokens consumed, latency, and response quality. This data is crucial for continuous evaluation and feedback to improve agent performance.

7. Memory Management: Implementing a mechanism to save and restore the state of an agent's workflow over time (including human-in-the-loop actions). This ensures durability and seamless recovery across runs. The concept of Run_ID and checkpoints within a thread-based architecture is introduced, where each query or follow-up action within a thread creates a run ID, and each node execution within a LangGraph flow saves a checkpoint. This data can be stored in a Memory DB like SQLite locally or Cloud Spanner in production.

8. Context Optimizer: A framework to forecast token usage and prune unnecessary context before it leads to excessive costs. This involves: selecting relevant parts of the context based on token limits, reranking information for relevance, clustering similar pieces of information, and compressing the final payload before sending it to the LLM. This significantly reduces token consumption and cost, with reported savings of 70-90% in some cases.

The Agent Infrastructure diagram illustrates a typical deployment flow: local development involves agent registration via a Dev Portal, which generates a repository containing the agent framework and local resources (memory, tracking & eval). Once the agent is built as a Docker image and passes security scans in a Deployment Pipeline, it's registered with a LangFuse URL for tracing and deployed to a GKE Cluster. The cluster provides Agent Workload Identity and dependent resources (Agent Memory, Track & Eval tables, Agent Secret management). Finally, the agent interacts with Shared Services like LangFuse, an AI Gateway (for LLM services), MCP Servers (for tools), and API Endpoints (for external services).

The MCP Gateway acts as a centralized authentication and routing hub, ensuring secure connections. It involves a one-time OAuth authentication, namespace lookup, JWE unwrapping, and user token attachment, with session states stored in Redis. All traffic is routed through the gateway, and direct access to backend MCP servers is prevented.

For MCP Gateway Monitoring & Compliance, an OpenTelemetry Collector gathers traces and metrics from the gateway. Traces are stored in Tempo, and metrics in Prometheus, both visualized through Grafana dashboards. Operational logs are sent to a Log Store (e.g., ELK, Loki, CloudWatch). This provides real-time visibility into system performance, error rates (like 401s), and token refresh failures.

Ultimately, the speaker emphasizes that building an enterprise framework, not just an AI agent, is paramount. Gateway-first security is foundational, MCP transforms APIs into safe tools for agents, human oversight is crucial for high-risk actions, trust is built through observability, privacy, and persistence, and context optimization is essential for cost management. This holistic approach ensures secure, efficient, and governable AI agent deployments in enterprise settings.

So I've been testing Gemini 4...

Theo - t3.gg · 2026-10-14

This video details the speaker's benchmarks and findings on Google's new Gemini 4 Argon AI model, focusing on its pricing, performance across various tasks like frontend development, 3D game generation, and complex code auditing. The speaker highlights Argon's surprisingly low accuracy and high hallucination rate in coding tasks compared to competitor models, despite its unique capabilities in vision-based tasks and integration with Google's services. He also critiques Google's pricing strategy of doubling prices after an introductory period, suggesting it discourages long-term development on their models.

read more

The speaker begins by noting the unusual 8-month gap since the last major Gemini Pro model release, with Google only releasing 'Flash' models in the interim. He introduces the new Gemini 4 Argon model, emphasizing its goal to solve hard problems in various domains including coding, finance, legal, cybersecurity, and science. Argon is touted as ranking #1 on the Vals Index, a GDP-weighted benchmark for economic sectors. However, the speaker expresses skepticism, citing discrepancies in pricing information for competitor models like Meta's 'Wonder Full' on the Vals Index, which incorrectly lists pricing for a private model.

The discussion then delves into pricing, revealing an introductory price of $2 per million input tokens and $10 per million output tokens, with a 95% discount for cached input tokens. Crucially, the price is set to double on December 1, 2026, to $4 per million input tokens and $20 per million output tokens. The speaker criticizes this strategy, stating it discourages developers from heavily building on the model due to uncertain long-term costs. He also points out that Argon is a 'token hungry' model, meaning it consumes more tokens per task, offsetting the perceived low introductory price. He notes an alarmingly low cache hit rate of 70-80% compared to 99%+ for other models, further increasing effective costs.

Regarding coding capabilities, Google claims Argon increases agentic code submissions by over 35% while reducing code rollbacks by nearly a third, and that it can refactor browser codebases in weeks instead of months. The speaker's personal experience, however, suggests these claims might be overstated, with a Google employee privately admitting to him that they still prefer older models like Opus 5.5 for coding tasks. In a Skatebench benchmark (a custom benchmark for naming skateboard tricks), Argon scored a perfect 100%, but this proved misleading due to API instability, with many errors occurring during early testing.

A more rigorous ts-rust analysis eval benchmark, where models audited a large Rust codebase, placed Argon dead last with a score of 4.38, less than half of GPT-6.1 Sol and Opus 5.5. Argon also took 43 minutes, significantly longer than other models. Most damningly, 24 out of 36 claims made by Argon were false, indicating a high hallucination rate (67%). In an auditing a real fleet failure task (a benchmark based on an 11-machine fleet workflow), Argon scored 5.31, again placing last. It made 14 false claims out of 40, translating to a 35% hallucination rate, and took 13 minutes, longer than Muse Wonder Full. The speaker finds these results extremely concerning, indicating that Argon is more likely to provide incorrect information than correct, making it unreliable for complex engineering tasks.

The speaker's personal attempts to fix a SQLite connection instability bug in T3 Code using Argon were also largely unsuccessful. Argon spent 30 minutes reading files and then proposed a 1200-line change that should have been 30 lines. It failed CI/Lint checks and then ceased to function. In contrast, Opus 5.5 solved the bug with a 21-line change. The speaker attributes these failures to 'Gemini-isms'—a tendency of Google models to over-read files, hallucinate incorrect solutions, and be generally unstable or uncooperative.

Despite these shortcomings, the speaker acknowledges Argon's unique capabilities, particularly its strong performance in vision-based tasks and integration with YouTube APIs, which are useful for his work as a YouTuber. However, he notes that many previously unique Google model strengths are now replicated or surpassed by other labs. He also criticizes the inconsistent model naming conventions within Google's ecosystem and the perceived lack of effort in making Antigravity (Google's agentic platform) developer-friendly, citing frequent crashes and difficulties in integration. He concludes that Gemini 4 Argon is not a useful model for code-related tasks and expresses a strong preference for Opus 5.5 or Muse Wonder Full due to their superior performance, reliability, and developer experience.

#OpenAI 722 #math solutions - what does mean? #agenticengineering #ai #chatgpt

Agentic Engineering · 2026-10-14

OpenAI recently released 722 mathematical solutions, generated by an unreleased frontier model, in response to 4,000 open research problems. These aren't 722 independent discoveries, but families of related results, with varying degrees of verification, including some with Lean formalization allowing mechanical checking. The significant implication is a shift in research: AI makes generating candidate discoveries cheap, while human effort remains crucial for verifying, understanding, and building upon these discoveries.

#Google paid $250,000 for this #Chrome #exploit #agenticengineering #gpt

Agentic Engineering · 2026-10-14

A team of four security researchers at the company Xbow leveraged AI agents, specifically GPT-5.6, to discover and exploit a full-chain vulnerability in Google Chrome within six days. This process, which traditionally takes intelligence agencies months, involved using different AI agents for various tasks like vulnerability discovery, exploit development, and sandbox escaping, guided by human oversight to refine directions and validate findings. Google paid Xbow $250,000 for this exploit, highlighting the accelerated pace of vulnerability discovery and the crucial role of human-AI collaboration in advanced cybersecurity. The exploit, which could lead to remote code execution on Windows machines by simply visiting a malicious webpage, underscores the evolving threat landscape where attackers can also leverage similar AI capabilities.

Anthropic launches model hardware standard for AI-driven lab control #AI #LabAutomation #Robotics

AI Honeycove · 2026-10-13

Anthropic's new Model Hardware Standard (MHS) enables AI agents like Claude to directly control physical lab equipment, drastically reducing the time it takes to connect disparate machines. By creating a shared interface, MHS streamlines the setup of complex robotic and analytical systems, moving from weeks of custom coding to just hours. This innovation allows AI to orchestrate entire lab systems, run experiments, and adjust physical parameters autonomously, promising to accelerate scientific discovery and automate research processes.

Redesigning source control at Anthropic

Simon Willison · 2026-10-13 · 3 min read

TLDR: Anthropic's AI-assisted coding drove a 12x surge in daily code merges (2,500 → 30,000/day in ~4 months), making Git unworkable at scale — so they built a custom centralized VCS called Scion optimized primarily for Claude as a code author, not human developers. The shift is significant enough that offline support was deliberately omitted, since Claude is the dominant user and is always online.

Every app on Earth is now open source... whether it likes it or not

Fireship · 2026-10-12

This video describes a satirical scenario where a developer, frustrated with Adobe's business model, reverse-engineered seven Adobe Creative Cloud apps using Rust and AI, then open-sourced them as "ArtCraft." It highlights the potential for AI-powered reverse engineering tools like "REA: Reverse Engineer Anything" to decompile and analyze proprietary software binaries, effectively making most applications "de facto open-source." The video humorously explores the implications for software development, suggesting a future where developers can rebuild custom, stripped-down versions of complex applications like Adobe Premiere Pro, and even backend services like Supabase, using AI to understand their underlying mechanisms and reimplement desired features.

OpenAI’s lead engineer was secretly a spy...

Fireship · 2026-10-14

This video describes the arrest of Abdulrahman Alfozan, an OpenAI engineer, for wiretapping an executive conference room. Despite his impressive academic and professional background (UC Berkeley, MIT, Meta, OpenAI), Alfozan exhibited remarkably poor tradecraft, using an off-the-shelf microcontroller wrapped in toilet paper and leaving behind a clear digital trail, including using his wife's ChatGPT account to research recording devices and leaving the device's MAC address in query logs. The incident, compounded by recent OpenAI internal issues, highlights internal security vulnerabilities and raises questions about his motivations, potentially linking him to a Saudi Arabian venture capital firm with ties to intelligence.

read more

The video reports on the recent arrest of Abdulrahman Alfozan, a 35-year-old Saudi national and US permanent resident, who was an Engineering Lead on the Agents Platform at OpenAI. He joined OpenAI in July 2025. The arrest, announced by the FBI, was for federal wiretapping after he allegedly planted a recording device in an executive conference room at OpenAI's headquarters.

Alfozan's alleged operation demonstrated surprisingly amateurish tradecraft for someone with his background (UC Berkeley, MIT, Meta, and OpenAI). His activities began about a month before the device was planted. He used an internal OpenAI productivity agent to research building addresses, meeting room capacities, calendars, and executive meeting schedules, including specific queries about where leadership and external partners typically meet.

Between September 24th and 30th, Alfozan reportedly used his wife's ChatGPT account to research how to build a continuous audio recorder. The queries included detailed specifications such as capturing microphone audio, saving compressed WAV recordings to an SD card, adding browser access/controls (timestamped filenames, downloading/deleting recordings, live listening), customizing Wi-Fi identity (MAC addresses, passive scanning), encrypting the SD card (NanoVol encryption), and protecting firmware/restricting USB output. The resulting device was a $23 Seeed Studio XIAO ESP32-S3 Sense microcontroller with a camera and microphone, paired with a $0.50 antenna, and concealed in toilet paper (which the video comically refers to as 'single-ply OpenAI toilet paper').

The device was installed on September 28th, 2026, by plugging it into a USB port on the back of a TV in an executive conference room. It was discovered by OpenAI's technical surveillance countermeasures team during a routine sweep on September 30th, 2026, just 48 hours later. OpenAI stated they used their 'models to understand the device and where it had come from,' which the video points out was likely trivial due to the device's MAC address and the digital breadcrumbs Alfozan left. Alfozan's wife's ChatGPT account had pasted the MAC address into queries just two days prior.

Further evidence included CCTV footage showing Alfozan near the executive conference room on six separate days leading up to the incident. To confirm, OpenAI secretly installed a camera behind the TV in the executive room, which captured Alfozan returning on October 5th. He was fired and arrested on October 9th, 2026, and charged with federal wiretapping, which carries a maximum penalty of 5 years imprisonment and a $250,000 fine.

The complaint does not speculate on Alfozan's motives or who he was working for, as wiretapping charges only require proving secret recording, not espionage. However, Alfozan's social media bio lists him as an investor at Raed Ventures, a VC firm in Riyadh, Saudi Arabia, where he is pictured alongside Prince Khalid Bin Bader Al Saud. Investigative journalist Scott Stedman highlighted that the Prince's father is a senior Saudi intelligence official, suggesting a potential link to foreign government involvement. This incident adds to a series of recent challenges for OpenAI, including scrapping a new AI model due to safety concerns and firing three safety researchers for mishandling sensitive information.