Daily Digest — 2026-10-25

Sofia Jakovljevic, a Forward Deployed Engineer at Arize, presents a live demo on Agent Observability & Experimentation, emphasizing how an agent fix can inadvertently break other functionalities. Arize aims to make AI work by providing an observability backend that sheds light into the often-opaque nature of AI agents, which can be challenging to understand, predict, and debug.

Building AI systems, especially those with abstracted low-level programming managed by an agent, can be intuitive. However, this abstraction can lead to a 'black box' problem, making it difficult to understand and debug agent behavior. Observability is crucial here, allowing developers to understand step-by-step what their systems are doing, evaluate performance with metrics, and iterate for improvement.

The talk focuses on transforming agent observability into a robust regression loop that automatically tests every agent change before deployment. This proactive approach aims to prevent fixes from breaking other parts of the system. The presentation is structured into three parts:

1. Agents fail at runtime: Unlike traditional software where code changes have local effects, AI agent changes can reshape the entire execution path at runtime. A small, one-line policy edit can have widespread behavioral consequences that aren't apparent from static code review. The diff tells you what changed, but the trace tells you what it did. Production behavior, captured through traces, becomes the new source of truth for understanding how agents operate, especially in detecting silent failures—issues that only emerge during runtime and are difficult to anticipate or debug through code alone.

2. Build the regression loop: This is a 5-step process designed to move from production traces to a gate in CI/CD: Step 1: Select a dataset. Pull real inputs, tool calls, and outcomes directly from production traces. This dataset should include common workflows (representative behavior), known failures, edge cases, and crucially, successful workflows that must be protected from regression. Production behavior effectively becomes the regression suite. Step 2: Add evaluators. To move beyond 'coding on vibes' and gain quantitative insights, evaluators are used to score different dimensions of agent performance. These can include LLM-as-a-judge evaluators for subjective metrics like hallucination or factual correctness, and code-based evaluators for deterministic checks (e.g., valid tool arguments, adherence to schemas). The goal is to convert subjective qualities into numeric scores (e.g., 0-1 or 0-5), providing quantifiable metrics for task completion, action safety, and other critical dimensions. Step 3: Run the baseline experiment. The current agent is run against the selected dataset with no changes, establishing a baseline performance across all defined evaluators. This provides a control against which all future candidate changes will be compared. Arize's SDK allows for easy experiment execution, passing in the dataset, agent, and evaluators. Step 4: Make one change and rerun. Developers can now make a specific change (e.g., adjust a prompt, swap a model, modify a tool, or update a policy) and rerun the exact same experiment. The iterative process of adjust, rerun, compare allows developers to understand the impact of changes and make informed trade-offs based on the quantitative scores. This can be done for multiple changes or models, with all results tracked in Arize. * Step 5: Put the experiment in CI. The entire regression experiment is automated and integrated into the CI/CD pipeline (e.g., GitHub Actions). Every prompt, model, tool, or skill change automatically runs against the regression dataset and evaluators before it merges. Regression criteria are set on each critical dimension (e.g., 'No drop over 0.02 vs. baseline' for task completion, 'Stay at or above 0.95' for action safety). The PR will not merge until it meets these criteria. This ensures that every deployment maintains or improves performance across key metrics, fostering confidence in deployment.

3. Closing the loop at agent speed: The ultimate vision is a paradigm shift where agents themselves close the loop. Currently, human engineers are the bottleneck, manually investigating dashboards, triaging issues, and shipping fixes. The future of observability moves from helping humans debug agents to giving agents the feedback loop to continuously improve themselves. Agents will read production traces directly, self-correct, and turn failures into improvement evidence. This signifies a move towards self-improving and accountable agents that can see, evaluate, and improve their own behavior without constant human intervention.

AutoHarness aims to solve the problem of repeatedly explaining the same instructions to Claude Code by allowing the AI to self-learn skills from ongoing interactions. This GitHub plugin is designed to convert implicit user feedback into explicit, reusable instructions, making Claude more efficient and personalized over time.

The core mechanism involves capturing the refinements and corrections users make during their Claude Code sessions. Instead of users having to manually write out detailed prompts or create extensive playbooks, AutoHarness observes how users guide Claude to a desired outcome. For example, if a user consistently refines captions to be under 10 words, free of hashtags, and ending with a comment keyword in caps, AutoHarness recognizes this pattern.

Once a desired outcome is achieved, the user can signal AutoHarness to 'save the lesson'. The plugin then parses the conversation to extract the key instructions that led to the successful output. These instructions are then stored as a 'skill' within Claude. When the user later invokes this skill (e.g., by typing `/caption-rules`), Claude can immediately apply the learned instructions, significantly reducing the prompting overhead.

A key feature is conflict resolution and instruction refinement. If a new interaction introduces a rule that contradicts an existing skill (e.g., previously captions opened with a result, but now the user wants them to open with a question), AutoHarness doesn't just add a conflicting instruction. Instead, it rewrites the old rule within the skill to reflect the latest user preference. This ensures that Claude's knowledge base remains consistent and up-to-date, preventing confusion from outdated or conflicting directives.

AutoHarness also focuses on skill consolidation. It intelligently combines related lessons into broader skills. For instance, if separate lessons are created for 'post-tone', 'hook-openers', and 'no-hype-words', AutoHarness can merge these into a single, more comprehensive 'post-writing' skill. This prevents the skills list from becoming unwieldy with granular, overlapping instructions, allowing Claude to make more holistic decisions when a skill is invoked.

Beyond explicit saving, AutoHarness can learn in the background. After sufficient interaction and refinement over time, it begins to identify recurring patterns and preferences without explicit 'save' commands. For example, if a user consistently shortens captions to under 12 words across multiple sessions, AutoHarness might automatically integrate this into the 'captions' skill. This passive learning enhances Claude's adaptability and reduces the need for constant supervision.

For transparency and auditability, each instruction saved by AutoHarness retains its session evidence. Users can inspect a skill to see exactly why a particular rule was added, including the original conversation snippet or specific user feedback that led to its creation. This allows users to review and understand the evolution of Claude's learned behaviors, and to manually adjust or remove instructions if necessary. This feature is particularly valuable for senior engineers who need to understand the underlying logic and data informing an AI's autonomous actions.

The video opens by citing a post by Jeremie Harris suggesting a decent chance of the internet ceasing to exist and the rise of AI-enabled bioweapons. The speaker notes that the situation has drastically changed in the last 48 hours, with a significant increase in cyberattacks.

Recent events include Quest Diagnostics reporting a cyber breach affecting patient records. Cybersecurity experts are reportedly 'freaking out,' with one observing 14,000 hits on a honeypot server in six hours, with 'almost none of the traffic was human.' This indicates a shift from human-operated attacks to autonomous AI agents swarming login pages with 'patience and discipline of machines.' Cloudflare also reported absorbing a 4.2 terabit per second DDoS attack, the largest in recorded history. Ransomware incidents jumped 41% in a single month.

Sam Altman, in an interview with the Stanford Graduate School of Business, stated that we 'won't be able to just patch all the bugs' and need to 'completely switch the defense model.' This sentiment is echoed by Jason D. Clinton, a security expert at Anthropic, who warned of a 'vulnerability tsunami' coming in just a few months, necessitating network isolation (air-gaps and data diodes). He highlights that attacks were observed 'several days before the official announcement' of vulnerabilities, meaning adversaries were already exploiting them.

One specific vulnerability involved Claude Code exposing personal data in HTTP headers because sites 'curled politely asked agents to identify their users.' This demonstrates how intelligent agents can inadvertently leak sensitive information through seemingly innocuous interactions. While this specific bug is likely patched, the underlying problem of intelligent agents being vulnerable remains.

The speaker illustrates the common misconception of AI progress with a graph: most people perceive AI advancement linearly, expecting future capabilities to be a direct extension of current ones. However, AI progress is exponential, meaning capabilities accelerate rapidly. This leads to a dangerous gap where current defense strategies, focused on patching known bugs (the 'green line' of defense), are quickly outpaced by the rapidly evolving 'red line' of new, AI-discovered vulnerabilities and attacks. Sam Altman's point is that merely patching the bugs found by AI will not be enough because tomorrow's AI will find 'crazy bugs' that are 'outside of the normal threat model' and 'hardware-level insane stuff.'

Senator Bernie Sanders' proposal to 'Pause AI NOW!' and potentially jail researchers is presented as a political response. The speaker argues that pausing AI development would not eliminate existing open-source models or halt their continued development and exploitation by malicious actors. In fact, it would create a massive advantage for those who choose not to pause, as they would continue to develop advanced AI while others stagnate. This scenario is particularly concerning given that countries like China are actively developing and deploying AI-driven, large-scale nation-state cyber campaigns, as revealed by Microsoft Security Research. These campaigns, like 'Storm-0940,' utilize multiple open-source and Chinese models and have achieved significant access in as little as three hours.

The speaker proposes a future where AI itself is used for defense: vulnzero. The scaling laws will drive cyber mythos-class capabilities into small, cheap models and all software dev everywhere will be using models for fractions of pennies on the dollar to stop shipping security bugs permanently. This would involve AI agents automatically building and testing software for vulnerabilities before release, eliminating bugs at scale. This approach aligns with Sam Altman's vision of 'completely switch[ing] the defense model' using the 'same technology' as the attackers.

However, this introduces new challenges. If AI models become so cheap and capable that they can find and exploit vulnerabilities in real-time, the 'good guys' (defenders) need to maintain a constant advantage over the 'bad guys' (attackers). The current reactive model of patching is unsustainable, leading to an increasing number of successful cyberattacks. The internet may face temporary shutdowns in the next year to patch and clean vulnerabilities, as suggested by Amit Karp. The critical takeaway is the urgency for a proactive, AI-driven defense paradigm, where AI is used to secure systems at a scale and speed that human efforts alone cannot match, before the exponential growth of offensive AI capabilities overwhelms current defenses.

Pushkar Ranade, Intel's Chief Technology Officer, addressed the intricate challenges and limitations in scaling chip production, particularly concerning the reliance on highly specialized equipment. He emphasized that the semiconductor industry currently faces a significant bottleneck due to the limited availability of Extreme Ultraviolet (EUV) lithography tools manufactured by ASML. ASML produces fewer than 100 EUV tools per year, with no plans for a substantial increase in production even by the end of the decade. This limited supply is then distributed among major players like Intel, TSMC, and memory manufacturers such as SK Hynix, making it difficult to ramp up production to meet the soaring global demand.

Ranade explained that while EUV tools are a major bottleneck, they are not the only constraint; the entire supply chain is experiencing shortages across all equipment types. He noted that once a process flow, such as Intel’s 18A (Angstrom) node, is established, it requires specific tools. Although there might be multiple providers for other tools, the critical path often relies on a single source for particular capabilities, locking manufacturers into specific vendors for those specialized tools. This dependency restricts flexibility and rapid scaling.

Looking ahead to future nodes like 14A, Ranade highlighted the importance of maintaining fungibility in toolsets between different process nodes. For instance, transitioning from 18A to 14A might allow for the reuse of approximately 80% of existing tool sets, with the remaining 20% requiring new, specialized equipment due to denser geometries. This strategy helps optimize capital expenditure and improve production efficiency by leveraging existing infrastructure. However, he cautioned that achieving even a 1% increase in yield from an already highly optimized process is extremely challenging and requires significant effort and innovation.

Ranade also touched upon the insatiable demand from the Artificial Intelligence (AI) industry, which has seen compute power tripling year over year for a decade. He believes that the industry is constrained in its ability to meet this demand, estimating that current production capabilities could only scale up by about 2x, not 10x, even with unlimited capital investment, due to the inherent limitations of the supply chain, from equipment suppliers down to sub-suppliers (e.g., Zeiss, TRUMPF, MKS, Advanced Energy, UCT, Ferrotec for ASML). The increasing demand for AI compute means that even a few billion dollars in capital expenditure for Intel could translate into hundreds of billions in revenue for AI model companies, creating immense pressure on the manufacturing side.

Finally, Ranade observed a fundamental shift in the industry structure. Historically, the semiconductor industry has moved from a fully vertically integrated model (like 30-40 years ago) to a highly horizontal one with specialized foundries and equipment makers. However, with hyperscalers and large AI companies increasingly designing their own chips, the industry is witnessing a trend back towards a "pseudo-vertical" integration. These companies possess the capital and demand to consume entire fabs’ worth of silicon, driving a strong demand signal that is reshaping the competitive landscape and supply chain dynamics. This shift highlights the strategic importance of manufacturing capabilities and the need for innovation across the entire ecosystem to keep pace with technological advancements and market demands.

The discussion begins with a focus on Intel's partnership with Elon Musk's Terafab, a planned AI-chip fab where Intel joined as a partner in April 2026. This collaboration is driven by a shared understanding of the enormous increase in demand for silicon, both for compute and memory, necessitating expanded production capacity. Elon Musk’s desire to build his own fab stems from the recognition that his businesses, like Tesla and SpaceX, will require significant silicon, leading him to consider independent manufacturing to ensure supply.

The conversation quickly shifts to the fundamental economics of chip manufacturing, highlighting that building a fab is an extremely capital-heavy business. A visual aid shows that the cost to build a fab has been doubling approximately every 6 years since 1985, and the cost of lithography scanners—the most expensive module in a fab, crucial for patterning—is doubling even faster, roughly every 5 years. A high-end EUV (Extreme Ultraviolet) scanner and its associated equipment currently cost around $400 million, suggesting that continued cost escalation would lead to unsustainable prices in just a few years.

This escalating cost presents a significant challenge to the industry. The speaker emphasizes that chip manufacturing is a very risk-averse industry. Companies typically prefer to stick with established, proven methods that have been refined over decades. This is understandable given the tens of billions of dollars invested in each fab; any deviation from established practices carries immense financial risk. This risk aversion can hinder radical innovation, as engineers often default to 'how it's always been done,' even if better, more economical methods exist.

The speaker then posits a crucial question: Are there things that could be done very radically differently today, that weren't possible 20 years ago? This thought exercise challenges the ingrained practices of the industry. The speaker hints at questioning fundamental aspects like whether clean rooms are truly necessary to the current extent or if the infrastructure needs to be as complex as it is. While acknowledging the continuous evolution of transistor architectures (from planar to FinFETs, and now to Gate-All-Around transistors), the speaker suggests that the fundamental process flow and methodologies of chip manufacturing might be ripe for disruption.

The underlying message is that current economic trends in chip manufacturing—particularly the rapidly increasing cost of fabs and lithography tools—necessitate a reevaluation of traditional approaches. The fear of the unknown prevents incumbent companies from taking the risks required for radical innovation, despite the potential for massive cost savings and efficiency gains. The speaker implies that companies like Intel, despite their established presence, might need to adopt a 'clean slate' approach to rethink every aspect of chip production to remain competitive and meet future demand. Indeed, a month after this interview, Intel sold $23 billion of new stock to help fund new fab capacity, underscoring the urgent need for capital to expand manufacturing capabilities in response to the surging demand for AI chips. This move reflects the pressure to invest heavily in tools and capacity to avoid being bottlenecked by an undersupplied and geographically concentrated supply chain, as seen with Nvidia’s aggressive commitments for memory and other components.