Harald Kirschner from the VS Code team presents on "Agents All the Way Down: How the VS Code Team Builds with What It Ships," detailing their experience with integrating AI agents into their software development lifecycle. The presentation emphasizes that while adopting AI is the easy part, the real challenge lies in surviving the increased velocity and evolving every engineering system and process around it.
Key metrics show a significant increase in developer trust in agent-written code: code survival (percentage of agent-written code that gets committed) rose from 55% (GPT-4.1 a year ago) to 86% (Claude Opus 4.6 today). This 31-point increase demonstrates growing confidence in AI-generated code. This success, however, led to new problems, primarily a massive increase in open issues (+3.4K) and open pull requests (PRs) (+850), necessitating a shift from monthly to weekly releases to match the newfound velocity.
The talk breaks down the agentic SDLC into a set of loops, each revealing the next challenge: Ship Faster, Hold Quality, and Learn Faster.
### Ship Faster: Agent-readiness as a Living System
Agent-readiness is treated not as a one-time setup but as a living system. This involves:
1. `AGENTS.md`, skills, docs: Creating documentation specifically for agents, giving them context beyond human understanding. This allows agents to quickly draft solutions.
2. Skills encode the experts: Leveraging the expertise of individual developers by encoding their knowledge into skills that every agent can run. For instance, an accessibility expert's knowledge can be formalized into a skill that runs automatically, preventing bottlenecks and ensuring broad application of best practices.
3. The litmus test: The ultimate test of agent-readiness is whether a non-engineer (like a Product Manager) can submit a reasonable PR generated by an agent that is accepted by the engineering team. This indicates genuine codebase quality and effective agent integration.
An unexpected benefit was the improvement in build times. An 8-minute build, tolerable for a human developer who can context-switch, becomes a 160-minute blocker if an agent runs 20 builds per session. The adoption of TypeScript Go, which improved build times tenfold, significantly benefited agent efficiency. This highlights that every developer annoyance is an agent wall; fixing these bottlenecks provides a bigger win for agents than for humans.
### Hold Quality at Speed
Maintaining quality at an increased release cadence is crucial. The release machinery becomes agentic infrastructure.
1. When agents change UI, they're guessing: Agents often struggle with visual consistency. To address this, VS Code developed a Component Browser that renders components outside a full build and runs VS Code with and without changes. It takes screenshots before and after modifications and posts them in the PR, catching visual and side-effect regressions. This works by leveraging tools like Playwright to automate browser interactions, and similar solutions exist for other platforms like Android and iOS. This allows engineers to quickly review UI changes without manually clicking through, accelerating the feedback loop.
2. Code review as a quality gate, not a bottleneck: Initially, AI-driven code reviews generated too much noise and were ignored by engineers. However, as the signal-to-noise ratio improved, AI reviews became mandatory on every PR. Now, 71% of feedback is actionable, and over 1 in 5 code reviews on GitHub are performed by agents, with over 12,000 organizations using it. This is a vital quality gate, ensuring human reviewers only focus on high-quality, relevant feedback. A GitHub Issue Triage Extension also helps filter and enrich issues, including identifying duplicates and suggesting labels and area owners, ultimately assigning them to the correct developer based on Git history and declared ownership. This layered AI pipeline in GitHub Actions filters spam, enriches issues (translates, requests repros, semantic deduplication), and assigns them, reducing manual triage effort.
### Learn Faster: The Prototype is the Spec
Public benchmarks weren't enough, so the team runs their own evaluations as first-class infrastructure every day.
1. VSC-Bench - our offline eval stack: This custom-built stack for VS Code's agent harness runs daily tasks, providing a 5-line smoke test to full benchmark suites. It measures task success hold while token cost drops, revealing effort calibration gaps. For example, a simple 5-character file could lead to 70x the tokens being used by the most expensive model, not necessarily the highest reasoning one. This insight is critical for understanding the true cost and efficiency of different models and harnesses.
2. Offline + online is the system: Offline evaluations have limitations, which A/B prompt experiments in production help catch. Both offline and online data are used together on every release to continuously improve the system. The team actively monitors rollouts, freezing at 8% if problems are detected, enabling quick rollbacks and target version fixes. This automate -> monitor -> adjust loop is crucial; you cannot automate and forget. Regression rate is monitored like a north star, and a small team of ~40 people manages a codebase with tens of millions of users by effectively leveraging these agentic systems.
In essence, the VS Code team's journey shows that building agents for your team, not just your users, is paramount. Adopting AI was easy; surviving your velocity is the work. The bottleneck constantly moves, and old problems don't vanish but compound, necessitating continuous iteration on the harness and feedback loops rather than just model swaps. Success is measured by asking: "What happens when you succeed?" and finding the longest feedback loops, as these represent the highest leverage for improvement.