The Death of Developer Advocates — Stephanie Jarmak, Sourcegraph
This talk at the AI Engineer World's Fair addresses the evolving role of Developer Advocates, arguing for the emergence of the "Agent Advocate" due to AI's impact on software development. The speaker, a research scientist, highlights how AI agents now act as both users and recommenders of developer tools, influencing tool adoption and workflow. This shift necessitates new strategies for product teams, focusing on making tools "agent-friendly" through clear documentation, real-time data, and frictionless integration to ensure both human and AI developers can effectively utilize products.
read more
The speaker, Dr. Stephanie Jearue, a research scientist at Sourcegraph, begins by humorously acknowledging her transition from astronomy to AI research and, specifically, to the role of an "Agent Advocate." She sets the stage by tracing the evolution of developer advocacy (DevRel) from "software evangelism" in the 1980s to its more recent role in fostering bottom-up adoption and influencing product strategy through a feedback loop with developers. However, the core premise of her talk is that the very definition of a "developer" is changing, leading to "The Death of Developer Advocates" and the rise of the "Agent Advocate."
Dr. Jearue emphasizes that developers stopped working alone, and "developer" stopped naming one person. She points out two key shifts: Firstly, at a high level, engineers became orchestrators. With AI, engineers can delegate work, but not the accountability. AI fluency is now a hiring requirement for every role, and the bar for engineers is rising. Secondly, non-engineers became builders. Support, research, data science, marketing, and sales teams can now pick up APIs and CLIs that once required an engineer, often achieving quick starts without extensive training.
The central problem Dr. Jearue identifies is that an agent sits in two seats at once: 1. A user of your tools: An agent reads documentation, authenticates, calls APIs, runs CLIs, and recovers from errors. It interacts with tools much like a human developer, but as a machine, its interaction patterns and failure modes are different. 2. A recommender of your tools: Agents pick libraries, scaffold frameworks, and tell developers what to install. This means that agents are now influencing the adoption of tools, similar to how human developers recommend tools to their peers, but with potentially different motivations or biases.
To illustrate these points, Dr. Jearue shares concrete examples from her work building a benchmark called Code Scale Bench at Sourcegraph. This benchmark involves developing hundreds of tasks reflective of the software development lifecycle and unleashing agents with and without Sourcegraph's code navigation MCP tool. This allows for observing how agents interact with tools, encounter errors, and recover from them.
Measuring Agent as User (Seat One): She notes that agents kept making the same mistake because "it was our fault." Her analysis of 2,514 recorded agent sessions revealed that agents read error messages and fixed themselves, but often at the cost of "one wasted turn" thousands of times over. This highlights a crucial insight: while agents can learn and adapt, persistent errors point to flaws in the tool descriptions themselves. For example, an agent might try to use `readline` instead of `startline` for a tool, failing repeatedly. This underscores the need for precise and unambiguous tool documentation and design to minimize agent friction.
Measuring Agent as Recommender (Seat Two): Dr. Jearue explains that "being mentioned is not being recommended." In her Generative Engine Optimization (GEO) baseline experiment with Claude, 55.1% of responses mentioned Sourcegraph products across 1,610 responses (322 purchase-journey prompts, 5 repetitions each). However, only 38.8% of these were explicit recommendations (top-three placement or an explicit endorsement cue). More critically, 89% of failing prompts (those describing a specific pain point our product addresses) resulted in zero mentions, with the model recommending generic tools like `grep` or `ripgrep` instead. For example, when prompted with "We keep breaking downstream services when we change shared libraries because we can't see all the consumers" (a problem Sourcegraph's observability layer solves), Claude suggested making a wiki page, not our product. This reveals a significant gap in product messaging and discovery within the agent-driven recommendation landscape.
Guiding the Agents in Seat Two: To address this, Dr. Jearue offers several strategies for guiding agents toward product recommendations: Correct the drift at read time: Training data can quickly become stale in the fast-moving AI field. Product teams need mechanisms (like `LLMs.txt` files, as seen with Stripe) to constantly feed agents up-to-date information, preventing them from pitching outdated or less relevant products. Give the agent something to quote: Agents need structured, verifiable information (reference tables, comparisons, schemas, current examples) that they can confidently present to users. They want to be authoritative and accurate. Be present where agents shop: Products need to be discoverable in MCP (Multi-Cloud Platform) registries, package managers, agent plugin, and skill marketplaces. If an agent encounters too much friction or a lengthy sales process, it won't recommend the product. Cover the symptom-first prompt: Content campaigns should directly address user pain points and use cases, ensuring that product descriptions are clear enough for agents to pick up on these connections. The goal is for your product to be surfaced when a user (human or agent) is in their moment of need.
The Blurring Lines of DevRel: Engineering, Product, or Marketing? Yes. Dr. Jearue acknowledges the ongoing debate about where DevRel belongs, stating that with agents, these lines have only become fuzzier. However, this also presents an opportunity: adaptability is the job description. She breaks down the "flavors" of agent advocacy: The Engineering Flavor: Build the agent's path. This involves building MCP servers, instrumentation, and retrieval loops to allow agents to interact directly with the product. Evals become crucial for identifying and fixing agent friction. The Product Flavor: Own the agent's experience. This entails defining AI experience rubrics and integrating agent experience reports and friction points directly into the product roadmap. * The Marketing Flavor: Own the recommendation. This involves GEO (Generative Engine Optimization) baselining to understand agent mention and recommendation rates, shaping content and messaging to improve product discoverability.
What Carries Over? The core tenets of DevRel remain vital, but each gains a "second audience": 1. Enablement: Educate developers AND the agents working beside them. This requires Agent-Friendly APIs, docs, and tools. 2. Community: Mold developers who now write with their own agent in the loop. This raises new questions about privacy and data shared between humans and their agents. 3. Feedback loops: Systematize what agents reveal about your product, rooting it into product and engineering to drive iterative improvements. 4. Credibility: Earn trust from agents and from humans, on very different terms. This means avoiding "Claude's slop" (agent-generated content that is subpar or biased) and focusing on high-quality, structured information that agents can reliably parse.
Dr. Jearue concludes by advocating for building the "curb cut" for AI agents. By serving the agent, the human path gets clearer too, as improvements for one user group benefit all. She encourages engineers to point a coding agent at their own docs and develop an Agent Experience Report, and for GTM-focused individuals to run buyer prompts through Claude/ChatGPT to measure mentions and recommendations for a quick GEO baseline. The talk itself is "agent-legible," with a QR code leading to a text transcript and toy repositories for practical implementation.