Daily Digest — 2026-07-25

Sam Shillace makes a compelling point about the difficulty of measuring the Return on Investment (ROI) for Artificial Intelligence, drawing a parallel to the early days of Google Docs. He argues that true innovation, especially in emerging fields like AI and Agentic Engineering, often doesn't have a clear, measurable path from the outset; it only appears obvious in retrospect, after someone has already navigated the complexities and unknowns.

He uses the example of Google Docs. Before it became an obvious product category, it was essentially an editable webpage running on slow, unreliable JavaScript, without modern frameworks or mature browsers, and lacking a familiar mental model. A collaborative online document didn't sound inevitable; it sounded confusing and experimental. Similarly, today's AI, particularly in Agentic Engineering, feels analogous. Companies are asking for ROI, which is reasonable, but this question implicitly assumes we already understand what the tool is supposed to be used for and how its value will be realized.

Consider an engineer using AI agents. These agents could be used to analyze a slow compliance workflow, improve the agent harness itself, document an unfamiliar codebase, brainstorm designs, or even build a personal website – all in parallel. How does one calculate the return on work that would not have been attempted at all without these agents? Is the ROI enormous because the work actually happened, or zero because it was never in the original plan? This highlights a fundamental flaw in traditional ROI measurement when applied to transformative technologies.

The core insight is that early in a technology shift, measurement is usually based on the old operating model. We tend to measure if AI performs existing tasks faster. However, the more significant effect, especially in Agentic Engineering, is that engineers start doing entirely different work. This could involve exploring more ideas, investigating problems that were previously too expensive or complex to even consider, and concurrently running several streams of execution that would be impossible with traditional methods.

This isn't to say every AI experiment is valuable; some are undoubtedly wasteful. However, demanding a clean business case before learning what the technology actually makes possible is akin to asking for productivity metrics of early Google Docs before people even understood what collaborative editing software was. Clear paths and obvious business cases mostly exist in retrospect. The real skill in Agentic Engineering right now isn't predicting the final workflow but building an organization that can effectively explore the thicket – navigating the unknown and discovering new possibilities – without pretending every experiment is already a clear road through it.

Last week, Chinese AI lab Moonshot launched Kimi K3, a new open-source large language model that has significantly impacted the AI landscape. Kimi K3 boasts an impressive 2.8 trillion parameters and a massive 1-million-token context window, instantly making it the largest open-source model available, parameter-mogging all previous open models.

Architecturally, Kimi K3 is a Mixture of Experts (MoE) model, similar to advanced models like Claude Fable and GPT-5.6 Sol. It features 896 total experts, with precisely 16 experts activating per token. Moonshot claims this MoE sparsity design provides an approximate 2.5x improvement in overall scaling efficiency compared to its predecessor, Kimi K2, allowing for more efficient conversion of compute into intelligence.

In terms of performance, Kimi K3 shows highly competitive results, particularly in coding benchmarks. On DeepSWE and Terminal Bench 2.1, it achieves scores of 87.3 and 88.3 respectively, placing it ahead of Claude Fable and GPT-5.6 Sol. On Program Bench, Kimi K3 scores 77.8, closely trailing GPT-5.6 Sol's 77.9. It also ranks first on the Code Arena | WebDev leaderboard with an Elo score of 1678, surpassing Claude Fable 5 and GPT-5.6 Sol.

However, the benchmark results should be viewed with some skepticism, as Moonshot primarily used its own Kimi Code harness for K3 evaluations, while competitors were run on different harnesses. Moonshot itself acknowledges that Kimi K3 still trails Fable and GPT-5.6 Sol overall, especially on benchmarks like Humanity’s Last Exam, where it scores about 10 points lower. Furthermore, Artificial Analysis measured a 51% hallucination rate for Kimi K3, which is a significant concern, particularly for coding tasks. The model also tends to be "token happy," generating more tokens than necessary, which could lead to increased inference costs despite its lower per-token pricing compared to competitors.

Despite these caveats, Kimi K3 demonstrates impressive capabilities in areas like UI design and data visualization, performing exceptionally well for an open model. For example, it can generate detailed UI designs for applications like "Horse Tinder," complete with userflows and interactions.

Moonshot announced that the full model weights will be released by July 27, 2026, enabling self-hosting. However, running such a massive model would require substantial data center-grade GPUs, making it inaccessible for personal gaming hardware. This open-weight release from a Chinese lab like Moonshot, alongside similar releases such as Alibaba's Qwen 3.8 (2.4T parameters, open-weight), is a significant development in the AI space.

Geopolitically, this trend is creating a clear divide. China's Communist Party is increasingly positioning itself as an advocate for free and open artificial intelligence, releasing powerful models and their weights. In contrast, Western frontier AI labs and regulatory bodies are pushing for more AI regulation and gatekeeping, citing potential dangers and risks, including the "fear narrative" of job displacement and national security concerns. Dean Ball of OpenAI, for instance, has argued that open-weight models are "decelerationist," a sentiment reminiscent of Steve Ballmer's critiques of Linux in the 1990s. The underlying driver for this divergence is often seen as economic, with closed models allowing frontier labs to maintain control over revenue streams.

For developers and engineers, the availability of increasingly powerful open-weight models from Chinese labs presents both opportunities and challenges. While it democratizes access to advanced AI, the practicalities of self-hosting, managing hallucination rates, and optimizing token usage remain critical considerations. The accelerating arms race also means that AI capabilities are evolving rapidly, pushing the boundaries of what open models can achieve.