This video discusses the importance of the harness (the framework or tooling) used with large language models (LLMs) for code generation. The speaker emphasizes that the harness often matters more than the specific LLM model itself, influencing whether the code generation works effectively. The best-performing harness is a constantly moving target, requiring continuous experimentation to adapt to evolving models and toolings.
TLDR: When building company AI norms, don't start by drafting policy — start by listening. Majors ran small, opt-in sessions asking open-ended questions like "what AI-related thing has annoyed you lately?", let themes emerge organically, and found that the process itself defused organizational tension before any formal document was released. The act of listening is both the research method and the change management strategy.
This video advocates for a new approach to AI model evaluation, moving beyond simple API-based black-box testing. The proposal, termed "Embedded Evaluators," suggests that third-party evaluators should be integrated directly within AI companies, having the same access and transparency as employees. This aims to eliminate information asymmetry and facilitate continuous, in-depth oversight of AI development, mirroring regulatory practices in industries like banking, and ensuring that safety checks are deeply woven into the entire training pipeline and processes.
The video introduces Jev, a new System One Model from TypeSafe AI, optimized for data processing and classification rather than text generation. Jev promises to be significantly faster and cheaper than traditional LLMs, offering type-safe, hallucination-free JSON outputs directly usable in software. It excels in tasks requiring quick, reliable decision-making based on structured data, such as real-time applications and complex workflows, by processing inputs in parallel.
The video discusses the launch of Jev, a new AI model developed by TypeSafe AI, co-founded by Diogo Almeida (co-inventor of ChatGPT and RLHF). Jev is distinct from typical Large Language Models (LLMs) as it is not designed for text generation or complex reasoning but for fast, reliable data classification and processing. The core concept behind Jev is System One Models, a reference to Daniel Kahneman's "Thinking, Fast and Slow." System One intelligence refers to fast, intuitive judgments made without effort, while System Two involves slow, deliberate reasoning. Jev embodies System One, handling tasks that require quick, direct classification.
Key features and advantages of Jev include:
Speed and Cost-Effectiveness: Jev is marketed as 20-200x faster and 40-400x cheaper than existing LLMs for its specialized tasks, with output tokens being free. This significant cost reduction and speed improvement make it viable for applications where traditional LLMs would be too slow or expensive. For instance, in a comparative demo, Jev responded to 27 questions in 0.114 seconds at a cost of $0.000001, whereas a GPT-5.6 Terra model took 8.566 seconds and cost $0.01308. This represents a 170x faster and 170x cheaper performance.
Structured, Type-Safe, and Hallucination-Free Outputs: Unlike general-purpose LLMs that often produce unstructured or hallucinated text, Jev is optimized for structured outputs. It returns JSON perfectly in a type-safe format, ensuring data integrity and predictability. The model never makes type errors and cannot hallucinate because possible outputs and their structure are defined in advance, making it highly reliable for programmatic use. This is a critical distinction for software engineers who need reliable and predictable API responses.
Use Cases and Applications: Jev is particularly suited for tasks requiring rapid, accurate classification and decision-making on structured data. Examples highlighted in the video include: Data organization and classification: Such as sorting videos by topic or prioritizing emails. Real-time applications: Its 70ms-500ms end-to-end response time (worst-case) makes it suitable for applications where UI/UX responsiveness is critical, like generating UI components based on user input or controlling a browser/game at lightning speed. Smart if-statements / AI-powered workflows: Jev can act as a fuzzy decision rule engine, classifying inputs to guide code execution paths, making traditional handwritten logic more robust and flexible. Map-reducing over big data: Efficiently processing petabytes of data to extract features and insights. Verification and guardrails: Scoring, judging, and verifying LLM outputs, detecting jailbreaks, and ensuring adherence to specific reasoning traces or formats. However, the video explicitly states that using Jev to judge* between outputs of other LLMs is not a good use case due to Jev's limited reasoning capabilities.
Technical Details and Architecture: TypeSafe AI built a new stack entirely focused on automation, incorporating a new model architecture, parallel samplers for maximum efficiency, and a training method called Reinforcement Learning for Calibrated Decisions (RLCD). Jev gives up string generation in favor of highly optimized structured outputs, processing all output probabilities in parallel rather than token-by-token. This hardware-aware design contributes to its efficiency.
Comparison with Existing LLMs and Programming Languages: The video emphasizes that Jev is not a replacement for LLMs that excel at complex reasoning or text generation (like GPT-4, Claude, or Fable). Instead, it's designed to complement them by handling the high-volume, low-latency, and structured data classification tasks. The video mentions BAML (Basically A Made-up Language) as another approach to address the challenge of interfacing agents with real-world code, aiming to define a language that agents understand and that provides a specific, type-safe output format. The fundamental difference between Jev and traditional LLMs is determinism: Jev guarantees the shape of its output, whereas LLMs are not deterministic and can sometimes hallucinate or misformat JSON, which is a critical issue for programmatic integration. Jev's strict adherence to output schema eliminates these risks, allowing for reliable integration into software systems, much like calling a function in a traditional programming language.
Anthropic's Claude Code has introduced a new 'manager' agent that can coordinate multiple coding agents, effectively creating a hierarchical structure for AI development. This coordinator can break down large objectives into parallel threads, manage project state, and route tasks to specialized sub-agents. This signifies a shift where human engineers may transition from orchestrating individual agents to defining high-level objectives and designing the agent organization itself.
Anthropic's Claude Code is evolving beyond simply providing individual coding agents, introducing a new capability that enables a manager agent to oversee and coordinate multiple sub-agents. This new feature fundamentally changes how developers interact with and leverage Claude Code, moving towards a more hierarchical and autonomous multi-agent system.
At its core, the innovation involves a persistent coordinator agent. This coordinator is given a larger, overarching objective. Its primary role is to dissect this objective into smaller, manageable tasks, which are then distributed among several specialized coding agents (referred to as 'threads'). Each of these threads operates as its own independent Claude session, potentially working on a separate copy and branch of the repository, as if multiple developers were working in parallel on different aspects of a project.
One of the most significant advancements is the coordinator's ability to maintain project-wide context and memory. It remembers what each sub-agent is doing, tracks dependencies between tasks, and intelligently routes new work to the appropriate thread or agent. This dramatically reduces the burden on the human developer, who previously had to act as the 'orchestrator,' manually managing tasks, dependencies, and individual agent interactions across separate terminal windows.
Furthermore, the sub-agents themselves retain the ability to launch their own sub-agents and workflows, as is typical with current multi-agent setups. This creates a nested hierarchy, where a top-level coordinator manages several mid-level agents, and those mid-level agents can further delegate tasks to lower-level agents or execute tools directly. This mirrors the structure of human engineering teams, where a project manager oversees team leads, who in turn manage individual contributors.
This shift implies a significant change in the role of the human engineer. Historically, engineers delegated code writing and then task execution to AI agents. With this new capability, the delegation extends to coordination and management. The engineer's responsibility moves up the abstraction ladder. Instead of meticulously defining every step or orchestrating agent interactions, the engineer will increasingly focus on defining clear, high-level objectives and designing the organizational structure of agents that can independently pursue and achieve those objectives. This could mean engineers spending more time on architectural design, objective function formulation, and strategic planning, rather than detailed task breakdown and agent supervision.
A first-hand account from an engineer at a large company describes a workplace where Claude Code has completely replaced human authorship of specs, tickets, tests, and code, with everyone from junior to senior engineers spending 12-13 hour days essentially just approving AI output without reading it. This matters because it's a concrete warning sign of how AI-assisted development can degrade into a cargo cult where shipping velocity becomes the only metric, institutional knowledge disappears, and no human actually understands what's in production.
Simon Willison argues that MCP remains valuable not for unrestricted terminal agents that can call APIs directly, but for controlled agentic systems that need scoped service access, secure credential handling, user-facing auth flows, and audit logging. This matters because it reframes MCP as a governance and security boundary layer rather than just an API abstraction, which is a meaningful architectural distinction when you're designing agent systems that can't operate in full YOLO mode.
This practitioner report by Dmitry Boyko from Maersk delves into the challenges of automating global logistics, emphasizing that the 'hard part' of AI is not just building agents, but rather transforming tribal knowledge into executable Standard Operating Procedures (SOPs). He argues against simple prompt engineering, advocating for a SOP-as-code approach within a three-component architecture: SOP Corpus, Agent Runtime, and SME Bench. The key takeaway is that production-grade AI systems require a refinement loop around agents, where human experts and engineers actively identify, diagnose, and fix failures through actionable corrections, ultimately leading to iterative improvements and a robust adaptive architecture that systematically eliminates dumb mistakes.
Dmitry Boyko's presentation, 'Tribal Dungeons of Global Shipping: How I Learned to Stop Worrying and Love the SOPs,' focuses on the practical challenges of deploying AI agents in the complex domain of global logistics, particularly within Maersk. He highlights that while 80% of logistics processes are already automated, the remaining 20% represent a 'long tail tax' of semi-automated edge cases handled across incomplete systems, relying on PDFs, spreadsheets, emails, legacy systems, local rules, and specialist judgment. This results in exception work where parallel state machines, such as booking, vessel, equipment, customs, and billing, become incoherent, requiring expert orchestration to resolve.
Boyko introduces the concept of a 'tribal dungeon': a process that exists but not as executable knowledge. This knowledge is spread across people, documents, screenshots, legacy systems, local variants, and exception rules. The core problem is that an agent cannot safely execute a process that the organization cannot represent reliably. To address this, he proposes turning this tribal knowledge into executable SOPs (Standard Operating Procedures). Unlike traditional 'legacy SOPs' which are collections of screenshots and comments instructing human actions, 'Agent SOPs' must include preconditions, a decision tree, verified identifiers, distilled backend calls + validation, a recovery path, and an evidence trail of successful execution.
He draws a clear distinction between prototype thinking (LLM -> tool call -> done) and production thinking (request -> country -> scenario -> identifiers -> dynamic tools -> guardrails -> evidence -> eval). The gap between these two is the hard part, requiring stable intent detection, safe tool calls, integration with legacy backends, and expert evaluation of results. The solution lies in a three-component architecture: 1. SOP Corpus: Executable process memory. 2. Agent Runtime: Classifier + execution loop + traces + dynamic tools. 3. SME Bench: Clustering traces + evaluation + correction + replay.
The critical insight is that 'the agent loop is not the system; the refinement loop around the agent is the system.' This involves a continuous cycle of observation, diagnosis, change, replay, deployment, and measurement.
Boyko emphasizes that SOP-as-code beats prompt engineering. While prompt engineering relies on 'one big instruction,' SOP-as-code is versioned, tool-scoped, replayed, reviewed, deployed, measured, country-aware, and linted. The SOPs are the asset: a living knowledge base adapted to the company's processes, becoming its process memory. For Maersk, this has resulted in an SOP corpus that is 20:1 ratio to runtime code, covers 110 countries and 25+ languages, and manages ~10K process intents/scenarios.
He demonstrates the system's operational scale with metrics showing over 200 active container instances handling requests with varying latency (from minutes to over 10 minutes, mainly due to legacy system dependencies). The Evaluation UX focuses on transforming 'noise to patterns' by clustering failed/rejected traces to provide SME correction and generate SOP change requests. The 'trace is the evidence object,' carrying details like intent, routing, SOP branch, verified facts, tools called, decision, output, and failure reason, enabling experts and engineers to agree on 'what happened.'
Corrections become executable changes, moving from generic responses to specific, rule-based actions with readiness checks and fallback templates. This leads to replayable results where real examples are replayed (with writes disabled) to verify that behavior improved. This process ensures production quality came from iteration, not initial design. With 100K+ incremental corrections over nine months, coverage and accuracy are 'earned, not designed.'
The cognitive effort ratio highlights that AI coding is only 10% of the effort, specs/SOP refinement is 20%, and evaluation/replay/testing/trace review is a significant 80%. This includes practices like TDD/BDD, linters, pairwise testing, data analysis, simulations, and SME feedback loops. The core idea is that 'AI coding made it viable. The work is everything around the code.'
Finally, Boyko concludes that 'discovery needs flexibility. Production needs a cage.' This harness doesn't give the agent more freedom but 'makes dumb mistakes structurally impossible' through mechanisms like verified identifiers, tool scoping, write gates, loop breakers, approvals, and replay before deploy. Each failure type (wrong shipment, workflow, tool, write, loop, SOP edit, assumption) is mapped to a structural prevention. Critical paths stay supervised through runtime gates and specialist reviews, ensuring safe handoffs and evidence trails.
The overall outcome is a methodology, not a bot: 1. Make work representable: Collect data & tribal knowledge into executable SOPs. 2. Make execution bounded: Tools -> scopes -> gates -> approvals. 3. Make behavior observable: Every run produces a trace. 4. Make correction cheap: SME feedback -> SOP diff -> replay. 5. Make improvement compound: Daily refinements -> reusable workflows as tools.
This methodology applies anywhere an enterprise has long-tail exceptions across systems, documents, and local rules. The AI agent is 'rented intelligence'; the adaptive architecture is the asset.
TLDR: Agent configuration files (CLAUDE.md, AGENTS.md, skill files) degrade in value over time as you keep appending rules reactively, often ballooning past 200 lines and actively hurting performance. Anthropic's own data shows they removed 80%+ of Claude Code's system prompt with no measurable quality loss, and research on personalized skills found they offer little advantage over generic ones. Treat these files like code: audit regularly, delete aggressively, and if a rule truly must hold, encode it in a test or hook rather than prose.
TLDR: Sun Microsystems failed not from bad strategy but from operational indifference — they had a startup ready to buy hardware in bulk, running their open-source software, and couldn't even return the call. The core lesson: a company that becomes bored with the basic mechanics of doing business (sales, customer responsiveness, execution) will fail regardless of how strong its technology or vision is.