Daily Digest — 2026-10-07

Hossein Niazmandi, Head of Solution Engineering West at Braintrust, delivered a session at the AI Engineer World's Fair titled 'Why building agent quality platforms is hard'. He began by introducing Braintrust as a platform focused on agent quality, aiming to help teams build and maintain confidence in their AI features and agents. The core idea revolves around two key pillars: evals and observability.

Evals are performed before an AI agent reaches production. This stage is crucial for experimentation, testing agent behavior, and building confidence before deployment. Observability, on the other hand, comes into play once the agent is in production, interacting with real users. Its purpose is to take the hypotheses built during offline development and reinforce them through continuous monitoring in live scenarios. Both evals and observability are intimately related to the same problem: understanding and improving agent quality.

Niazmandi then delved into why evals are important. Large Language Models (LLMs) inherently possess extreme variability and are non-deterministic. This flexibility, while powerful, introduces significant risks. AI agents, using LLMs as their 'brain', are increasingly becoming the primary interaction point for customers. Therefore, teams need confidence that their agents will behave reliably and produce expected outcomes. Without proper evals, companies face risks related to brand consistency (inconsistent behaviors), compliance (agents saying or doing the wrong things), and high maintenance costs if systems are difficult to debug or control. The goal of evals is to reduce uncertainty before launch, ensuring a good user experience and predictable agent behavior.

He then addressed a common misconception: 'Evals is just a UI on a spreadsheet, right?' While spreadsheet-based evals can be a first step, they quickly become insufficient. At their simplest, eval systems involve: 1) executing agents against test inputs, 2) viewing outputs or scores, and 3) having a set of input examples to invoke the agent. The main advantage of spreadsheets is zero barrier to entry. However, the disadvantages are significant: it's primarily for documenting, not experimenting, it’s a slow process, and it’s not conducive to team collaboration.

The next evolution for teams is often a Vibe-coded UI. This involves a more durable persistence layer (database) and allows for human review, bringing in product managers and subject matter experts who might not be comfortable with code. This bespoke UI helps integrate more personas into the fold and accelerates iteration cycles. However, it still falls short because it does not encourage dynamic experimentation and primarily acts as a reporting tool. Teams cannot easily dynamically adjust prompts or other parameters, and there's a lack of insight into what's happening in production.

This leads to the concept of The Improvement Loop, which connects observability and evals. The ideal system allows teams to: Observe every input, output, trace, and failure in production; Analyze patterns to understand what's breaking, from whom, and why; Eval these failure patterns by turning them into test cases for offline development; and Improve the agent quality, shipping new configurations without introducing regressions. This creates a continuous hill-climbing cycle for agent quality.

Niazmandi then outlined the challenges of Fitting the flywheel together, specifically highlighting issues with agent tracing platforms: Agent traces are nasty, voluminous, and numerous. Unlike traditional observability where events are steady, deterministic, and small (1-5 KB spans), AI traces involve: high ingestion rates (up to 100k spans/sec), large, continuous writes, huge payloads (50 KB to 10 MB, sometimes GB), semi-structured data shapes with deep nesting and full-text fields, and complex read patterns that require both aggregate high-volume trends and detailed single-trace inspection with interactive filtering and querying. Traditional cloud data warehouses often struggle under this volume and load.

He concluded by emphasizing that measuring agent quality is a systems problem, not a UI/UX problem. The goal for the future is to build a system that delivers insights about the unknown-unknowns of agents, is easy to onboard for both humans and agents, incorporates non-functional requirements like RBAC and data masking, and performs observability by default through an integrated gateway. This ultimate Self-improvement loop, as depicted, will feature coding agents iteratively making changes and suggesting improvements, with humans primarily responsible for reviewing the outcomes and making deployment decisions.

Marc Klingen, co-founder of Langfuse, presented a talk titled "The Self-Improving OSS Agent Stack" at the AI Engineer World's Fair, focusing on the evolving landscape of AI agent development. He highlighted that as AI models become more capable, the traditional approach of hand-crafting prompts is being replaced by designing loops that allow agents to self-improve. Klingen envisions a future where teams operate at a higher level of abstraction, delegating more complex tasks to increasingly sophisticated models.

He outlined a reference process for building great AI agents, comprising a continuous loop with both online and offline components:

1. Online Tracing: Recording actual agent executions, including inputs, outputs, and intermediate steps, to capture real-world usage patterns and identify potential issues. 2. Online Monitoring: Analyzing traces to detect continuous issues, unexpected loops, and collect user feedback (e.g., thumbs up/down, explicit messages). 3. Offline Dataset Building: Creating or curating datasets based on observed production traces, including failing examples, to facilitate controlled experimentation. 4. Offline Experimentation: Modifying agent parameters, prompts, or tools, and running these variants against the built datasets to compare performance. 5. Offline Evaluation: Quantitatively assessing agent performance using various metrics and judges (human or AI-based) to identify improvements or regressions. Once an improved variant is found, it is then deployed back into production, restarting the loop.

Klingen emphasized that building agents differs significantly from traditional application development, largely due to the iterative and often unpredictable nature of LLMs. The challenge lies in bringing the online and offline components together effectively, as inaccurate benchmarks or out-of-sync production data can lead to suboptimal agent performance.

He presented a layered model of loops for agent development, moving from low-level token loops (what's the next token) to higher-level agent loops (read file, run install), goal loops (get data, run agent, evaluate results), meta loops (examine failures, propose fixes), and finally, a roadmap loop (find failures from traces, decide how to fix, apply fix, deploy new version). The highest levels of the roadmap loop are where humans are currently most involved.

Klingen then detailed the automation of these loops, stressing that the first thing to automate is proposing fixes. When armed with production traces, evaluation criteria, and maintained datasets, AI can be used to suggest modifications to agent implementations. More recently, teams are also leveraging AI to maintain evaluation criteria and datasets themselves, automating the generation of new test cases or refinement of existing ones based on observed failures.

However, Klingen cautioned that human oversight remains crucial, particularly in reviewing proposed changes to prevent overfitting or introducing unintended behaviors. The goal is not full automation in all layers, but rather an ideal balance of human-agent collaboration. Humans should focus on the 'judgment' aspect—setting the direction, defining failure patterns, and reviewing high-level changes—while AI handles the more tedious and repetitive tasks of generating and testing fixes. This approach shifts human effort towards higher-leverage work, leading to higher quality agents with lower time investment compared to purely manual AI engineering or fully automated but potentially low-quality solutions.

As a concrete example, Klingen demonstrated a Changelog Writer agent designed to automate the process of updating release notes. The agent takes merged Pull Requests (PRs) as input, drafts changelog entries, and then presents them for human review. User feedback (approvals or change requests) is fed back into the system, allowing the agent to continuously improve its ability to generate clear, accurate, and user-friendly changelog drafts. Langfuse's platform, with its tracing and evaluation capabilities, facilitates this continuous improvement cycle. The demo illustrated how the agent could identify and fix issues like leaking internal jargon or lacking clarity in user-facing language, demonstrating a quantifiable improvement in the `user-facing_language` metric while maintaining `format_compliance` and `accuracy`.

Klingen concluded by emphasizing that this approach drives the need for scalable data systems that can handle continuous ingestion of traces, evaluations, and user feedback. Agents will increasingly operate on and derive insights from this data, making data ownership and management critical for long-term self-improvement.

The core problem: agentic workloads on laptops are fundamentally broken

The catalyst for Theo's migration is the shift in how he uses AI coding tools. He is no longer using agents for single small edits; instead he runs end-to-end agentic loops — agent writes the change, opens a PR, waits for review, addresses feedback — cycles that can take tens of minutes to hours. Running OpenAI Codex or Claude locally on a MacBook during these workflows causes severe CPU/thermal thrashing, rendering the machine unusable for anything else. The viral "MacBook left slightly open running agents" photo resonated because he was living it. Compounding this: he frequently needs to leave his desk (events, meetings, Ubers) while work is still in flight, which is simply incompatible with a laptop as the compute host.

The economics of subscription-subsidized inference

A key analytical point drawn from Semi Analysis research shapes his tooling decision. Subscription plans for frontier models are heavily subsidized relative to actual inference cost:

- Claude Pro: $20/month plan → ~$400/month of real inference; $100 plan → ~$2,000; $200 plan → ~$8,000. - OpenAI Codex: $200/month plan → ~$700 of inference; $100 plan → ~$3,500; $200 plan → ~$14,000.

This subsidy is so large that cloud-hosted agentic IDE products like Devin or Cursor's remote execution become economically irrational for individual developers who can access these subscription plans. Those products make sense for enterprises on API pricing who can't use consumer subs, but for a solo developer or small team, the right move is to bring your own always-on compute and run the subsidized subscription models against it. This is the strategic framing behind his hardware investment.

The hardware setup

The two primary Linux machines are an HP ZBook (mentioned as the workhorse he had already been using heavily) and a Framework laptop (referred to as "BB1" / Black Box One, a new framework he just plugged back in). Both are running Linux. He also has a Mac Mini (desk mini) and a second Mac Mini running an Open Claw instance shared with his team, plus a ComfyUI rig for image generation. All machines are networked together via Tailscale, forming a private WireGuard mesh that works across locations and network conditions — including spotty 5G in an Uber — without VPN configuration overhead.

The Linux boxes have 32 threads each, giving them substantial parallelism for concurrent agent runs without the thermal and resource contention issues of Apple Silicon laptops under agentic load. The always-on nature means he can kick off a long agent loop, close his laptop, leave the building, and the work continues uninterrupted on the remote machine.

SSH workflow with persistent tmux sessions

The SSH-based workflow is the primary interface. Key setup details:

- Ghostty is the terminal emulator, wrapped with cmux (his preferred tmux wrapper for Ghostty) which provides a sidebar with pinned machine connections, notifications, and persistent pane management. - All target machines are configured so that SSH login automatically attaches to or creates a tmux session. This means any running process — whether a btop monitoring view or an active Codex agent loop — persists across disconnections. Reconnecting from anywhere instantly resumes exactly where he left off. This single change he describes as transformative for the workflow. - The pinned sidebar in cmux shows all machines (desk mini, open claw instance, BB1/Framework, ZBook, ComfyUI rig) for one-click SSH attachment.

This pattern — tmux as a session persistence layer over Tailscale SSH — is the architectural backbone. It decouples the client device (MacBook, phone, whatever is nearby) from the compute host entirely. The agent runs on the Linux box in a tmux pane; the developer's terminal is just a view into that session.

T3 Code as a GUI layer

Beyond raw SSH, he also uses T3 Code for cases where a more IDE-like or graphical interface over the remote machines is useful. He doesn't go deep on this in the transcript excerpt, but it is positioned as complementary to SSH+tmux rather than a replacement — the SSH/tmux path is the primary workhorse.

Why not just use cloud IDEs?

He explicitly evaluated Devin and Cursor's remote/cloud execution modes and finds them genuinely usable. The blocker is purely economic given the inference subsidy math above. For anyone on enterprise API pricing, or anyone who cannot access the consumer subscription plans, those products are reasonable. But for developers who can use Claude Pro or Codex subscriptions, self-hosting the compute and running agents via SSH captures the subsidy while achieving the same "close your laptop and walk away" property.

Practical implications for senior engineers

The architecture Theo describes — always-on Linux boxes + Tailscale mesh + tmux session persistence + SSH from any client — is a straightforward pattern but one that many Mac-primary developers haven't needed until agentic workloads made laptop-as-primary-compute untenable. The key insight is treating your development machine the way you'd treat a server: stateless client, stateful remote session. Tmux has always enabled this; what's new is the reason to care (multi-hour agent loops) and the network layer (Tailscale making the mesh trivial). The inference subsidy analysis is also directly actionable: before paying for a cloud agentic IDE seat, calculate whether your subscription model's effective inference budget already exceeds what that product offers.

Andrej Karpathy, a highly respected figure in the AI community known for pioneering LLMs, recently highlighted Anthropic’s Claude Tag as a groundbreaking development in LLM UI/UX. He asserts that this is the third major redesign of LLM interfaces, moving beyond the initial web-based chat interfaces and the subsequent standalone desktop applications.

Karpathy’s key insight is that Claude Tag represents a new paradigm for interacting with LLMs: it’s 'inline' with all other human activity org-wide. This means Claude isn't just a separate tool you interact with; it integrates directly into your existing communication and workflow environments, specifically mentioning Slack in the context of Claude Tag. The goal is to make LLMs 'just work' by seamlessly plugging into various tools, integrations, compute environments, memory, and security protocols.

Claude Tag essentially allows Claude to join the team in a seamless way, enabling users to communicate with it as they would with another person. This facilitates a broad range of workloads by allowing Claude to access channels and tools, and crucially, to delegate tasks while humans focus on other work. This persistent, asynchronous entity works alongside human teams, leveraging its context-awareness to provide more deeply integrated and useful assistance.

This shift is significant because it moves LLMs from being reactive, isolated tools to proactive, collaborative team members. The ability to operate 'inline' within platforms like Slack means Claude can be always-on, observing conversations, processing information, and offering assistance without requiring users to switch contexts or manually feed it data. This context-rich interaction allows for more sophisticated task delegation and support, as Claude understands the ongoing team dynamics and organizational knowledge.

For a senior software engineer, this implies several potential impacts: 1. Shift in LLM Application Design: Future applications may focus less on standalone LLM interfaces and more on deep integrations into existing enterprise software. Engineers will need to consider how LLMs can become ambient, context-aware participants in workflows. 2. Architectural Implications: Designing systems where LLMs are 'inline' means developing robust APIs, secure access protocols for sensitive organizational data (channels, tools), and efficient methods for context management and persistent memory for the LLM. 3. Human-AI Collaboration: This model encourages a more blended human-AI team, where the LLM is not just a utility but a delegated participant. This requires careful thought about how tasks are assigned, monitored, and how feedback loops are established between humans and the LLM. 4. Security and Data Governance: Integrating LLMs directly into internal communication channels raises critical questions about data privacy, access control, and ensuring the LLM operates within organizational compliance boundaries. The 'persistent, asynchronous entity' aspect means constant vigilance is required. 5. User Experience Design: The UI/UX for such integrated LLMs will need to be intuitive, allowing users to interact naturally while clearly understanding Claude's capabilities and limitations within the team context.

While the concept of an LLM as a 'slack bot' might initially sound underwhelming given Karpathy's stature, the underlying paradigm shift — making LLMs ubiquitous, context-aware, and deeply integrated within organizational workflows — is indeed a substantial leap forward in how we perceive and utilize AI in the enterprise.

The video introduces Ajax, a new uncensored AI model developed by YouTube personality PewDiePie for his Odysseus project. Odysseus is a self-hosted AI workspace for chat, agents, research, and local model workflows, boasting nearly 90,000 GitHub stars. The core concept behind Ajax is to provide a powerful, private, and unrestricted AI agent, a direct counterpoint to highly censored and controlled commercial models like those from OpenAI.

PewDiePie's motivation for creating Ajax stems from his negative experiences with existing large AI models, particularly OpenAI, which he claims banned his account twice for 'distillation'—a process of training smaller models on the outputs of larger, more intelligent models. This highlights a key tension in the AI community: the desire for open, powerful models versus the restrictive terms of service and alignment efforts of proprietary labs. The video provocatively suggests that OpenAI views distillation as a severe transgression, even worse than genocide in a metaphorical sense, due to its potential impact on their recurring revenue.

Ajax itself is a fine-tuned version of Alibaba's Qwen 3.5 9B parameter model. The goal was to create a model that serves as an 'always-on agent' for daily tasks (search, browsing, email, calendar) while ensuring complete privacy and circumventing the refusal mechanisms built into commercial models. This uncensoring was achieved through a technique called directional ablation, using a tool named Heretic. Directional ablation identifies and surgically removes specific 'abliteration parameters' within the model's neural network, effectively eliminating its refusal behaviors and allowing it to respond to queries that aligned models would reject, such as those related to manufacturing illicit substances or bypassing security measures.

The fine-tuning process for Ajax involved several key steps: 1. Supervised Fine-Tuning (SFT): This initial phase focused on teaching Ajax how to effectively use the tools within the Odysseus workspace. PewDiePie aimed for 20,000 high-quality successful agent traces for training. However, he found it extremely difficult to collect sufficient human-generated data, ultimately only gathering around 300 useful samples. This led him to resort to generating synthetic data, which was then aggressively filtered down to about 2,000 examples. He also attempted to solicit data from his community but with limited success, underscoring the challenge of building high-quality datasets for fine-tuning. 2. Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO): After SFT, GRPO was employed to further enhance the model's intelligence and performance in its specific domain. GRPO is a variant of Proximal Policy Optimization (PPO). Unlike traditional PPO, GRPO forgoes a separate 'value model' (critic/teacher) and instead estimates the baseline for learning from group scores. The model attempts the same job multiple times, scores the results, and then learns to favor outputs that perform better than the group's average. This method allows for significant reductions in training resources while still yielding a more intelligent model in its specific application domain. 3. Decensoring with Heretic and Directional Ablation: As mentioned, this final step involved using the Heretic tool to identify and remove specific 'abliteration parameters' from the fine-tuned Qwen 3.5 model. This process is designed to eliminate censorship and refusal behaviors without significantly degrading the model's core capabilities or introducing harmful biases beyond the explicit goal of removing alignment-based refusals. PewDiePie emphasized that his goal was to decensor the base model for broader utility, but not for sexual or violence compliance.

The overall project emphasizes the benefits of self-hosting and local control over AI models. By fine-tuning and uncensoring an open-source model on his own hardware, PewDiePie aims to create an AI agent that is truly private and responsive to the user's needs, without external corporate oversight, content policies, or recurring cloud fees.

The video also includes a sponsored segment for Namespace, a platform that provides 'ackshually fast' GitHub Actions runners and Devbox environments. Namespace uses custom server racks with powerful hardware (including M5 Apple silicon for Mac/iOS builds) and offers full observability, allowing users to SSH into live runners to debug issues. It also provides robust security features like egress filtering and secret injection to protect codebases and credentials from AI agents operating within dev environments. Namespace's focus on speed, observability, and security for developer workflows aligns with the broader theme of efficient and controlled local AI development discussed in the main content.