Daily Digest — 2026-09-18

The discussion with Emily Sands, Head of Data & AI at Stripe, centers on the emergence of the 'agent economy' and Stripe's strategic role in providing the underlying economic infrastructure for AI. The core premise is that AI agents are increasingly performing economic transactions, creating both opportunities and significant challenges, particularly around trust and fraud.

Sands categorizes the agent economy's foundational 'four rails': 1. Monetization & Billing: Moving beyond traditional subscription models to usage-based and outcomes-based billing. This is critical for AI products where marginal costs are high (e.g., inference costs) unlike traditional SaaS with near-zero marginal costs. Stripe's acquisition of Metronome is cited as an example of strengthening this rail. 2. Fraud & Abuse: Unlike traditional card fraud or chargebacks which occur at the transaction, much of the new abuse in the agent economy happens before the transaction. Examples include account creation fraud, free trial abuse, and token theft (stealing valuable digital assets, not just money). Stripe's existing Radar fraud detection product, while world-class for traditional transactions, isn't fully equipped for this pre-transactional abuse. The challenge is that fraudsters will always seek to exploit valuable assets, and now those assets include tokens and free access to AI services. Stripe is addressing this by shifting focus from scoring transactions to identifying risky customers at earlier stages (e.g., account creation, free trial initiation). 3. Agents As Independent Actors: Enabling agents to act independently requires a secure framework for them to transact. This includes features like Link: A Wallet for Agents, which allows agents to make payments on behalf of users while maintaining identity and trust. This also involves defining clear approval ladders where users can set thresholds (e.g., under $5, no approval; first $100, no approval; above limit, always approve) for agent-initiated transactions. This aims to balance automation with user control and confidence. 4. Financial Infrastructure: This rail encompasses stablecoins, stored balances, and treasury solutions necessary for micro-transactions common in the agent economy. The underlying principle is creating reliability rails that guarantee transaction integrity and foster confidence in the system.

A key observation is that Product-Led Growth (PLG), which heavily relies on free trials and easy onboarding, can break down in the agent economy if fraud is left unsolved. This forces companies to either kill free trials or revert to slower, sales-led growth models, hindering innovation. The speaker notes that 'agents don't talk to sellers,' highlighting a disconnect where automated agents (the 'missing customer') bypass traditional sales interactions, requiring new approaches to customer engagement and fraud prevention.

The discussion touches on the difference between rails for money (traditional payment processing, chargebacks) and rails for intelligence (understanding the true cost and value of AI inferences). To effectively manage the agent economy, businesses must optimize both sides: revenue & monetization, and the true cost of intelligence. This requires deep insights into the cost of running AI models and delivering outcomes, shifting from subscription-based pricing to usage-based and even outcomes-based models.

The complexity of pricing in this new economy is presented as a 'pricing brain teaser' due to: 1. Heterogeneous Value: Value differs significantly by customer. 2. Hidden Willingness to Pay: Customers may not fully reveal how much they value an outcome. 3. Dynamic Product: AI products are constantly changing, making static pricing difficult.

The 'one solvable case' for outcome-based pricing is when the outcome can be directly tied to token consumption and is conditional on useful evaluations and a measurable outcome. This allows for a pricing model based on the cost of tokens plus a markup, facilitated by a router that selects the most efficient underlying AI model based on predefined quality and cost constraints.

Sands emphasizes that the narrative around agents (e.g., the 'Hugging Face' model of autonomous, self-serving agents) often differs from the reality of how money is actually transacted. A more constrained, rational approach is currently observed in financial interactions. Stripe aims to build trust through experience, starting with small, low-risk transactions (like buying coffee) and gradually extending capabilities as confidence grows. The speaker concludes by acknowledging that the industry is still 'stumped' by these complex pricing and trust challenges, but Stripe is actively working on building the necessary infrastructure for this evolving agent economy.

AirLLM is an innovative open-source Python library designed to facilitate the local inference of large language models (LLMs) on consumer-grade hardware, specifically addressing the common limitation of GPU VRAM. Traditionally, running LLMs with tens or hundreds of billions of parameters requires substantial GPU memory, often necessitating cloud-based supercomputers or high-end professional GPUs. AirLLM overcomes this by implementing a unique layer-by-layer loading mechanism.

Instead of attempting to load the entire massive model into VRAM at once, AirLLM streams individual transformer layers from disk into CPU RAM. Once a layer is in CPU RAM, it is then transferred to the GPU for computation. After processing, the output is returned to CPU RAM, and the next layer is loaded. This sequential loading and processing strategy allows AirLLM to handle models significantly larger than the available GPU VRAM, treating the hard drive as an extended memory pool, akin to 'reading a book page by page' rather than memorizing it entirely upfront.

Key to AirLLM's efficiency is its integration of features like Flash Attention. Flash Attention is a highly optimized attention mechanism designed to reduce memory usage and increase the speed of transformer models. By keeping memory usage 'almost flat' even with long input sequences, Flash Attention complements AirLLM's layer-by-layer loading by further minimizing the temporary VRAM footprint during computation. This combined approach ensures that the total memory footprint on the GPU remains minimal at any given time, allowing for the processing of very large models on limited hardware.

With AirLLM, users can run models such as Llama 3.3 70B-Instruct directly on devices like a MacBook or a standard gaming computer equipped with a single 4GB GPU (or even larger models like 405B Llama 3.1 on 8GB VRAM). This capability democratizes access to powerful AI models, lowering the barrier to entry for:

Students and Researchers: Who can experiment with and develop on large models without requiring expensive computational resources. Indie Developers: Enabling the creation of AI-powered applications that can run locally on user devices, offering privacy benefits and reducing reliance on cloud APIs.

The library's design focuses on inference memory optimization, making it a valuable tool for edge computing scenarios or environments where internet connectivity to cloud GPUs is unreliable or undesirable. AirLLM's source code is typically available on platforms like GitHub, licensed under permissive terms (e.g., Apache-2.0), encouraging community contributions and further development. The project regularly updates its capabilities, including support for various model architectures and CPU inference, further broadening its applicability.

The discussion begins with the core question of whether AIs will eventually learn continuously and rapidly from individual deployment instances, forming a 'hive mind' of shared experience. Currently, the feedback loop for model improvement is described as 'very weak,' with occasional updates rather than real-time, continuous learning.

One perspective is that the economic incentives will be a primary driver for this shift, potentially even more than technical challenges. Companies might be hesitant to allow model providers to learn from all their deployment data if it erodes their unique business advantage. This suggests that the initial advancements in continuous learning might not involve a single, globally shared model, but rather more modular and segmented approaches.

Modular techniques like LoRA (Low-Rank Adaptation) or 'cartridges' (essentially highly compressed Key-Value caches) are proposed as immediate pathways. These methods allow for specialized, localized learning without fundamentally altering the large base model. For example, a law firm might use a base model but apply a specific 'cartridge' containing legal knowledge or operational procedures unique to their practice. This modularity could be more palatable to businesses as it preserves some proprietary advantage while still enabling model improvement.

However, the long-term vision involves more direct and continuous updates to the base model itself. The evolution is predicted to move from quarterly model releases to weekly, daily, and eventually hourly or continuous updates. This would create a scenario where the base model is constantly learning and evolving. The panel acknowledges that this continuous learning at a micro-level, where a single model is updated iteratively with small amounts of new data, currently faces significant technical hurdles. Research has shown that repeatedly updating a model with new, small datasets can lead to catastrophic forgetting, where the model loses previously learned information, and degradation of general capabilities.

Techniques like on-policy distillation have shown some promise in mitigating these issues by pushing the 'horizon' of forgetting further out. However, they don't eliminate the problem entirely. Reinforcement Learning (RL) is highlighted as being good at acquiring new capabilities but less effective at incorporating explicit factual knowledge without being destructive to existing knowledge. The challenge lies in creating methods that can tweak the model minimally to incorporate new information without causing widespread changes or forgetting. This suggests a need for techniques that can precisely target and update specific knowledge or skills within the model while preserving its overall integrity.

The panel also points out a fundamental bottleneck: the current need to retrain a model from scratch with all historical and new data to fully integrate learning and overcome the limitations of iterative updates. While taking a pre-trained base and applying mid-training with new data (plus some RL) is becoming more feasible, continuously doing this indefinitely without starting over remains a challenge. The ideal continuous learning regime, where a model forever expands its knowledge without requiring complete retraining, is still a major open research question related to model plasticity and catastrophic forgetting when faced with non-stationary data distributions. Successfully addressing these technical issues would be a significant step towards a true 'hive mind' learning environment for AI.

Legora, a collaborative AI platform for legal work, faced significant engineering challenges in scaling its search capabilities. The platform supports two main types of search: Project Search, for internal documents ranging from tens to millions, and Legal Research, which involves deep dives across billions of laws, regulations, and historical cases.

Initially, Legora used a single Elasticsearch cluster for all search workloads. This simple setup worked well for hundreds of thousands of documents, with raw documents stored in blob storage and indexed in Elasticsearch. However, as Legora expanded into new geographical regions (US, EU, APAC), they faced regulatory requirements for full processing isolation per region. This led to the adoption of a multi-region Elasticsearch setup, where each region had its own independent Elasticsearch cluster, effectively multiplying the initial simple setup by the number of regions. While this satisfied the regulatory requirements and ensured regional data residency, it introduced significant operational overhead due to managing multiple identical setups.

The next set of challenges arose from enterprise requirements, specifically the demand for full physical tenant isolation and customer-managed encryption keys. This meant each client needed their own dedicated database and the ability to control their encryption keys to encrypt and decrypt their data at rest, allowing them to revoke access to their data from Legora if needed. To address this, Legora transitioned to PostgreSQL with PGVector, using `diskANN` and `tsvector` for search. They implemented hash-partitioned tables where the hash key combined the tenant and project IDs, allowing for physical separation of tenant data within PostgreSQL. This approach streamlined operations by consolidating all data within PostgreSQL and leveraging existing OLTP isolation. However, this setup introduced its own problems, such as unbalanced partitions due to varied project sizes and access patterns (hot and cold projects on the same partitions). As data grew, PostgreSQL performance degraded significantly, leading to 20s P99 retrieval and insert latency (from initial 100s of milliseconds), primarily because the working set across partitions could not fit in memory, causing frequent cache thrashing.

This performance bottleneck led Legora to adopt Turbopuffer, an object-storage-native search engine. Turbopuffer's architecture is fundamentally different: it writes directly to object storage (e.g., S3, GCS, Azure Blob), eliminating the need for disk replication or complex consensus protocols. This makes it cost-efficient and enables unlimited read replicas with query-index separation. For Legora, Turbopuffer allowed for one namespace per project, leveraging the object storage's inherent isolation capabilities. This design led to a dramatic improvement in latencies, achieving 294ms average insert latency and 80.9ms average sparse search latency, improving performance by an order of magnitude compared to PostgreSQL. The operational simplicity was a key benefit, as a single Turbopuffer cluster could serve all tenants across different regions, with infrequent access projects residing efficiently in blob storage.

Key Learnings and Architecture Decisions:

1. Retrieval is central to legal reasoning tasks, making search performance and accuracy critical for Legora's value proposition. 2. Turbopuffer excels for Legora because it offers a simple operational model across 70+ tenants with CMK (Customer-Managed Keys) and custom data residency. This is achieved by defining a namespace per project, which maps directly to isolated storage buckets or keys within the object storage system. This allows for isolation-native design where each namespace (or organization, or even sub-organization) can have its own isolated storage and encryption keys, satisfying stringent enterprise requirements. 3. Cost-efficient retrieval for long-tail, infrequently accessed projects/legal sources is a significant advantage. Turbopuffer's object-storage-native design means that cold data can reside in cheaper, slower storage tiers, while hot data is effectively cached higher up the memory hierarchy (CPU registers, L1/L2/L3 Cache, Main Memory DRAM, NVMe SSD). This "puffing up" strategy optimizes performance by keeping frequently accessed data in faster memory tiers and less frequently accessed data in slower, cheaper storage. The goal is to maximize data pushed down the memory hierarchy to achieve the best performance-cost ratio, minimizing round trips to slower storage. 4. Vector search in Turbopuffer uses a tree-like clustering approach (similar to a complicated B-tree) to organize 100s of billions of vectors. The higher levels of the tree (centroids) are kept in faster memory, while the actual data vectors (leaves) reside in SSDs or object storage. This minimizes the latency incurred from navigating the graph, which would be prohibitive with disk-based systems due to multiple 200ms round trips. 5. Fast text search also respects the memory hierarchy. Text search queries involve looking up terms in a hash map (inverted index) and then intersecting document ID lists. Turbopuffer minimizes round trips by optimizing the retrieval and intersection of these lists. By aggressively compressing these lists and intelligently caching relevant parts of the dictionary, it reduces the memory bandwidth needed for intersections. The system dynamically decides what parts of the dictionary are most relevant and keeps them in faster memory, ensuring efficient retrieval even for very rare terms (higher scoring in BM25).

This transition allowed Legora to focus on making Legora the best product, rather than wrestling with search infrastructure and scalability challenges. Furthermore, it resulted in a happy CFO due to significantly reduced operational costs.