HTML is All You Need (for Agents to Make Graphics) - Amol Kapoor, Nori
This video argues that AI agents struggle with spatial reasoning in visual tools like PowerPoint or Figma, which are designed for human interaction (clicking, dragging). This limitation stems from the mismatch between how humans perceive visual layouts (graphically) and how agents process these tools (through proprietary, pixel-based data structures). The solution proposed is to leverage HTML as an intermediary: a language-based medium that models can understand and manipulate structurally, allowing them to create complex visual artifacts like slides, documents, and even videos efficiently by reasoning about meaning rather than pixels. This approach significantly speeds up content generation and eliminates manual “fiddling” for human users.
read more
The core argument presented is that coding agents are often misunderstood. While commonly perceived as only writing code, they are capable of doing almost anything. The key challenge, however, lies in understanding how to interact with them effectively. The video highlights a critical mismatch between human interaction with visual tools and AI agent capabilities.
Traditional visual design tools like PowerPoint, Google Slides, Figma, and Canva are built for human interaction. They involve actions like clicking, dragging, dropping, resizing, and snapping to a grid. These actions are intuitive for humans because we perceive and reason about the world geospatially. However, the underlying data structures of these tools are often proprietary formats that represent visual elements in terms of pixels or coordinates (e.g., XML representations of shapes and positions).
When an AI agent is tasked with generating content using these human-centric tools, the output is often poor: elements overlap, text is unreadable, and alignments are off. This is because agents, despite their advanced capabilities, lack the inherent spatial reasoning that humans possess for visual canvases. Benchmarks like ARC-AGI, designed to test spatial reasoning, reveal this weakness in current models, showing outputs that are functionally incorrect or nonsensical when asked to generate SVG (Scalable Vector Graphics), which is essentially a "wall of numbers" describing paths and shapes programmatically.
The video asserts that the problem isn't with the AI models themselves, but with the medium they are forced to interact with. Asking an AI to manipulate pixels on a canvas is akin to asking a human to hand-write SVG code – it's unnatural and inefficient. AI models think in language, in terms of words, tokens, and semantic structures. Therefore, to enable agents to create effective visual artifacts, they need to be given tools that align with their native medium of understanding.
The proposed solution is to use HTML (HyperText Markup Language). HTML offers a language-based structure for visual layout that AI models can readily comprehend and manipulate. Unlike proprietary formats, HTML elements like `<h1>`, `<div>`, `<table>`, or `<canvas>` have inherent meanings built into the language. A browser then interprets this meaningful structure and renders it into pixels. This means the AI model never directly places a coordinate or thinks about pixels; it reasons about the semantic structure, and the browser handles the visual translation. This approach provides several advantages:
Intuitive for Models: HTML's structured, semantic nature aligns with how AI models process information. Rich Visual Effects: By manipulating HTML, agents can achieve charts, layouts, fonts, and animations, all derived from the underlying structure. Flexibility and Editability: HTML is human-readable and editable, allowing for easy collaboration and fine-tuning after initial generation. Broad Compatibility: HTML renders effectively across various platforms and devices.
Nori Agentic, the company behind this video, applies this principle to generate various visual artifacts. They build all their slide decks, docs, and even videos (like the one being watched) using HTML. This allows them to create fully formatted and branded content from a single prompt. For example, a quarterly report document generated by their agent will have color, vibrancy, and adherence to brand guidelines, because the agent understands the meaning of a 'heading' or a 'note' within the HTML structure. This approach drastically reduces the "fiddling" time associated with traditional tools, freeing up human effort for higher-level thinking, vision, and storytelling.
The video concludes with a key takeaway for engineers working with AI: "Stop thinking like a user. Think like the model." This means providing agents with the right language – HTML – for graphics generation, enabling them to excel in creating visual artifacts by leveraging their strengths in structured reasoning.