[PyDataGlobal2025] The Boringly Simple Loop Powering Generative AI Applications
Lecturer
Sebastian Wallkötter is an AI builder and open-source maintainer. He holds a doctorate in social robotics and computer science from Uppsala University and maintains the widely used image-processing library Image.io, which records tens of millions of monthly downloads. After working as a product manager for a data platform at a Scandinavian media company he relocated to Boston; his current focus is the architectural patterns that underlie contemporary generative-AI systems.
Abstract
Beneath the proliferation of generative-AI frameworks lies a single, almost trivial control structure: a nested pair of while loops. The outer “turn” loop accumulates conversation history; the inner “trace” loop repeatedly invokes a language model until it ceases to request tools. This article dissects that core loop, shows how its presence or absence yields completions, chat assistants, agents, and co-pilots, and then examines the complementary discipline of context engineering—the principled management of the information that enters the model’s window. Together these two ideas supply a first-principles map that allows practitioners to construct sophisticated systems without mandatory dependence on any particular framework.
The Core Loop and Its Four Constituents
Every language-model call is defined by four elements: the chosen model, a system prompt that supplies instructions, a history of prior messages, and a list of available tools. History itself is an ordered sequence of typed messages—user, thinking, tool-call, tool-result, and assistant—whose legal transitions are strictly constrained. A user message is followed by an assistant message (or by intermediate thinking and tool activity); tool calls are answered by tool results; the entire block may repeat.
When these constituents are assembled into application logic the resulting control flow is remarkably simple. After initial setup the program waits for a user message (the turn loop). Upon receipt it enters an inner loop that repeatedly calls the model, executes any requested tools, appends the tool results to history, and continues until the model finally emits an assistant message (the trace loop). That assistant message is returned to the user, history is retained, and the outer loop resumes. The pattern is deliberately boring; its power derives from the fact that nearly every generative application is a variation on these two loops.
Taxonomy of Applications Derived from the Loop
Absence of both loops yields ordinary completions—single-shot tasks such as sentiment classification or automated replies. Presence of the turn loop alone produces a chat assistant capable of multi-turn conversation but unable to act on the external world. Presence of the trace loop alone produces a classical agent that can call tools yet does not accumulate conversational history across independent invocations. Presence of both loops yields what may be termed a co-pilot: an interactive assistant that both converses and acts, typical of coding agents and desktop assistants.
Because completions and pure agents possess clear start and end points they can themselves be wrapped as tools. An agent that calls other agents as tools is the elementary building block of multi-agent systems. Two dominant organizational patterns emerge. Workflows simply daisy-chain agents; a supervisor agent treats subordinate agents as ordinary tools and remains unaware of their internal structure. Reflection (an actor-critic arrangement) improves output quality by iteratively critiquing and revising intermediate results. All of these configurations remain intelligible once the underlying turn and trace loops are understood.
Context Engineering: Programming the Window
Context engineering is the disciplined practice of deciding what information enters the model’s context window so that the system remains reliable without unnecessary cost. Language models care primarily about the immediate next action—which tool to invoke or which assistant message to emit. Any material that does not contribute to that decision is surplus.
A practical system prompt therefore contains two principal sections: an ordered list of instructions and a compact knowledge base organized by descriptive headings. Tools are supplied separately, commonly via the Model Context Protocol. Contrary to frequent advice against “mega-prompts,” contemporary models tolerate surprisingly large static contexts; a window of one to two hundred thousand tokens is roughly the length of a short novel and is often sufficient for an entire application. Premature fragmentation should therefore be resisted until empirical evidence of degradation appears.
When static prompts eventually become untenable, four failure modes—collectively the “deadly sins”—signal the need for lazy loading. Context overload occurs when a long trace of prior tool calls teaches the model to repeat the same action indefinitely. Context poisoning arises when the window contains stale or incomplete information about the external world. Useless data appears when instructions or tools irrelevant to the current task remain visible and are erroneously followed. Inconsistency arises when contradictory facts coexist and occasionally trigger erratic behavior. Each of these pathologies is silent; the model simply begins to fail without raising an exception.
Lazy loading restores reliability by injecting material only at the moments it becomes necessary: at agent initialization for known constants, after the user message for information deducible from the request, and during tool execution for previously unknown dependencies. Retrieval-augmented generation, skills, sub-agents, and memory are simply named strategies for performing that lazy loading—of knowledge, of instructions, of tools, and of personalization data, respectively.
Implications for Design and Standardization
Understanding the core loop and the principles of context engineering liberates practitioners from framework lock-in. Systems can be constructed directly against model APIs, instrumented with ordinary logging, and evolved by the gradual introduction of retrieval, skills, or multi-agent coordination only when required. Standardization efforts such as the Model Context Protocol are valuable insofar as they lower the cost of experimentation, yet the field remains young; excessive early standardization risks freezing patterns that have not yet been fully explored. The most robust path is therefore to master the simple loop, manage context with deliberate frugality, and introduce complexity only when measurement demonstrates its necessity.