Posts Tagged ‘VoxxedDaysAmsterdam2026’
[VoxxedDaysAmsterdam2026] Framework Desktop and Incus OS: An Efficient Setup for Local LLM Deployment
Lecturer
Peter Smink works with Team Roxy and collaborates with AMD on advanced hardware solutions. He focuses on practical approaches to running AI workloads locally, emphasizing privacy, cost control, and energy efficiency through modern container and virtualization technologies.
Abstract
Running large language models locally addresses critical concerns around data privacy, operational costs, and environmental impact, yet requires careful hardware and software configuration. Peter Smink presents the combination of Framework Desktop hardware with Incus OS as a compelling solution for local LLM deployment. The discussion covers the advantages of this setup, practical implementation steps, encountered challenges, and real-world performance characteristics. Through detailed examination of the installation process, GPU passthrough configuration, and model execution, the presentation demonstrates how this approach enables efficient, private, and sustainable AI development while maintaining flexibility for broader container and virtual machine workloads.
Advantages of Local LLM Deployment with Modern Hardware and Virtualization
Local execution of large language models offers distinct benefits compared to cloud-based alternatives. Privacy concerns are eliminated when sensitive data never leaves organizational infrastructure. Operational costs become predictable without recurring API charges or token-based billing. Energy consumption can be monitored and optimized at the hardware level, supporting sustainability goals. Additionally, local setups provide greater control over model selection and experimentation, unconstrained by provider limitations or network dependencies.
The Framework Desktop platform, powered by AMD Ryzen processors with integrated AI capabilities, delivers high performance within a compact and power-efficient form factor. Configurations supporting up to 128 GB of memory and efficient power envelopes ranging from 80 watts upward make it suitable for demanding workloads while maintaining reasonable energy profiles. The modular design allows for future upgrades and ensures hardware longevity beyond specific AI use cases.
Incus OS complements this hardware by providing a modern, secure, and flexible virtualization and containerization layer. Built on immutable Linux foundations with secure boot and TPM support, it offers robust isolation for workloads. The system includes built-in update mechanisms and supports both containers and virtual machines through a unified command-line interface. This versatility allows teams to run AI inference alongside other development or production services on the same infrastructure.
Implementation Process and Configuration Details
Setting up the environment begins with client preparation. The Incus client must be installed and configured with appropriate certificates for secure communication with the target system. This client serves as the primary interface for managing the remote Incus instance.
Image creation follows, utilizing the flasher tool to generate a customized Incus OS image. Configuration files specify critical parameters including the target disk, graphics drivers for AMD hardware, and PCI passthrough settings for GPU access. The process ensures that the resulting image includes necessary drivers and security configurations tailored to the Framework Desktop hardware.
On the hardware side, BIOS preparation involves enabling secure boot and clearing existing certificates to accommodate the new installation. CPU and memory settings are optimized for the installation phase. Once the USB image is created and booted, the automated installation process configures the system according to the provided specifications.
Post-installation steps focus on network configuration and virtual machine setup. A dedicated network is established for VM connectivity. The AI server virtual machine is then created with substantial memory allocation and direct GPU passthrough. This configuration enables the virtual machine to leverage hardware acceleration for model inference.
Within the virtual machine, environment preparation includes kernel updates, installation of necessary tools, and configuration of user groups for GPU access. The latest AMD graphics drivers ensure optimal performance. Verification steps confirm that the GPU is properly recognized and accessible to the inference software.
Operational Characteristics and Performance Considerations
The resulting setup demonstrates strong performance for local inference tasks. Token generation rates vary based on model size and configuration, with observed averages ranging from 25 to 60 tokens per second and peaks approaching 90 tokens per second under optimal conditions. Power consumption remains efficient, typically between 18 and 130 watts depending on workload intensity.
The combination supports models that may exceed the capacity of individual graphics cards by leveraging system memory and CPU resources effectively. Privacy is maintained as all processing occurs locally without external data transmission. Cost predictability eliminates concerns about variable cloud pricing or usage spikes.
The platform’s reusability adds significant value. Should AI-specific requirements evolve, the hardware remains fully functional as a general-purpose server or development workstation. This contrasts with specialized AI accelerators that may become obsolete or underutilized outside narrow use cases.
Challenges and Lessons Learned
Several practical challenges emerge during implementation. Certificate and client configuration require precise sequencing to ensure secure connectivity. Incorrect order or missing privileges can result in systems that fail to communicate properly. GPU passthrough configuration demands careful attention to hardware and driver compatibility.
Kernel updates and driver installations must align with the specific hardware platform. Recent changes in driver APIs have occasionally created compatibility hurdles, though newer versions have improved stability significantly. Memory and storage requirements for both the host system and virtual machines necessitate careful planning to avoid resource contention.
Despite these considerations, the overall setup process proves manageable with appropriate documentation and systematic verification at each stage. The modular nature of both hardware and software components allows for iterative refinement as requirements evolve.
Strategic Value for AI Development and Beyond
This hardware and software combination addresses multiple strategic objectives simultaneously. Privacy-conscious organizations gain a viable alternative to cloud services without sacrificing performance. Development teams benefit from rapid iteration cycles and direct hardware access for experimentation. Cost-sensitive projects maintain predictable operational expenses while avoiding vendor lock-in.
The solution extends beyond pure AI inference. The underlying Incus platform supports diverse workloads, making the infrastructure adaptable to changing organizational needs. Teams can experiment with different models, quantization techniques, and inference engines while maintaining consistent operational procedures.
Energy efficiency aligns with growing sustainability requirements in technology operations. The ability to monitor and control power consumption at the hardware level supports both environmental goals and operational cost management.
As AI adoption continues across industries, solutions that balance capability, control, and efficiency become increasingly valuable. The Framework Desktop paired with Incus OS represents one practical approach to achieving these objectives while maintaining flexibility for future requirements.
Links:
[VoxxedDaysAmsterdam2026] Un-Observable AI Is Untrustworthy AI: Building Reliable Systems Through Comprehensive Observability
Lecturer
Annie Freeman is a Developer Advocate at Coralogix, specializing in full-stack observability platforms and the responsible deployment of AI applications. With a background in green software practices and a focus on sustainability in technology, Annie explores how visibility into AI systems can address challenges related to cost, ethics, and operational reliability.
Abstract
The rapid adoption of AI systems, particularly those involving large language models and agentic workflows, introduces significant complexities around trust, resource consumption, and ethical behavior. Traditional monitoring approaches often prove insufficient for these dynamic environments. Annie Freeman examines how observability, implemented through OpenTelemetry, can establish robust systems of trust around AI applications. By analyzing four distinct layers of observability—from development tools to quality monitoring—the discussion highlights practical strategies for instrumenting AI workloads, detecting issues such as hallucinations or policy violations, and implementing real-time guardrails. These insights enable organizations to build AI solutions that are not only performant but also accountable and sustainable.
The Fundamental Challenge: Why Traditional Monitoring Falls Short for AI
AI systems differ fundamentally from conventional software in their non-deterministic nature. The same input can produce varying outputs, agentic loops may execute unpredictable numbers of tool calls, and decision-making processes remain opaque. This unpredictability creates multiple layers of risk: potential harm from inappropriate responses, escalating operational costs from uncontrolled resource usage, and difficulties in capacity planning due to variable inference demands.
Users require consistent and reliable experiences. Company leadership must ensure investments yield clear business value without runaway expenses. Developers, increasingly reliant on AI coding assistants as production dependencies, need confidence in the generated outputs. Traditional metrics focused on uptime or basic performance fail to capture these nuances. Without targeted observability, teams operate with limited visibility into model behavior, making it impossible to verify ethical alignment or optimize resource utilization effectively.
Establishing Foundational Observability: Development and Operational Layers
Observability begins at the development stage, where AI coding tools such as Claude Code or CodeWhisperer generate substantial portions of application logic. These tools emit OpenTelemetry data natively, providing metrics on token usage, cost per session, model selection patterns, and code acceptance rates. Such visibility transforms subjective assessments of tool effectiveness into data-driven insights, enabling teams to optimize developer productivity and identify which models deliver the highest value for specific tasks.
Operational metrics extend this foundation into production environments. Key signals include token consumption trends, model invocation patterns, and response finish reasons. These indicators function analogously to HTTP status codes, revealing whether completions result from natural termination, length limits, or other constraints. High-spending users or unusual patterns, such as excessive retry loops, become immediately apparent. Organizations can then implement targeted optimizations, such as adjusting model sizes for specific use cases or imposing limits on tool call iterations.
The unified nature of OpenTelemetry ensures that AI telemetry integrates seamlessly with existing application monitoring. This avoids data silos and enables comprehensive system analysis. Teams gain the ability to correlate AI behavior with broader application performance, facilitating more informed architectural decisions.
Enhancing Decision Transparency and Real-Time Protection
Decision tracing provides critical context for understanding not just what an AI system produces but why it arrived at particular conclusions. By instrumenting agentic loops with custom spans, teams can capture detailed information about each step: input validation, prompt construction, tool selection, and reasoning chains. This granular visibility transforms black-box operations into auditable processes.
OpenTelemetry’s semantic conventions standardize the collection of this data, ensuring consistency across different AI workloads. Traces reveal the complete journey of a request, from initial user input through multiple reasoning iterations to final output. Such transparency supports debugging, compliance requirements, and continuous improvement efforts.
Quality monitoring introduces an additional safeguard layer. Small language models serve as specialized evaluators, analyzing outputs for hallucinations, toxicity, policy violations, or relevance issues. These evaluators operate with high accuracy due to their focused training, providing rapid feedback without the latency of larger models. When combined with guardrails, this approach enables real-time intervention. Suspicious inputs or outputs can be blocked before reaching users, maintaining system integrity and user trust.
Practical Implementation and Long-Term Benefits
Implementing these observability layers requires intentional design but yields substantial returns. OpenTelemetry’s vendor-neutral approach prevents lock-in while leveraging existing infrastructure investments. Teams can begin with basic instrumentation and progressively add sophistication as needs evolve.
The framework supports multiple stakeholder requirements simultaneously. Users benefit from consistent, safe interactions. Leadership gains visibility into costs and value delivery. Developers receive actionable insights for refining both AI components and their integration with business logic.
As AI adoption accelerates, observability becomes the cornerstone of responsible deployment. Systems built with comprehensive monitoring demonstrate greater reliability, ethical alignment, and operational efficiency. The investment in observability infrastructure pays dividends through reduced incidents, optimized resource usage, and enhanced organizational confidence in AI capabilities.
By treating observability as integral to AI system design rather than an afterthought, teams can move beyond experimental prototypes toward production-grade solutions that earn and maintain user trust.
Links:
[VoxxedDaysAmsterdam2026] Ouvroir de Code Potentiel: Discovering Creativity Through Constraints in Programming
Lecturer
Anders Norås is a software engineer and speaker known for exploring unconventional approaches to coding and language design. He frequently presents on topics that challenge traditional programming practices while revealing deeper insights into how languages shape thought processes.
Abstract
The Oulipo literary movement of the 1960s imposed artificial constraints on writing to spark creativity and produce novel works. Anders Norås applies similar principles to programming, reimagining familiar exercises under unusual restrictions. By removing common language features or enforcing mathematical structures, developers gain fresh perspectives on problem-solving. The presentation demonstrates how such constraints reveal hidden capabilities within languages, encourage exploration of alternative paradigms, and foster deeper appreciation for the tools we use daily. Far from mere novelty, this approach yields practical lessons about flexibility, idiom discovery, and creative thinking in software development.
The Oulipo Tradition and Its Relevance to Programming
The Ouvroir de littérature potentielle, or Workshop of Potential Literature, sought to expand creative possibilities through self-imposed rules. Authors produced works without using specific letters, following mathematical patterns, or adhering to other arbitrary constraints. These limitations, rather than stifling expression, forced innovative solutions and surprising results.
Programming shares fundamental similarities with writing. Both involve crafting structures from symbolic systems to convey meaning or achieve outcomes. Both benefit from deliberate practice and exploration of form. Just as Oulipo writers discovered new literary techniques, programmers can uncover language capabilities and problem-solving approaches by temporarily restricting their usual tools and patterns.
This method serves multiple purposes. It combats the complacency that develops from repeatedly solving problems the same way. It encourages examination of features developers might otherwise overlook. Most importantly, it transforms routine tasks into opportunities for discovery and renewed engagement with the craft.
Exploring Familiar Problems Under Novel Constraints
Classic programming exercises provide ideal subjects for constraint-based experimentation. The FizzBuzz problem, for instance, typically relies heavily on conditional statements. Removing the ability to use if statements or ternary operators forces alternative implementations that reveal language-specific idioms and capabilities.
One approach in C# leverages pattern matching within switch expressions to handle the logic. The resulting code, while unconventional, demonstrates how modern language features can replace traditional control structures. Translating the same constraint to Java yields an elegant, albeit unusual, solution using string repetition methods. These variations highlight how different languages encourage distinct thinking patterns even when solving identical problems.
Chessboard traversal problems offer another rich domain. The standard Warnsdorff’s rule provides an efficient algorithmic solution for visiting every square exactly once. However, constraint exercises might require solving the same task through exhaustive backtracking, ant colony optimization simulating natural behavior, genetic algorithms, or Monte Carlo tree search. Each method exposes different aspects of computational thinking and language expressiveness.
The value lies not in replacing established algorithms but in understanding the range of possible approaches. By deliberately limiting options, developers gain appreciation for why certain solutions feel natural in specific languages and discover techniques transferable to everyday work.
Benefits and Practical Applications
Constraint-based programming yields several advantages. It reveals language features that receive little attention in typical development. It encourages deeper engagement with syntax and semantics, moving beyond surface-level usage. It fosters creativity by breaking habitual patterns, often leading to more elegant or insightful solutions even after constraints are lifted.
Teams can apply these ideas in several ways. Code katas or brown-bag sessions focused on constrained implementations build collective knowledge and discussion. Exploring how the same problem manifests across languages strengthens architectural thinking and technology evaluation skills. Individual developers benefit from occasional deliberate practice that prevents skill stagnation.
The approach also serves as an effective teaching tool. Students or new team members gain rapid insight into language philosophy when asked to solve problems while avoiding common constructs. The resulting discussions illuminate design decisions embedded in language evolution.
Broader Implications for Software Development Culture
Modern development increasingly involves instructing AI systems rather than writing every line manually. In this environment, human strengths shift toward system thinking, trade-off analysis, and creative problem framing. Constraint exercises hone precisely these capabilities by forcing reconsideration of fundamental assumptions.
The Oulipo-inspired mindset aligns with broader movements in software craftsmanship that value deliberate practice and reflection. It echoes the Japanese concept of finding meaning in the space between elements—discovering insights that emerge when conventional approaches are temporarily set aside.
Programming communities benefit when members periodically step outside comfort zones. New libraries, paradigms, and techniques often arise from individuals willing to question established norms. By cultivating curiosity through playful constraint, developers contribute to collective advancement while maintaining personal engagement with the craft.
Embracing Constraints as Catalysts for Growth
The central insight from Oulipo applied to code is that limitations can liberate. By temporarily removing familiar tools or imposing unusual rules, programmers discover unexpected pathways and deepen their mastery of available ones. What begins as an exercise in absurdity often yields practical wisdom and renewed appreciation for the languages and techniques we employ daily.
This practice requires no special resources beyond willingness to experiment. A simple problem, a chosen constraint, and honest reflection suffice to begin. Over time, the habit of viewing constraints as invitations rather than obstacles transforms how developers approach challenges both large and small.
In a field where routine can dull creativity, the workshop of potential code offers a refreshing reminder that innovation often hides in the space between what we usually do and what becomes possible when we choose differently.