Posts Tagged ‘PeterSmink’
[VoxxedDaysAmsterdam2026] Framework Desktop and Incus OS: An Efficient Setup for Local LLM Deployment
Lecturer
Peter Smink works with Team Roxy and collaborates with AMD on advanced hardware solutions. He focuses on practical approaches to running AI workloads locally, emphasizing privacy, cost control, and energy efficiency through modern container and virtualization technologies.
Abstract
Running large language models locally addresses critical concerns around data privacy, operational costs, and environmental impact, yet requires careful hardware and software configuration. Peter Smink presents the combination of Framework Desktop hardware with Incus OS as a compelling solution for local LLM deployment. The discussion covers the advantages of this setup, practical implementation steps, encountered challenges, and real-world performance characteristics. Through detailed examination of the installation process, GPU passthrough configuration, and model execution, the presentation demonstrates how this approach enables efficient, private, and sustainable AI development while maintaining flexibility for broader container and virtual machine workloads.
Advantages of Local LLM Deployment with Modern Hardware and Virtualization
Local execution of large language models offers distinct benefits compared to cloud-based alternatives. Privacy concerns are eliminated when sensitive data never leaves organizational infrastructure. Operational costs become predictable without recurring API charges or token-based billing. Energy consumption can be monitored and optimized at the hardware level, supporting sustainability goals. Additionally, local setups provide greater control over model selection and experimentation, unconstrained by provider limitations or network dependencies.
The Framework Desktop platform, powered by AMD Ryzen processors with integrated AI capabilities, delivers high performance within a compact and power-efficient form factor. Configurations supporting up to 128 GB of memory and efficient power envelopes ranging from 80 watts upward make it suitable for demanding workloads while maintaining reasonable energy profiles. The modular design allows for future upgrades and ensures hardware longevity beyond specific AI use cases.
Incus OS complements this hardware by providing a modern, secure, and flexible virtualization and containerization layer. Built on immutable Linux foundations with secure boot and TPM support, it offers robust isolation for workloads. The system includes built-in update mechanisms and supports both containers and virtual machines through a unified command-line interface. This versatility allows teams to run AI inference alongside other development or production services on the same infrastructure.
Implementation Process and Configuration Details
Setting up the environment begins with client preparation. The Incus client must be installed and configured with appropriate certificates for secure communication with the target system. This client serves as the primary interface for managing the remote Incus instance.
Image creation follows, utilizing the flasher tool to generate a customized Incus OS image. Configuration files specify critical parameters including the target disk, graphics drivers for AMD hardware, and PCI passthrough settings for GPU access. The process ensures that the resulting image includes necessary drivers and security configurations tailored to the Framework Desktop hardware.
On the hardware side, BIOS preparation involves enabling secure boot and clearing existing certificates to accommodate the new installation. CPU and memory settings are optimized for the installation phase. Once the USB image is created and booted, the automated installation process configures the system according to the provided specifications.
Post-installation steps focus on network configuration and virtual machine setup. A dedicated network is established for VM connectivity. The AI server virtual machine is then created with substantial memory allocation and direct GPU passthrough. This configuration enables the virtual machine to leverage hardware acceleration for model inference.
Within the virtual machine, environment preparation includes kernel updates, installation of necessary tools, and configuration of user groups for GPU access. The latest AMD graphics drivers ensure optimal performance. Verification steps confirm that the GPU is properly recognized and accessible to the inference software.
Operational Characteristics and Performance Considerations
The resulting setup demonstrates strong performance for local inference tasks. Token generation rates vary based on model size and configuration, with observed averages ranging from 25 to 60 tokens per second and peaks approaching 90 tokens per second under optimal conditions. Power consumption remains efficient, typically between 18 and 130 watts depending on workload intensity.
The combination supports models that may exceed the capacity of individual graphics cards by leveraging system memory and CPU resources effectively. Privacy is maintained as all processing occurs locally without external data transmission. Cost predictability eliminates concerns about variable cloud pricing or usage spikes.
The platform’s reusability adds significant value. Should AI-specific requirements evolve, the hardware remains fully functional as a general-purpose server or development workstation. This contrasts with specialized AI accelerators that may become obsolete or underutilized outside narrow use cases.
Challenges and Lessons Learned
Several practical challenges emerge during implementation. Certificate and client configuration require precise sequencing to ensure secure connectivity. Incorrect order or missing privileges can result in systems that fail to communicate properly. GPU passthrough configuration demands careful attention to hardware and driver compatibility.
Kernel updates and driver installations must align with the specific hardware platform. Recent changes in driver APIs have occasionally created compatibility hurdles, though newer versions have improved stability significantly. Memory and storage requirements for both the host system and virtual machines necessitate careful planning to avoid resource contention.
Despite these considerations, the overall setup process proves manageable with appropriate documentation and systematic verification at each stage. The modular nature of both hardware and software components allows for iterative refinement as requirements evolve.
Strategic Value for AI Development and Beyond
This hardware and software combination addresses multiple strategic objectives simultaneously. Privacy-conscious organizations gain a viable alternative to cloud services without sacrificing performance. Development teams benefit from rapid iteration cycles and direct hardware access for experimentation. Cost-sensitive projects maintain predictable operational expenses while avoiding vendor lock-in.
The solution extends beyond pure AI inference. The underlying Incus platform supports diverse workloads, making the infrastructure adaptable to changing organizational needs. Teams can experiment with different models, quantization techniques, and inference engines while maintaining consistent operational procedures.
Energy efficiency aligns with growing sustainability requirements in technology operations. The ability to monitor and control power consumption at the hardware level supports both environmental goals and operational cost management.
As AI adoption continues across industries, solutions that balance capability, control, and efficiency become increasingly valuable. The Framework Desktop paired with Incus OS represents one practical approach to achieving these objectives while maintaining flexibility for future requirements.