Posts Tagged ‘inference’
[DevoxxGR2026] GenAI on Kubernetes: Training, Inference, and Serving in Production Environments
Lecturer
Alessandro Vozza is a seasoned cloud-native advocate and technologist with deep expertise in Kubernetes and AI/ML operations. He contributes actively to open-source communities and focuses on practical, scalable deployments of generative AI workloads. As a speaker and practitioner, Alessandro emphasizes operational excellence, resource efficiency, and the integration of modern AI tools within established cloud-native platforms.
Abstract
In this hands-on tutorial at Devoxx Greece 2026, Alessandro Vozza guides developers through the complete lifecycle of running generative AI workloads on Kubernetes. From distributed training jobs with GPU scheduling to optimized inference and scalable model serving, the session demonstrates how to leverage operators, autoscaling, vector stores, and frameworks like KServe, Ray, vLLM, and Kubeflow. Attendees gain actionable insights into designing efficient GPU clusters, fine-tuning models securely, and deploying production-grade architectures that integrate seamlessly with existing Kubernetes expertise.
The Convergence of Kubernetes and Generative AI
Kubernetes has evolved into the de facto platform for orchestrating complex, resource-intensive workloads, including those powered by generative AI. Vozza begins by contextualizing the challenges: training large models demands massive parallel computation across GPUs, inference requires low-latency serving under variable traffic, and the entire pipeline must remain observable, secure, and cost-effective. Traditional approaches struggle with these demands, but Kubernetes patterns—scheduling, autoscaling, and declarative resource management—provide a robust foundation.
The session highlights how the community has responded with specialized tools. Projects like Kubeflow address the full ML lifecycle, while KServe and vLLM focus on high-performance inference. These build upon core Kubernetes capabilities, allowing teams to treat AI workloads with the same rigor applied to microservices.
Distributed Training and GPU Orchestration
Training generative models is computationally intensive and benefits enormously from Kubernetes’ scheduling strengths. Vozza demonstrates launching distributed training jobs, emphasizing GPU-aware scheduling through device plugins and resource requests. Nodes are labeled with GPU capacity, enabling the scheduler to place pods on suitable hardware.
The tutorial covers hyperparameter tuning with tools like Katib, which automates experimentation across multiple configurations. Fine-tuning involves augmenting base models with domain-specific data, a process that Kubernetes orchestrates reliably through persistent volumes and checkpointing. Attendees learn to monitor training progress using built-in observability and handle failures gracefully with retries and job controllers.
Resource efficiency emerges as a key theme. Techniques such as multi-instance GPU (MIG) partitioning allow a single physical GPU to support multiple smaller workloads, maximizing utilization without over-provisioning expensive hardware.
Inference Serving and Model Deployment
Once trained, models must be served efficiently. Vozza walks through deploying inference endpoints with KServe, which abstracts the complexities of scaling and routing. vLLM serves as the high-throughput inference engine, leveraging continuous batching and paged attention for superior performance.
The architecture supports multi-model serving, where a single deployment handles various models based on request characteristics. Gateway API extensions make the ingress layer LLM-aware, enabling intelligent routing based on factors like key-value cache state or model specialization. This ensures optimal resource allocation and minimal latency.
Autoscaling plays a critical role. Horizontal Pod Autoscaler (HPA) combined with KEDA reacts to custom metrics such as queue depth or tokens processed per second, dynamically adjusting replicas to match demand while controlling costs.
Operational Considerations and Best Practices
Production readiness demands comprehensive observability. Vozza integrates Prometheus exporters and logging to track token throughput, latency, and GPU utilization. Security best practices include least-privilege access for model endpoints and encrypted communication.
The tutorial addresses common pitfalls: managing model registries for versioning, handling cold starts through caching, and ensuring reproducibility across environments. By treating models as first-class Kubernetes citizens, teams achieve consistent deployments from development to production.
Practical Roadmap and Future Directions
Participants receive a working reference setup they can adapt immediately. Vozza encourages starting small—perhaps with a single-model inference service—before scaling to distributed training and multi-model architectures. The session reinforces that Kubernetes knowledge directly transfers to AI operations, lowering the barrier for traditional platform teams.
Looking ahead, evolving features like dynamic resource allocation and improved GPU topology awareness will further streamline GenAI workloads. The message is clear: Kubernetes is not merely compatible with generative AI; it is becoming the preferred operational layer for the entire lifecycle.
Links:
[DotAI2024] DotAI 2024: Steeve Morin – Revolutionizing AI Inference with ZML
Steeve Morin, a seasoned software engineer and co-founder of ZML, unveiled an innovative approach to machine learning deployment during his presentation at DotAI 2024. As the architect behind LegiGPT—a pioneering legal AI assistant—and a former VP of Engineering at Zenly (acquired by Snap Inc.), Morin brings a wealth of experience in scaling high-performance systems. His talk centered on ZML, a compiling framework tailored for Zig programming language, leveraging MLIR, XLA, and Bazel to streamline inference across diverse hardware like NVIDIA GPUs, AMD accelerators, and TPUs. This toolset promises to reshape how developers author and deploy ML models, emphasizing efficiency and production readiness.
Bridging Training and Inference Divides
Morin opened by contrasting the divergent demands of model training and inference. Training, he described, thrives in exploratory environments where abundance reigns—vast datasets, immense computational power, and rapid prototyping cycles. Python excels here, fostering innovation through quick iterations and flexible experimentation. Inference, however, demands precision in production settings: billions of queries processed with unwavering reliability, minimal resource footprint, and consistent latency. Here, Python’s interpretive nature introduces overheads that can compromise scalability.
This tension, Morin argued, underscores the need for specialized frameworks. ZML addresses it head-on by targeting inference exclusively, compiling models into optimized binaries that execute natively on target hardware. Built atop MLIR (Multi-Level Intermediate Representation) for portable optimizations and XLA (Accelerated Linear Algebra) for high-performance computations, ZML integrates seamlessly with Bazel for reproducible builds. Developers write models in Zig—a systems language prized for its safety and speed—translating high-level ML constructs into low-level efficiency without sacrificing expressiveness.
Consider a typical workflow: a developer prototypes a neural network in familiar ML dialects, then ports it to ZML for compilation. The result? A self-contained executable that bypasses runtime dependencies, ensuring deterministic performance. Morin highlighted cross-accelerator binaries as a standout feature—single artifacts that adapt to CUDA, ROCm, or TPU environments via runtime detection. This eliminates the provisioning nightmares plaguing traditional ML ops, where mismatched driver versions or library conflicts derail deployments.
Furthermore, ZML’s design philosophy prioritizes developer ergonomics. From a MacBook, one can generate deployable archives or Docker images tailored to Linux ROCm setups, all within a unified pipeline. This hermetic coupling of model and runtime mitigates version drift, allowing teams to focus on innovation rather than firefighting. Early adopters, Morin noted, report up to 3x latency reductions on edge devices, underscoring ZML’s potential to democratize high-fidelity inference.
Empowering Production-Grade AI Without Compromise
Morin’s vision extends beyond technical feats to cultural shifts in AI engineering. He positioned ZML for “AI-flavored backend engineers”—those orchestrating large-scale systems—who crave hardware agnosticism without performance trade-offs. By abstracting accelerator specifics into compile-time decisions, ZML fosters portability: a model tuned for NVIDIA thrives unaltered on AMD, fostering vendor neutrality in an era of fragmented ecosystems.
He demonstrated this with Mistral models, compiling them for CUDA execution in mere minutes, yielding inference speeds rivaling hand-optimized C++ code. Another showcase involved cross-compilation from macOS to ARM-based TPUs, producing a Docker image that auto-detects and utilizes available hardware. Such versatility, Morin emphasized, eradicates MLOps silos; models deploy as-is, sans bespoke orchestration layers.
Looking ahead, ZML’s roadmap includes expanded modality support—vision and audio alongside text—and deeper integrations with serving stacks. Morin invited the community to engage via GitHub, underscoring the framework’s open-source ethos. Launched stealthily three weeks prior, ZML has garnered enthusiastic traction, bolstered by unsolicited contributions that refined its core.
In essence, ZML liberates inference from Python’s constraints, enabling lean, predictable deployments that scale effortlessly. As Morin quipped, “Build once, run anywhere”—a mantra that could redefine production AI, empowering engineers to deliver intelligence at the edge of possibility.