Jonathan Lalou's Blog

Posts Tagged ‘MLOps’

Lecturer
Alessandro Vozza is a seasoned cloud-native advocate and technologist with deep expertise in Kubernetes and AI/ML operations. He contributes actively to open-source communities and focuses on practical, scalable deployments of generative AI workloads. As a speaker and practitioner, Alessandro emphasizes operational excellence, resource efficiency, and the integration of modern AI tools within established cloud-native platforms.

Abstract
In this hands-on tutorial at Devoxx Greece 2026, Alessandro Vozza guides developers through the complete lifecycle of running generative AI workloads on Kubernetes. From distributed training jobs with GPU scheduling to optimized inference and scalable model serving, the session demonstrates how to leverage operators, autoscaling, vector stores, and frameworks like KServe, Ray, vLLM, and Kubeflow. Attendees gain actionable insights into designing efficient GPU clusters, fine-tuning models securely, and deploying production-grade architectures that integrate seamlessly with existing Kubernetes expertise.

The Convergence of Kubernetes and Generative AI

Kubernetes has evolved into the de facto platform for orchestrating complex, resource-intensive workloads, including those powered by generative AI. Vozza begins by contextualizing the challenges: training large models demands massive parallel computation across GPUs, inference requires low-latency serving under variable traffic, and the entire pipeline must remain observable, secure, and cost-effective. Traditional approaches struggle with these demands, but Kubernetes patterns—scheduling, autoscaling, and declarative resource management—provide a robust foundation.

The session highlights how the community has responded with specialized tools. Projects like Kubeflow address the full ML lifecycle, while KServe and vLLM focus on high-performance inference. These build upon core Kubernetes capabilities, allowing teams to treat AI workloads with the same rigor applied to microservices.

Distributed Training and GPU Orchestration

Training generative models is computationally intensive and benefits enormously from Kubernetes’ scheduling strengths. Vozza demonstrates launching distributed training jobs, emphasizing GPU-aware scheduling through device plugins and resource requests. Nodes are labeled with GPU capacity, enabling the scheduler to place pods on suitable hardware.

The tutorial covers hyperparameter tuning with tools like Katib, which automates experimentation across multiple configurations. Fine-tuning involves augmenting base models with domain-specific data, a process that Kubernetes orchestrates reliably through persistent volumes and checkpointing. Attendees learn to monitor training progress using built-in observability and handle failures gracefully with retries and job controllers.

Resource efficiency emerges as a key theme. Techniques such as multi-instance GPU (MIG) partitioning allow a single physical GPU to support multiple smaller workloads, maximizing utilization without over-provisioning expensive hardware.

Inference Serving and Model Deployment

Once trained, models must be served efficiently. Vozza walks through deploying inference endpoints with KServe, which abstracts the complexities of scaling and routing. vLLM serves as the high-throughput inference engine, leveraging continuous batching and paged attention for superior performance.

The architecture supports multi-model serving, where a single deployment handles various models based on request characteristics. Gateway API extensions make the ingress layer LLM-aware, enabling intelligent routing based on factors like key-value cache state or model specialization. This ensures optimal resource allocation and minimal latency.

Autoscaling plays a critical role. Horizontal Pod Autoscaler (HPA) combined with KEDA reacts to custom metrics such as queue depth or tokens processed per second, dynamically adjusting replicas to match demand while controlling costs.

Operational Considerations and Best Practices

Production readiness demands comprehensive observability. Vozza integrates Prometheus exporters and logging to track token throughput, latency, and GPU utilization. Security best practices include least-privilege access for model endpoints and encrypted communication.

The tutorial addresses common pitfalls: managing model registries for versioning, handling cold starts through caching, and ensuring reproducibility across environments. By treating models as first-class Kubernetes citizens, teams achieve consistent deployments from development to production.

Practical Roadmap and Future Directions

Participants receive a working reference setup they can adapt immediately. Vozza encourages starting small—perhaps with a single-model inference service—before scaling to distributed training and multi-model architectures. The session reinforces that Kubernetes knowledge directly transfers to AI operations, lowering the barrier for traditional platform teams.

Looking ahead, evolving features like dynamic resource allocation and improved GPU topology awareness will further streamline GenAI workloads. The message is clear: Kubernetes is not merely compatible with generative AI; it is becoming the preferred operational layer for the entire lifecycle.

Links:

• Video: GenAI on Kubernetes
• Kubeflow
• KServe

Lecturer

Beth Torres heads strategic accounts for Eviden within the Atos Group, facilitating client alignment with artificial intelligence transformation initiatives. Kevin Davis serves as CTO of the AWS business group at Eviden, architecting machine learning operations and generative operations platforms. Eric Trell functions as AWS Cloud lead for Atos, optimizing hybrid and multi-cloud infrastructures.

Abstract

This scholarly examination articulates the distinction between conventional artificial intelligence adoption and genuine AI-first organizational identity, wherein intelligence permeates decision-making, customer engagement, and product architecture. It contrasts startup-native implementations with enterprise retrofitting, delineates MLOps/GenOps operational frameworks, and establishes ethical governance across model construction, deployment guardrails, and continuous monitoring. Cloud-enabled legacy data accessibility emerges as a pivotal enabler, alongside considerations for responsible artificial intelligence stewardship.

Conceptual Differentiation: AI Adoption versus AI-First Organizational Paradigm

The progression from cloud-first to AI-first organizational models necessitates embedding artificial intelligence as foundational infrastructure rather than peripheral augmentation. Whereas startups construct products with intelligence intrinsically woven throughout, established enterprises frequently append capabilities—exemplified by chatbot overlays—onto legacy systems.

AI-first identity manifests through operational preparedness: strategic platforms enabling accelerated use-case development by abstracting foundational complexities including data acquisition, quality assurance, and infrastructure provisioning. Artificial Intelligence Centers of Excellence institutionalize this preparedness, directing resources toward rapid return-on-investment validation through structured experimentation.

MLOps and GenOps frameworks streamline model lifecycle management at enterprise scale, addressing data integrity, ethical transparency, and governance requirements. Cloud-first positioning substantially facilitates this transition; mainframe-resident operational data, previously inaccessible for generative applications, becomes replicable to AWS environments without comprehensive modernization.

Ethical Governance and Technical Enablement Mechanisms

Responsible artificial intelligence necessitates multilayered ethical consideration. A tripartite framework structures this responsibility:

During model construction, training corpora undergo scrutiny for bias, provenance, and representativeness. Deployment guardrails leverage AWS-native capabilities to enforce content policies and contextual grounding. Continuous monitoring implements anomaly detection with predefined response protocols, calibrated according to interface interactivity levels.

\# Conceptual Bedrock guardrail implementation
import boto3

bedrock = boto3.client('bedrock-runtime')
guardrail = {
    'contentPolicy': [{'blockedTopics': ['prohibited-content']}],
    'contextualGrounding': True
}
response = bedrock.invoke_model(
    modelId='anthropic.claude-3',
    body=prompt,
    guardrailConfig=guardrail
)

Security compartmentalization within Bedrock preserves data isolation for sensitive domains such as healthcare. Production readiness extends beyond prompt efficacy to encompass data validation, accuracy verification, and misinformation mitigation within innovation toolchains.

Strategic Ramifications and Transformation Imperatives

AI-first positioning defends against startup disruption by enabling comparable innovation velocity. Ethical frameworks safeguard reputational integrity while ensuring output reliability. Cloud-mediated legacy data accessibility democratizes generative capabilities across historical systems.

Organizational consequences include systematic competitive advantage through intelligence-permeated operations, regulatory alignment via auditable governance, and cultural evolution toward experimentation-driven development. The paradigm compels reevaluation of educational curricula to incorporate technology ethics as core competency.

Links:

• Video: What is an AI-first company?

At Devoxx Poland 2022, Robert Wcisło and Łukasz Matug, data scientists at UBS, shared insights on ensuring the success of AI and NLP projects, drawing from their experience implementing AI solutions in a large investment bank. Their presentation highlighted critical success factors for deploying machine learning (ML) models into production, addressing common pitfalls and offering practical guidance across the project lifecycle.

Understanding the Challenges

The speakers noted that enthusiasm for AI often outpaces practical outcomes, with 2018 data indicating only 10% of ML projects reached production. While this figure may have improved, many projects still fail due to misaligned expectations or inadequate preparation. To counter this, they outlined a simplified three-phase process—Prepare, Build, and Maintain—integrating Software Development Lifecycle (SDLC) and MLOps principles, with a focus on delivering business value and user experience.

Prepare Phase: Setting the Foundation

Łukasz emphasized the importance of the Prepare phase, where clarity on business needs is critical. Many stakeholders, inspired by AI hype, expect miraculous solutions without defining specific outcomes. Key considerations include:

• Defining the Output: Understand the business problem and desired results, such as labeling outcomes (e.g., fraud detection). Reduce ambiguity by explicitly defining what the application should achieve.
• Evaluating ML Necessity: ML excels in areas like recommendation systems, language understanding, anomaly detection, and personalization, but it’s not a universal solution. For one-off problems, simpler analytics may suffice.
• Red Flags: ML models rarely achieve 100% accuracy, requiring more data and testing for higher precision, which increases costs. Highly regulated industries may demand transparency, posing challenges for complex models. Data availability is also critical—without sufficient data, ML is infeasible, though workarounds like transfer learning or purchasing data exist.
• Universal Performance Metric: Establish a metric aligned with business goals (e.g., click-through rate, precision/recall) to measure success, unify stakeholder expectations, and guide development priorities for cost efficiency.
• Tooling and Infrastructure: Align software and data science teams with shared tools (e.g., Git, data access, experiment logs). Ensure compliance with data restrictions (e.g., GDPR, cross-border rules) and secure access to production-like data and infrastructure (e.g., GPUs).
• Automation Levels: Decide the role of AI—ranging from no AI (human baseline) to full automation. Partial automation, where models handle clear cases and humans review uncertain ones, is often practical. Consider ethical principles like fairness, compliance, and no-harm to avoid bias or regulatory issues.
• Model Utilization: Plan how the model will be served—binary distribution, API service, embedded application, or self-service platform. Each approach impacts user experience, scalability, and maintenance.
• Scalability and Reuse: Design for scalability and consider reusing datasets or models to enhance future projects and reduce costs.

Build Phase: Crafting the Model

Robert focused on the Build phase, offering technical tips to streamline development:

• Data Management: Data evolves, requiring retraining to address drift. For NLP projects, cover diverse document templates, including slang or errors. Track data provenance and lineage to monitor sources and transformations, ensuring pipeline stability.
• Data Quality: Most ML projects involve smaller datasets (hundreds to thousands of points), where quality trumps quantity. Address imbalances by collaborating with clients for better data or using simpler models. Perform sanity checks to ensure representativeness, avoiding overly curated data that misaligns with production (e.g., professional photos vs. smartphone images).
• Metadata and Tagging: Use tags (e.g., source, date, document type) to simplify debugging and maintenance. For instance, identifying underperforming data (e.g., low-quality German PDFs) becomes easier with metadata.
• Labeling Strategy: Noisy or ambiguous labels (e.g., misinterpreting “bridges” as Jeff Bridges or drawings vs. physical bicycles) degrade model performance. Aim for human-level performance (HLP), either against ground truth (e.g., biopsy results) or inter-human agreement. A consistent labeling strategy, documented with clear examples, reduces ambiguity and improves data quality. Tools like AWS Mechanical Turk or in-house labeling platforms can streamline this process.
• Training Tips: Use transfer learning to leverage pre-trained models, reducing data needs. Active learning prioritizes labeling hard examples, while pseudo-labeling uses existing models to pre-annotate data, saving time if the model is reliable. Ensure determinism by fixing seeds for reproducibility during debugging. Start with lightweight models (e.g., BERT Tiny) to establish baselines before scaling to complex models.
• Baselines: Compare against prior models, heuristic-based systems, or simple proofs-of-concept to contextualize progress toward HLP. An 85% accuracy may be sufficient if it aligns with HLP, but 60% after extensive effort signals issues.

Maintain Phase: Sustaining Performance

Maintenance is critical as ML models differ from traditional software due to data drift and evolving inputs. Strategies include:

• Deployment Techniques: Use A/B testing to compare model versions, shadow mode to evaluate models in parallel with human processes, canary deployments to test on a small traffic subset, or blue-green deployments for seamless rollbacks.
• Monitoring: Beyond system metrics, monitor input (e.g., image brightness, speech volume, input length) and output (e.g., exact predictions, user behavior like query frequency). Detect data or concept drift to maintain relevance.
• Reuse: Reuse models, data, and experiences to reduce uncertainty, lower costs, and build organizational capabilities for future projects.

Key Takeaways

The speakers stressed reusing existing resources to demystify AI, reduce costs, and enhance efficiency. By addressing business needs, data quality, and operational challenges early, teams can increase the likelihood of delivering impactful AI-NLP solutions. They invited attendees to discuss further at the UBS stand, emphasizing practical application over theoretical magic.

Links:

• UBS company website
• Video of the talk