Posts Tagged ‘AlexSoto’
[SpringIO2025] Taming Testing of AI apps by Alex Soto
Lecturer
Alex Soto is the Director of Developer Experience at Red Hat, a Java Champion, and an advocate for open-source software. With over 17 years in the tech industry, he specializes in Java development, software automation, and AI integration. Soto is a prolific author, having co-authored books like “Applied AI for Enterprise Java Developers” and “Quarkus Cookbook,” and he frequently speaks on testing, cloud-native applications, and AI challenges.
Abstract
This article examines the complexities of testing AI-integrated applications, addressing challenges like non-deterministic outputs, hallucinations, and bias. It discusses strategies for ensuring reliability, including synthetic data generation, evaluation metrics, and model-assisted testing. Drawing on practical examples, it highlights methodologies for validating both deterministic and probabilistic components, emphasizing the role of data scientists and robust testing frameworks in building trustworthy AI systems.
Challenges in Testing AI-Integrated Applications
Integrating large language models (LLMs) into applications introduces unique testing hurdles, primarily due to their non-deterministic nature. Responses from models like GPT or Grok vary even for identical inputs, complicating assertions. For instance, querying an image might yield “cat” one time and “kitten” another, rendering strict equality checks ineffective. This unpredictability stems from the probabilistic architecture of LLMs, which prioritize generating plausible answers over consistency.
Hallucinations exacerbate this: models may produce inconsistent outputs (e.g., “Alex is tall and short”), input-output mismatches (e.g., rude responses despite politeness prompts), or factually incorrect information (e.g., “the Earth is flat”). Such behaviors, akin to journalists offering opinions on unfamiliar topics, necessitate specialized testing to detect and mitigate risks.
Traditional testing paradigms falter here, as AI components act as “black boxes.” Developers must treat models as external services, focusing on integration points while acknowledging limited control over internal mechanics.
Strategies for Handling Non-Determinism and Hallucinations
To address non-determinism, employ evaluation metrics over binary pass/fail. Tools like Ragas compute faithfulness (alignment with context), answer relevance, and contextual precision. For example, in retrieval-augmented generation (RAG), Ragas assesses if responses accurately reflect retrieved documents, using scores from 0 to 1.
Synthetic data generation enhances testing realism. LLMs can create diverse datasets, simulating user inputs without privacy concerns. In a pet clinic demo, a model populates forms with realistic personas, verifying outputs against expectations.
For hallucinations, chain-of-thought prompting guides models toward reasoned responses, reducing errors. Assertions check for inconsistencies, such as ensuring polite outputs or factual accuracy via external verifiers.
Code for Ragas evaluation in Java:
import dev.langchain4j.rag.query.Query;
import io.ragas.RagasEvaluator;
RagasEvaluator evaluator = new RagasEvaluator();
Query query = new Query("What is Spring Boot?");
String response = model.generate(query);
double faithfulness = evaluator.evaluateFaithfulness(response, context);
assert faithfulness > 0.8;
This quantifies response quality, enabling threshold-based assertions.
Model-Assisted Testing and Integration Approaches
Leverage AI for test creation and execution. Tools like MCPlaywright use models to script browser interactions, generating tests dynamically. In the pet clinic example, prompts instruct models to navigate, fill forms with synthetic data, and verify tables, outputting pass/fail.
Involve data scientists early for model-specific insights, ensuring tests cover bias and drift. Test deterministic parts (e.g., API routing) separately from AI components, using mocks for isolation.
Be resource-conscious: unnecessary politeness in prompts wastes compute (e.g., “thank you” equates to energy for three water bottles). Focus on rude, direct interactions for efficiency.
Implications for Reliable AI Development
Testing AI apps demands a paradigm shift toward probabilistic validation, blending traditional unit tests with advanced evaluators. Synthetic data and model-assisted tools democratize realistic testing, but require strong testing fundamentals. As AI permeates critical systems, these strategies ensure fairness, safety, and robustness, mitigating risks like hallucinations in production.
Future directions include AI-driven test optimization, reducing human effort while enhancing coverage. Developers must balance innovation with rigor, treating AI as an enhancement rather than a core dependency.
Links:
[SpringIO2019] Zero Downtime Migrations with Spring Boot by Alex Soto
Deploying software updates without disrupting users is a cornerstone of modern DevOps practices. At Spring I/O 2019 in Barcelona, Alex Soto, a prominent figure at Red Hat, delivered a comprehensive session on achieving zero downtime migrations in Spring Boot applications, particularly within microservices architectures. With a focus on advanced deployment techniques and state management, Alex provided actionable insights for developers navigating the complexities of production environments. This post delves into his strategies, enriched with practical demonstrations and real-world applications.
The Evolution from Monoliths to Microservices
The shift from monolithic to microservices architectures has transformed deployment practices. Alex began by contrasting the simplicity of monolithic deployments—where a single application could be updated during off-hours with minimal disruption—with the complexity of microservices. In a microservices ecosystem, services are interconnected in a graph-like structure, often with independent databases and multiple entry points. This distributed nature amplifies the impact of downtime, as a single service failure can cascade across the system.
To address this, Alex emphasized the distinction between deployment (placing a service in production) and release (routing traffic to it). This separation is critical for zero downtime, allowing teams to test new versions without affecting users. By leveraging service meshes like Istio, developers can manage traffic routing dynamically, ensuring seamless transitions between service versions.
Blue-Green and Canary Deployments
Alex explored two foundational techniques for zero downtime: blue-green and canary deployments. In blue-green deployments, a new version (green) is deployed alongside the existing one (blue), with traffic switched to the green version once validated. This approach minimizes disruption but risks affecting all users if the green version fails. Canary deployments mitigate this by gradually routing a small percentage of traffic to the new version, allowing teams to monitor performance before a full rollout.
Both techniques rely on robust monitoring, such as Prometheus, to detect issues early. Alex demonstrated a blue-green deployment using a movie store application, where a shopping cart’s state was preserved across versions using an in-memory data grid like Redis. This ensured users experienced no loss of data, even during version switches, highlighting the power of stateless and ephemeral state management in microservices.
Managing Persistent State
Persistent state, such as database schemas, poses a significant challenge in zero downtime migrations. Alex illustrated this with a scenario involving renaming a database column from “name” to “full_name.” A naive approach risks breaking compatibility, as some users may access the old schema while others hit the new one. To address this, he proposed a three-step migration process:
- Dual-Write Phase: The application writes to both the old and new columns, ensuring data consistency across versions.
- Data Migration: Historical data is copied from the old column to the new one, often using tools like Spring Batch to avoid locking the database.
- Final Transition: The application reads and writes exclusively to the new column, with the old column retained for rollback compatibility.
This methodical approach, demonstrated with a Kubernetes-based cluster, ensures backward compatibility and uninterrupted service. Alex’s demo showed how Istio’s traffic management capabilities, such as routing rules and mirroring, facilitate these migrations by directing traffic to specific versions without user impact.
Leveraging Istio for Traffic Management
Istio, a service mesh, plays a pivotal role in Alex’s strategy. By abstracting cross-cutting concerns like service discovery, circuit breaking, and security, Istio simplifies zero downtime deployments. Alex showcased how Istio’s sidecar containers handle traffic routing, enabling techniques like traffic mirroring for dark launches. In a dark launch, requests are sent to both old and new service versions, but only the old version’s response is returned to users, allowing teams to test new versions in production without risk.
Istio also supports chaos engineering, simulating delays or timeouts to test resilience. Alex cautioned, however, that such practices require careful communication to avoid unexpected disruptions, as illustrated by anecdotes of misaligned testing efforts. By integrating Istio with Spring Boot, developers can achieve robust, scalable deployments with minimal overhead.
Handling Stateful Services
Stateful services, particularly those with databases, require special attention. Alex addressed the challenge of maintaining ephemeral state, like shopping carts, using in-memory data grids. For persistent state, he recommended strategies like synthetic transactions or throwaway database clusters to handle mirrored traffic during testing. These approaches prevent unintended database writes, ensuring data integrity during migrations.
In his demo, Alex applied these principles to a movie store application, showing how a shopping cart persisted across blue-green deployments. By using Redis to replicate state across a cluster, he ensured users retained their cart contents, even as services switched versions. This practical example underscored the importance of aligning infrastructure with business needs.
Lessons for Modern DevOps
Alex’s presentation offers a roadmap for achieving zero downtime in microservices. By combining advanced deployment techniques, service meshes, and careful state management, developers can deliver reliable, user-focused applications. His emphasis on tools like Istio and Redis, coupled with a disciplined migration process, provides a blueprint for tackling real-world challenges. For teams like those at Red Hat, these strategies enable faster, safer releases, aligning technical excellence with business continuity.