Posts Tagged ‘RAG’
[DevoxxFR2026] Measuring the Unmeasurable: Evaluating Generative AI Systems
Lecturer
Erin Pacquetet is an expert in AI evaluation and product development at SCIAM, a Paris-based consulting firm. With a background in linguistics and extensive experience guiding enterprises through the complexities of deploying generative AI applications, she specializes in bridging technical implementation with business requirements and robust quality assurance.
Abstract
Generative AI systems promise transformative capabilities but present unique evaluation challenges due to their creative and unpredictable nature. Erin Pacquetet addresses this paradox by outlining comprehensive strategies for assessing systems that blend linguistic fluidity with strict factual accuracy. Using a Retrieval-Augmented Generation (RAG) chatbot as a running case study, the presentation examines limitations of traditional metrics, the role of LLM-as-a-judge approaches alongside their inherent biases, the necessity of human evaluation, and continuous monitoring to detect drift. Attendees gain practical frameworks for building reproducible evaluation pipelines that balance innovation with reliability in production environments.
The Fundamental Challenge of Evaluating Generative Systems
Generative AI introduces a core tension between creativity and control. Organizations adopt large language models precisely because they handle diverse, uncontrolled inputs and produce personalized outputs. Yet this very strength complicates evaluation. Traditional deterministic testing works for rule-based systems but falls short when outputs vary naturally while needing to remain accurate, relevant, and safe.
In the case study of an insurance company’s customer-facing RAG chatbot, the system must answer questions about policies while adhering to brand tone, regulatory constraints, and response length limits. A single question like “Is home insurance mandatory for tenants in France?” could yield multiple valid responses of varying quality. Evaluation must therefore move beyond binary correctness to nuanced assessment across multiple dimensions.
Effective evaluation pipelines transform qualitative judgments into quantitative, scalable measurements. This requires simulating realistic inputs, generating outputs, and assessing them against well-defined criteria. The process must cover ideal scenarios, expected real-world usage, and adversarial cases to ensure robustness before production deployment.
Simulating Inputs: Ideal, Realistic, and Adversarial Scenarios
The foundation of any evaluation lies in a carefully constructed dataset representing the full spectrum of potential interactions. For the insurance chatbot, inputs fall into three categories.
Ideal inputs are perfectly formed questions with clear intent and complete context, such as grammatically correct inquiries directly related to covered products. These establish baseline performance and set high acceptance thresholds.
Realistic inputs mirror actual user behavior: keyword-based queries, vague phrasing, oral-style language, partial context, or minor errors. Testing these ensures the system handles the messy reality of production traffic rather than sanitized examples.
Adversarial inputs probe vulnerabilities: prompt injections, attempts to elicit harmful content, off-topic questions, or malicious efforts to bypass safeguards. These reveal security weaknesses and edge cases that could damage reputation or expose risks.
Creating this dataset demands collaboration between technical teams and domain experts. Business stakeholders define what constitutes success for each category, translating abstract requirements into concrete examples. This exercise often reveals inconsistencies in initial specifications, forcing clarification before development advances.
The resulting evaluation dataset serves as both a benchmark and a living artifact. It evolves with the product, incorporating new failure modes discovered in production and expanding coverage as usage patterns emerge.
Generating and Assessing Outputs: Metrics and Human Judgment
Once inputs are prepared, the system generates outputs for evaluation. Assessment occurs along two primary axes: output quality and operational performance.
Output quality encompasses factual accuracy, relevance to the query, completeness of information, and safety. For the RAG chatbot, responses must draw correctly from policy documents, address the specific question asked, provide sufficient detail without excess length, and maintain an appropriate empathetic tone.
Traditional metrics prove insufficient. String matching fails to capture semantic equivalence across varied phrasings. Semantic similarity measures can overlook critical omissions or subtle inaccuracies. Probabilistic approaches, particularly LLM-as-a-judge, offer greater flexibility by leveraging models to analyze outputs against detailed criteria.
A well-crafted judge prompt might instruct the model to identify contradictions or omissions between a generated response and a reference answer, returning a binary judgment. This constrains the evaluation task sufficiently to reduce variance while maintaining nuance. Multiple specialized judges can target different aspects: one for factual consistency, another for tone alignment, and a third for regulatory compliance.
Human evaluation remains essential for validation. Domain experts review samples to calibrate automated metrics, ensuring alignment between machine judgments and business expectations. This human-in-the-loop process establishes confidence thresholds for each metric.
Operational metrics complement quality assessment. Response latency, cost per inference, and system stability must meet production requirements. A perfectly accurate but slow response fails as a product. Monitoring these dimensions alongside quality creates a holistic view of system readiness.
Building and Maintaining Evaluation Pipelines
A complete pipeline integrates input simulation, output generation, and multi-faceted assessment into an automated workflow. Teams execute evaluations frequently: after prompt modifications during development, before major releases, and continuously in production to detect regression or drift.
The evaluation dataset evolves as the central reference point. Production logs reveal new query patterns or failure modes, which teams incorporate to strengthen coverage. Regular human review sessions ensure metrics remain aligned with changing business needs and user expectations.
For the insurance chatbot, this meant balancing completeness against brevity, factual precision against approachable language, and safety against helpfulness. The dataset captured these trade-offs explicitly, allowing systematic optimization rather than guesswork.
Challenges persist. Judge models can inherit biases or exhibit inconsistency. Human evaluators introduce subjectivity. Thresholds require careful tuning to avoid both false confidence and excessive caution. Success demands iterative refinement and cross-functional collaboration.
From Evaluation to Production Confidence
Robust evaluation bridges the gap between promising prototypes and reliable production systems. By systematically addressing the inherent variability of generative outputs, teams build the confidence necessary for deployment.
The insurance chatbot case demonstrates that evaluation is not merely technical validation but a strategic discipline. It forces clarification of requirements, surfaces hidden assumptions, and creates shared understanding across technical and business stakeholders.
As generative AI proliferates, organizations that master evaluation gain competitive advantage. They deploy innovative capabilities with appropriate safeguards, iterating rapidly while maintaining quality. The discipline transforms the “unmeasurable” into something manageable, turning potential risk into sustainable value.
Links:
[MiamiJUG] Retrieval-Augmented Generation: Building Deterministic AI for Production
Lecturer
Frank Greco is a Java Champion, enterprise architect, and senior consultant specializing in Artificial Intelligence and Cloud computing. He is the founder and Chairman of NYJavaSIG and a co-author of JSR #381 “VisRec,” the Java API for visual recognition. Frank is a recognized educator and technical leader who has presented at major global conferences including JavaOne, DevNexus, and Devoxx.
Abstract
This article provides an analytical framework for integrating Large Language Models (LLMs) into production Java environments using Retrieval-Augmented Generation (RAG). By moving beyond simple chat interfaces to programmatic API access, developers can build AI systems that are grounded in verified enterprise data. The analysis explores prompt engineering methodologies—such as Few-Shot and Chain of Thought (CoT)—and the architectural role of vector databases in mitigating model hallucinations while ensuring data security and version control.
Methodologies in Prompt Engineering
Prompting is the primary mechanism for steering the behavior of a neural network. Unlike traditional programming, prompting is probabilistic rather than deterministic. Frank identifies several advanced techniques to improve model reliability:
- Zero-Shot and Few-Shot Learning: Few-shot prompting provides the model with specific examples of the desired input-output pattern, significantly improving the accuracy of complex tasks.
- Chain of Thought (CoT): This instructs the model to “think step-by-step,” detailing its reasoning process before providing a final answer. This methodology is critical for reducing logical errors.
- Persona Identification: Assigning a specific role to the model (e.g., “Act as a Java security expert”) helps contextualize the response and refine the output tone.
Architectural Implementation: Retrieval-Augmented Generation (RAG)
To overcome the limitations of an LLM’s static training data, enterprises utilize RAG to ground the model in real-time, private data. In a RAG architecture, a user query is first used to search a knowledge base—typically a Vector Database—for relevant documents. This retrieved context is then injected into the prompt, allowing the LLM to generate an answer based on specific facts rather than general probabilities.
This approach offers several production-grade benefits:
- Reduced Hallucinations: By providing the model with the necessary facts, the likelihood of it “making up” information is significantly decreased.
- Data Security: RAG allows models to use private company information without that data being used to train the underlying public model.
- Traceability: Responses can be cited back to specific source documents found in the vector database.
Production Challenges and Ethical Considerations
Implementing AI at scale introduces significant engineering overhead. Developers must manage Prompt Versioning to ensure consistent behavior across deployments and navigate the legal implications of AI-generated content. Furthermore, because these are probabilistic systems, Frank warns that if a wrong answer poses a high risk to the business, generative AI may not be the appropriate solution. Engineers must balance the productivity gains of AI with the need for rigorous safety guardrails and human-in-the-loop verification.
Links:
[PyDataGlobal2025] What’s Next in AI for Data and Data Management
Lecturer
Lisa Amini is a Distinguished Engineer at IBM and Director of Data & AI Platforms Research, where she also leads IBM’s AI Horizons Network. Her career at IBM Research spans more than two decades and includes foundational work on stream processing systems that became the InfoSphere Streams product, leadership of the IBM Research laboratory in Ireland, and earlier roles directing knowledge and reasoning research. She has guided interdisciplinary efforts across cloud computing, artificial intelligence, and quantum computing, always with an emphasis on technologies that can be deployed at enterprise scale.
Abstract
Recent advances in large language models have catalyzed a wave of AI-assisted tools for data management and operations, ranging from code-generation assistants for data-flow pipelines to retrieval-augmented generation systems and increasingly autonomous data agents. This keynote examines the rapid evolution of generative and agentic capabilities, situates them within the broader data-management stack, and explores both near-term practical applications and longer-horizon research directions. Particular attention is given to the shift from human-operated systems augmented by copilots toward semi-autonomous stacks in which agents design, optimize, remediate, and continuously evaluate data products. The discussion balances technical opportunity with the enduring requirements of price-performance, open-source interoperability, and hybrid data architectures.
The Accelerating Capability Curve and the Emergence of Agency
The pace at which machine-learning benchmarks reach human-level performance has changed dramatically. Tasks that once required decades of incremental progress—handwriting recognition, for example—now reach parity within a few years or even months. Reading comprehension and predictive reasoning benchmarks follow similarly steep trajectories. While these evaluations remain narrow and do not constitute artificial general intelligence, they illustrate an unprecedented rate of improvement. Simultaneously, the cost per inference continues to fall even as model size and training compute grow, a trend driven by better systems design and algorithmic efficiency.
Within this landscape the progression from predictive models to generative models to conversational systems and finally to agents marks a qualitative shift. Agents do not merely answer questions; they dynamically control application flow, make decisions, take actions, and attempt self-correction. In the data domain this agency opens the possibility of systems that no longer wait for humans to formulate every query or repair every broken pipeline. Instead, agents can probe schema, resolve ambiguity, hypothesize data products, evaluate their own output, and iterate.
Transforming the Data Landscape and the Complementary Task Stack
Unstructured data has long existed, yet only recently has it assumed central importance. Machines can now reason over images, generate multimodal content, and extract structured signals from free text at scale. Classical database, warehouse, and lakehouse architectures, optimized primarily for structured tables, must therefore accommodate new access patterns. Retrieval-augmented generation pipelines replace static queries with dynamic retrieval-plus-generation cycles. User interaction moves from fixed application-generated SQL toward speculative, multi-step agent dialogues that probe metadata, formulate candidate queries, and refine them in light of intermediate results.
A useful conceptual inversion is to view the traditional storage–compute–query stack alongside a complementary human-task stack: infrastructure design, workload optimization, data discovery, enrichment, flow creation, remediation, governance, and insight generation. Each of these human activities constitutes fertile ground for agentic automation. Early systems already demonstrate learnable components inside query optimizers and routers; more ambitious research explores whether agents can search the design space of kernel-level software itself.
From Automation to Autonomy: Data Products and Continuous Evaluation
The practical goal is not merely to accelerate individual steps but to move entire workflows from human-operated to human-supervised. Consider the request to stand up a data stack and associated data products for a new application—robo-trading, for instance, that must combine public market data with sentiment signals and support periodic rebalancing. A multi-agent system can be tasked with discovering relevant sources, hypothesizing an ideal schema, populating that schema from heterogeneous tables and documents, extracting structured fields from natural-language text, and packaging the result as a governed data product.
Critical to autonomy is the ability to evaluate quality without constant human intervention. One effective strategy generates natural-language questions that a domain expert would plausibly ask of the intended data product, translates those questions into executable queries, and then monitors coverage metrics (tables and columns touched), topic coverage, query complexity, and latency. Agents iterate—adding sources, refining transformations, simplifying views—until the metrics stabilize within acceptable bounds or progress plateaus and human guidance is required. The same loop can later serve as continuous monitoring: questions that once succeeded can be re-executed to detect drift or regression.
Similar patterns apply to operational remediation. When a data-flow pipeline fails, agents can examine logs, generate natural-language root-cause hypotheses, propose script repairs, and, under appropriate guardrails, test those repairs in a sandbox before presenting them for approval. Across the spectrum of use, build, and optimize activities, the user’s role gradually shifts from operator to approver or observer.
Enduring Constraints and the Research Horizon
Price-performance remains non-negotiable; open-source components continue to enable rapid composition of storage formats, query engines, and table formats; hybrid architectures that span on-premises, cloud, and edge locations persist. Benchmarks, data contracts, open lineage standards, and carefully scoped open-weight models supply the interfaces and evaluation harnesses that allow agents to interoperate safely. Research prototypes already explore operator libraries that let developers request high-level transformations while large language models synthesize the concrete implementations behind the scenes.
The path forward is incremental. Fully autonomous data stacks will not appear overnight. Yet the combination of generative models, agent frameworks, and rigorous evaluation loops is already moving concrete workloads—data-product curation, flow repair, insight generation—along the continuum from assistance toward autonomy. The opportunity for data scientists and engineers is to shape the metrics, tools, and governance practices that will keep these systems both powerful and trustworthy.
Links:
[VoxxedDaysBucharest2026] Building a Sarcastic, Agentic Pair Programmer: Alexander Chatzizacharias on Crafting Playful LLM Workflows
Lecturer
Alexander Chatzizacharias is a software engineer at JDriven, a specialized consultancy in the Netherlands focused on JVM technologies and modern software development practices. With a unique background blending Dutch and Greek influences and a keen interest in game studies, Alexander brings creativity and playful thinking to technical challenges. He frequently speaks on topics including Java, Spring Boot, AI applications, and innovative development workflows.
Abstract
As mainstream AI coding assistants converge toward similar polished but somewhat generic experiences, Alexander Chatzizacharias demonstrates how to build a highly personalized, characterful AI pair programmer named “Pip.” Inspired by interactions with a sarcastic colleague named Ricardo, Pip incorporates personality through vectorized Slack history, utilizes Spring Boot and Kotlin, runs entirely locally with Qwen models via Ollama, and employs sophisticated workflows, multi-vector RAG, and the Model Context Protocol (MCP) to create delightful and productive assistance while addressing challenges like non-determinism and model drift.
The Homogenization of AI Assistants and the Quest for Personality
Alexander observes that leading AI coding tools have converged on remarkably similar chat-based interfaces and interaction patterns, largely influenced by OpenAI’s design choices. While incremental improvements continue, the overall experience feels increasingly uniform. This observation inspired the creation of Pip — an intentionally quirky, sarcastic AI pair programmer that injects personality drawn from real colleague interactions.
By processing Slack conversation history into vector embeddings stored in Qdrant, Pip can retrieve and emulate Ricardo’s characteristic sarcastic tone, witty retorts, and playful threats (such as threatening to delete poorly written code). This transforms the assistant from a neutral tool into a more engaging, human-like collaborator that questions unclear requirements, offers humorous feedback, and makes the development process more enjoyable.
Technical Architecture: Workflows, Agents, and Local Execution
Pip is implemented as a Spring Boot application written in Kotlin, with an IntelliJ IDEA plugin providing the frontend interface. Everything runs locally to maintain privacy and control: Qwen 3.5 models served through Ollama handle the language tasks.
Rather than pursuing fully autonomous agents, Alexander favors structured workflows that provide greater determinism and reliability — attributes particularly valued in enterprise environments. A categorization agent, functioning as an LLM-as-Judge, routes incoming queries to appropriate specialized handlers. Each handler uses carefully crafted system prompts derived from Slack history to consistently embody the desired personality traits.
The architecture incorporates multiple specialized agents for response generation, sophisticated RAG pipelines leveraging both dense and sparse vector representations with ColBERT reranking for improved retrieval quality, and integration with the Model Context Protocol (MCP) for tool usage such as playing music or generating memes when appropriate.
RAG, Tools, and the Challenges of Non-Determinism
Retrieval-Augmented Generation forms a cornerstone of Pip’s capabilities, dynamically pulling relevant context to overcome the inherent token limitations of even advanced models. Multi-vector search strategies combine semantic understanding with keyword precision for more reliable information retrieval from project documentation, codebases, and conversation history.
Tool integration via MCP enables rich interactions but introduces additional complexity due to the non-deterministic nature of local models. Alexander discusses practical challenges including prompt sensitivity to model updates (“model locking” strategies), the art of prompt engineering which he likens to “vibe checking,” and the necessity of implementing guardrails to maintain appropriate behavior boundaries.
Implications for Future AI Development
Alexander encourages attendees to experiment with building personalized, domain-specific AI assistants using accessible open-source tools. While acknowledging the increasing commercialization of AI, he emphasizes the current window of opportunity for creative, playful implementations that enhance both productivity and developer satisfaction.
Pip serves as an inspiring example of how thoughtful combination of RAG techniques, vector databases, workflow orchestration, and personality injection can create AI tools that feel genuinely collaborative rather than merely functional.
Links:
[AWSReInvent2025] High-Performance Storage Architectures for AI/ML, Analytics, and HPC Workloads
Lecturer
Aditi is a Senior Product Manager for Amazon FSx at Amazon Web Services (AWS). With years of experience working directly with customers on high-performance workloads, she focuses on pushing the technical boundaries of what is possible with cloud storage to meet the demands of modern compute-intensive applications.
Abstract
This article examines the critical role of high-performance storage in supporting modern AI/ML, analytics, and High-Performance Computing (HPC) workloads. As organizations scale their compute resources—incorporating hundreds or thousands of CPU and GPU cores—storage often becomes the primary bottleneck, preventing linear performance scaling. We explore the technical architectures of Amazon FSx and Amazon S3, focusing on how these services address the needs of both “lift-and-shift” file-based applications and “cloud-native” S3-based data lakes. By analyzing customer use cases in genomics, media rendering, and large language model (LLM) training, we detail the methodologies for achieving peak performance at scale.
The Storage Bottleneck in Compute-Intensive Workloads
Modern high-performance workloads are characterized by their extreme reliance on massive datasets and high-core-count compute clusters. In an ideal cloud environment, adding more compute resources should lead to a proportional increase in work completed—a concept known as linear scaling. However, traditional storage solutions often fail to keep pace with the throughput demands of these clusters, leading to a performance plateau.
When storage becomes the bottleneck, compute instances sit underutilized as they compete for access to the same data store. This is particularly detrimental given that 90% to 95% of the expenditure for these workloads is typically allocated to compute resources. Consequently, an inefficient storage layer not only extends the time to insight but also significantly increases the total cost of ownership (TCO). To avoid this, storage must be architected to scale linearly alongside compute.
Navigating the Path to the Cloud: File Systems vs. Object Storage
Organizations generally approach high-performance storage on AWS from two distinct backgrounds: those with long-standing on-premises file-based workflows and those who have built native cloud applications around object storage.
The Persistence of File-Based Architectures
Despite the rise of object storage, file systems remain the preferred interface for many researchers and developers due to three primary factors: Familiar Interface: The intuitive nature of files and directories simplifies complex data management for data scientists and developers.
* Granular Permissions: File systems provide robust POSIX permissions, allowing for fine-grained control over which users can read, write, or execute specific files.
* Consistent Data Access:* For workloads where multiple users or compute nodes access the same data simultaneously, the strong consistency of file systems ensures that all parties see the most recent data updates.
Amazon FSx for High-Performance File Access
Amazon FSx addresses these needs by providing fully managed file systems that offer the performance of local storage with the scalability of the cloud. For “lift-and-shift” scenarios, FSx allows organizations to move their existing HPC and AI/ML pipelines to AWS without refactoring their applications.
Accelerating Generative AI and ML Workloads
The emergence of generative AI has placed a renewed emphasis on data strategy. Whether an organization is building a model from scratch or fine-tuning a foundational model, the quality and accessibility of its proprietary data are the primary differentiators.
Retrieval Augmented Generation (RAG)
To move beyond generic AI responses and reduce hallucinations, many organizations are implementing Retrieval Augmented Generation (RAG). RAG allows foundational models to access evolving, large-scale data lakes without requiring the data to be manually loaded into a prompt.
The RAG methodology involves:
1. Vectorization: Converting organizational data into vectors—numeric representations that capture semantic meaning.
2. Semantic Search: Using spatial similarity to compare a query vector against the data lake’s vectors to find the most relevant information.
3. Augmentation: Feeding the retrieved context back into the model to generate a more accurate and business-specific response.
Ingestion and Data Strategy with Amazon S3
Amazon S3 serves as the foundational data lake for these AI workflows due to its cost-effectiveness and virtually unlimited scalability. Organizations typically utilize two ingestion patterns:
* Batch Ingestion: Suitable for static or infrequently changing data such as historical records and product catalogs.
* Real-Time Ingestion: Essential for agentic workflows where AI models must respond to the latest available information.
Modernizing Self-Managed Databases with Amazon FSx
While fully managed services like Amazon RDS are popular, certain business and technical requirements drive organizations toward self-managed database architectures on AWS.
Drivers for Self-Managed Databases
Organizations choose to self-manage databases like Oracle, SQL Server, or SAP HANA for several reasons:
* Granular Control: The ability to choose specific versions of the database engine and the underlying operating system.
* Custom Protection Policies: Implementing specific backup intervals and recovery procedures that may not be available in managed services.
* High Resilience: Scaling databases across multiple Availability Zones or regions with custom failover configurations.
Optimization through Storage Features
A common oversight in database deployment is the potential for the storage layer to add significant value beyond simple data persistence. Amazon FSx file systems (including FSx for NetApp ONTAP, OpenZFS, and Windows File Server) enable features like:
* Snapshots and Cloning: Facilitating rapid testing and database upgrades by creating near-instantaneous copies of production environments.
* Performance Tuning: Choosing the right FSx service can significantly optimize the TCO and performance of database environments, particularly for high-transaction workloads.
Conclusion
As compute power continues to expand, the storage layer must evolve from a passive repository into a high-performance engine. By leveraging Amazon FSx and S3, organizations can eliminate storage bottlenecks, enabling their most demanding AI, HPC, and database workloads to scale linearly and cost-effectively in the cloud.
Links:
- Amazon FSx Product Page
- Amazon S3 Product Page
- AWS re:Invent 2025 – High-performance storage for AI/ML, analytics, and HPC workloads (STG336)
- AWS re:Invent 2025 – Accelerate gen AI and ML workloads with AWS storage (STG201)
- AWS re:Invent 2025 – Improve self-managed database performance and agility with Amazon FSx (STG337)
[PyDataGlobal2025] Where Have All the Metrics Gone? Evaluating Generative Systems in a Multi-Dimensional Era of Error
Lecturer
Dr. Rebecca Bilbro is a data scientist and co-creator of the Yellowbrick library, an open-source diagnostic visualization toolkit that extends the scikit-learn and Matplotlib APIs. She has taught machine learning at Georgetown University and continues to work at the intersection of model interpretation, visual analytics, and the practical evaluation of contemporary AI systems. Her earlier work focused on making the model-selection process more transparent through visual diagnostics; the present discussion extends that concern into the generative era.
Abstract
Classical machine-learning metrics—F1 score, mean squared error, area under the ROC curve—once supplied a comforting scalar signal of progress and a shared vocabulary for model comparison. In the generative era those metrics have largely receded from daily practice. Large language models can fail simultaneously along semantic, stylistic, structural, behavioral, and temporal dimensions, rendering single-axis notions of accuracy ill-defined. This article enumerates recurring failure modes observed in production generative systems, argues for an experimental rather than assumptive framing of new projects, advocates systematic task decomposition as a prerequisite for meaningful measurement, and illustrates how lightweight, purpose-built metrics can restore visibility, accountability, and iterative improvement.
The Retreat of Scalar Certainty and the New Landscape of Error
Traditional supervised learning treated error as a one-dimensional discrepancy between a model’s prediction and a ground-truth label. Diagnostic visualizations such as residual plots for regression and confusion matrices for classification made that discrepancy tangible and actionable, guiding feature engineering, model selection, and hyperparameter search. The resulting workflow was comparatively linear: obtain data, explore, select and tune models, optimize a scalar metric, serialize the artifact, and await the next batch of data.
Generative models invert many of these assumptions. Practitioners increasingly consume foundation models rather than train them from scratch, and therefore inhabit the application side of the former training–deployment boundary. On that side, error is higher-dimensional and less crisply defined. An output may be factually incorrect, stylistically inappropriate, structurally malformed, behaviorally unbounded, or temporally outdated—frequently several of these at once. The feedback that arrives is no longer a number but a qualitative complaint: the system hallucinated, the tone is wrong, the answer is inconsistent, the result is too vague, or the model performed an action it should never have attempted. Even the word “accuracy” has become ambiguous; it may refer to mathematical correctness, to perceived alignment with user intent, or simply to the subjective impression that the output “feels right.”
A Working Taxonomy of Generative Failure Modes
Several distinct modes of failure recur with sufficient regularity to merit explicit names and separate measurement strategies. Domain failure occurs when an output employs correct jargon and plausible structure yet contains subtle inaccuracies detectable only by a genuine subject-matter expert. The classic illustration is an authoritative but incorrect set of instructions for a skilled manual task; only embodied expertise reveals the error. Form failure arises when content is otherwise acceptable but the required schema—JSON, HTML, a particular report template—is violated, breaking downstream automated systems. Mode collapse, familiar from the literature on generative adversarial networks, appears when synthetic data or requested stylistic variants exhibit insufficient diversity, collapsing into high-probability phrasings and structures. Consistency failure manifests as contradictory answers to essentially identical prompts issued on different occasions or with minor rephrasing. Boundary failure describes the model performing tasks outside its intended scope, such as a narrowly purposed customer-service agent solving advanced mathematical problems. Temporal failure reflects the model’s blindness to the passage of time, leading it to recommend deprecated APIs, outdated function signatures, or facts that have been superseded.
Collectively these modes demonstrate that a single scalar metric cannot capture the relevant notions of quality. Each mode points toward a different remediation strategy—retrieval grounding, schema enforcement, diversity sampling, consistency regularization, capability gating, or temporal knowledge injection—and therefore demands its own measurement instrument.
Experimental Framing, Task Decomposition, and the Rejection of the Mega-Prompt
Projects that begin with the declarative sentence “We are building an agent that can \ldots” implicitly treat the desired capability as already achieved and thereby skip the experimental design necessary to surface failure. Reframing the same ambition as “We want to test whether an agent can \ldots” converts the effort into a set of measurable hypotheses and observable failure modes. Because different applications are vulnerable to different subsets of the taxonomy, practitioners are advised to select two to four high-stakes modes at the outset rather than attempt exhaustive coverage. A system that emits structured records will prioritize form failure; a research assistant will prioritize hallucinated citations; a regulated healthcare application will prioritize boundary violations and leakage of protected information.
Once the relevant modes are chosen, the monolithic “mega-prompt” that attempts to solve an entire workflow inside a single generation becomes counterproductive. Any failure is global and unlocalizable; root-cause analysis is nearly impossible. Systematic task decomposition restores visibility. Retrieval, fact-checking, synthesis, formatting, and validation become separate stages, each with bounded inputs, explicit success criteria, and its own characteristic failure signature. The resulting pipeline resembles classical extract–transform–load systems or production machine-learning workflows and permits the same style of stepwise debugging and incremental improvement.
Constructing Intentional Measurement Strategies
Measurement itself must be reinvented for the generative setting. Binary, machine-checkable criteria—valid HTML, schema compliance enforced at generation time by libraries such as Pydantic or Outlines—can eliminate entire classes of form failure before they reach downstream consumers. Where domain expertise remains indispensable, simple Likert-scale ratings collected from subject-matter experts convert qualitative impressions into trackable numeric signals that can be monitored over successive iterations. Consistency can be quantified by generating multiple outputs from an identical prompt and computing average pairwise similarity in embedding space, yielding an interpretable score between zero and one. None of these metrics is universal or theoretically privileged; each is an intentional response to a previously identified failure mode and is valuable precisely because it is tailored.
The open-source ecosystem already supplies many of the required building blocks. Parameterized testing frameworks allow systematic variation of prompts and automatic checking of outputs. Structural validation libraries constrain generation. Tracing and logging harnesses provide the observability needed to locate failures inside a decomposed pipeline. The remaining work is largely cultural: the willingness to define operational notions of “good” rather than to inherit a default scalar from an earlier paradigm, and the humility to treat every new agent as an experiment whose failure modes must be anticipated, isolated, and measured.
Links:
[SpringIO2025] Real-World AI Patterns with Spring AI and Vaadin by Marcus Hellberg / Thomas Vitale
Lecturer
Marcus Hellberg is the Vice President of AI Research at Vaadin, a company specializing in tools for Java developers to build web applications. As a Java Champion with nearly 20 years of experience in Java and web development, he focuses on integrating AI capabilities into Java ecosystems. Thomas Vitale is a software engineer at Systematic, a Danish software company, with expertise in cloud-native solutions, Java, and AI. He is the author of “Cloud Native Spring in Action” and an upcoming book on developer experience on Kubernetes, and serves as a CNCF Ambassador.
- Marcus Hellberg on LinkedIn
- Marcus Hellberg on GitHub
- Thomas Vitale on LinkedIn
- Thomas Vitale on GitHub
Abstract
This article examines practical patterns for incorporating artificial intelligence into Java applications using Spring AI and Vaadin, transitioning from experimental to production-ready implementations. It analyzes techniques for memory management, guardrails, multimodality, retrieval-augmented generation, tool calling, and agents, with implications for security, user experience, and system integration. Insights emphasize robust, observable AI workflows in on-premises or cloud environments.
Memory Management and Streaming in AI Interactions
Integrating large language models (LLMs) into applications requires addressing their stateless nature, where each interaction lacks inherent context from prior exchanges. Spring AI provides advisors—interceptor-like mechanisms—to augment prompts with conversation history, enabling short-term memory. For instance, a MessageChatMemoryAdvisor retains the last N messages, ensuring continuity without manual tracking.
This pattern enhances user interactions in chat-based interfaces, built here with Vaadin’s component model for server-side Java UIs. A vertical layout hosts message lists and inputs, injecting a ChatClientBuilder to construct clients with advisors. Basic interactions involve prompting the model and appending responses, but for realism, streaming via reactive fluxes improves responsiveness, subscribing to token streams and updating UI progressively.
Code illustration:
ChatClient chatClient = builder.build();
messageInput.addSubmitListener(submitEvent -> {
String message = submitEvent.getMessage();
MessageItem userItem = messageList.addMessage("You", message);
chatClient.stream(new Prompt(message))
.subscribe(response -> {
userItem.append(response.getResult().getOutput().getContent());
});
});
Streaming suits verbose responses, reducing perceived latency, while observability integrations (e.g., OpenTelemetry) trace interactions for debugging nondeterministic behaviors.
Guardrails for Security and Validation
AI workflows must mitigate risks like sensitive data leaks or invalid outputs. Input guardrails intercept prompts, using on-premises models to check for compliance with policies, blocking unauthorized queries (e.g., personal information). Output guardrails validate responses, reprompting for corrections if deserialization fails.
Advisors enable this: a default advisor with a local chat model filters inputs/outputs. For example, querying an address might be blocked if flagged, preventing cloud exposure. This ensures determinism in structured outputs, converting unstructured text to Java objects via JSON instructions.
Implications include privacy preservation in regulated sectors and integration with Spring Security for role-based tool access.
Multimodality and Retrieval-Augmented Generation
LLMs extend beyond text through multimodality, processing images, audio, or videos. Spring AI’s entity methods augment prompts for structured extraction, e.g., parsing attendee details from images into tables for programmatic use.
Retrieval-augmented generation (RAG) combats hallucinations by embedding external data as vectors in stores like PostgreSQL. A RetrievalAugmentationAdvisor retrieves relevant documents via similarity search, augmenting prompts. Customizations allow empty contexts for fallback to model knowledge.
Example:
VectorStore vectorStore = // PostgreSQL vector store
DocumentRetriever retriever = new VectorStoreDocumentRetriever(vectorStore);
RetrievalAugmentationAdvisor advisor = RetrievalAugmentationAdvisor.builder()
.documentRetriever(retriever)
.queryAugmentor(QueryAugmentor.contextual().allowEmptyContext(true))
.build();
This pattern grounds responses in proprietary data, with thresholds controlling retrieval scope.
Tool Calling, Agents, and Dynamic Integrations
Tool calling empowers LLMs as agents, invoking external functions for tasks like database queries. Annotations describe tools, passed to clients for dynamic selection. For products, a service might expose query/update methods:
@Tool(description = "Fetch products from database")
public List<Product> getProducts(@P(description = "Category filter") String category) {
// Database query
}
Agents orchestrate tools, potentially via Model Context Protocol for external services. Demonstrations include theme generation from screenshots, editing CSS via file system tools, highlighting nondeterminism and the need for safeguards.
In conclusion, these patterns enable production AI, emphasizing modularity, security, and observability for robust Java applications.
Links:
[DotJs2025] Prompting is the New Scripting: Meet GenAIScript
As generative paradigms proliferate, scripting’s syntax strains under AI’s amorphous allure—prompts as prosaic prose, yet perilous in precision. Yohan Lasorsa, Microsoft’s principal developer advocate and Angular GDE, unveiled GenAIScript at dotJS 2025, a JS-inflected idiom abstracting LLM labyrinths into lucid loops. With 15 years traversing IoT’s interstices to cloud’s canopies, Yohan likened this lexicon to jQuery’s jubilee: DOM’s discord domesticated, now GenAI’s gyrations gentled for mortal makers.
Yohan’s yarn recalled jQuery’s jihad: browser balkanization banished, events etherealized—20 years on, GenAI’s gale mirrors, models multiplying, APIs anarchic. GenAIScript’s grace: JS carapace cloaking complexities—await ai.chat('prompt') birthing banter, ai.forEach(items, 'summarize') distilling dossiers. Demos danced: file foragers (fs.readFile), prompt pipelines (ai.pipe(model).chat(query)), even AST adventurers refactoring Angular artifacts—CLI’s churn supplanted by semantic sorcery.
This superstructure spans: agents’ autonomy (ai.agent({tools})), RAG’s retrieval (ai.retrieve({query, store})), even vision’s vignettes (ai.vision(image)). Yohan’s yield: ergonomics eclipsing exhaustion—built-ins for Bedrock, Ollama; extensibility via plugins. Caveat’s cadence: tool for tinkering, not titanic tomes—yet frameworks’ fledglings may flock hither.
GenAIScript’s gospel: prompting’s poetry, scripted sans strife—democratizing discernment in AI’s ascent.
jQuery’s Echo in AI’s Era
Yohan juxtaposed jQuery’s quirk-quelling with GenAI’s gale—models’ menagerie, APIs’ anarchy. GenAIScript’s girdle: JS’s jacket jacketting journeys—chat’s cadence, forEach’s finesse.
Patterns’ Parade and Potentials
Agents’ agency, RAG’s recall—pipelines pure, vision’s vista. Yohan’s yarns: Angular migrations mended, Bedrock bridged—plugins’ pliancy promising proliferation.
Links:
[NDCMelbourne2025] How to Work with Generative AI in JavaScript – Phil Nash
Phil Nash, a developer relations engineer at DataStax, delivers a comprehensive guide to leveraging generative AI in JavaScript at NDC Melbourne 2025. His talk demystifies the process of building AI-powered applications, emphasizing that JavaScript developers can harness existing skills to create sophisticated solutions without needing deep machine learning expertise. Through practical examples and insights into tools like Gemini and retrieval-augmented generation (RAG), Phil empowers developers to explore this rapidly evolving field.
Understanding Generative AI Fundamentals
Phil begins by addressing the excitement surrounding generative AI, noting its accessibility since the release of the GPT-3.5 API two years ago. He emphasizes that JavaScript developers are well-positioned to engage with AI due to robust tooling and APIs, despite the field’s Python-centric origins. Using Google’s Gemini model as an example, Phil demonstrates how to generate content with minimal code, highlighting the importance of understanding core concepts like token generation and model behavior.
He explains tokenization, using OpenAI’s byte pair encoding as an example, where text is broken into probabilistic tokens. Parameters like top-k, top-p, and temperature allow developers to control output randomness, with Phil cautioning against overly high settings that produce nonsensical results, humorously illustrated by a chaotic AI-generated story about a gnome.
Enhancing AI with Prompt Engineering
Prompt engineering emerges as a critical skill for refining AI outputs. Phil contrasts zero-shot prompting, which offers minimal context, with techniques like providing examples or system prompts to guide model behavior. For instance, a system prompt defining a “capital city assistant” ensures concise, accurate responses. He also explores chain-of-thought prompting, where instructing the model to think step-by-step improves its ability to solve complex problems, such as a modified river-crossing riddle.
Phil underscores the need for evaluation to ensure prompt reliability, as slight changes can significantly alter outcomes. This structured approach transforms prompt engineering from guesswork into a disciplined practice, enabling developers to tailor AI responses effectively.
Retrieval-Augmented Generation for Contextual Awareness
To address AI models’ limitations, such as outdated or private data, Phil introduces retrieval-augmented generation (RAG). RAG enhances models by integrating external data, like conference talk descriptions, into prompts. He explains how vector embeddings—multidimensional representations of text—enable semantic searches, using cosine similarity to find relevant content. With DataStax’s Astra DB, developers can store and query vectorized data efficiently, as demonstrated in a demo where Phil’s bot retrieves details about NDC Melbourne talks.
This approach allows AI to provide contextually relevant answers, such as identifying AI-related talks or conference events, making it a powerful tool for building intelligent applications.
Streaming Responses and Building Agents
Phil highlights the importance of user experience, noting that AI responses can be slow. Streaming, supported by APIs like Gemini’s generateContentStream, delivers tokens incrementally, improving perceived performance. He demonstrates streaming results to a webpage using JavaScript’s fetch and text decoder streams, showcasing how to create responsive front-end experiences.
The talk culminates with AI agents, which Phil describes as systems that perceive, reason, plan, and act using tools. By defining functions in JSON schema, developers can enable models to perform tasks like arithmetic or fetching web content. A demo bot uses tools to troubleshoot a keyboard issue and query GitHub, illustrating agents’ potential to solve complex problems dynamically.
Conclusion: Empowering JavaScript Developers
Phil concludes by encouraging developers to experiment with generative AI, leveraging tools like Langflow for visual prototyping and exploring browser-based models like Gemini Nano. His talk is a call to action, urging JavaScript developers to build innovative applications by combining AI capabilities with their existing expertise. By mastering prompt engineering, RAG, streaming, and agents, developers can create powerful, user-centric solutions.
Links:
[AWSReInventPartnerSessions2024] Architecting Real-Time Generative AI Applications: A Confluent-AWS-Anthropic Integration Framework
Lecturer
Pascal Vuylsteker serves as Senior Director of Innovation at Confluent, where he pioneers scalable data streaming architectures that underpin enterprise artificial intelligence systems. Mario Rodriguez functions as Senior Partner Solutions Architect at AWS, specializing in generative AI service orchestration across cloud environments. Gavin Doyle leads the Applied AI team at Anthropic, directing development of safe, steerable, and interpretable large language models.
Abstract
This scholarly examination delineates a comprehensive methodology for constructing real-time generative AI applications through the synergistic integration of Confluent’s streaming platform, Amazon Bedrock’s managed foundation model ecosystem, and Anthropic’s Claude models. The analysis elucidates data governance centrality, retrieval-augmented generation (RAG) with continuous contextual synchronization, Flink-mediated inference execution, and vector database orchestration. Through architectural decomposition and configuration exemplars, it demonstrates how these components eliminate data silos, ensure temporal relevance in AI outputs, and enable secure, scalable enterprise innovation.
Governance-Centric Modern Data Architecture
Enterprise competitiveness increasingly hinges upon real-time data streaming capabilities, with seventy-nine percent of IT leaders affirming its strategic necessity. However, persistent barriers—siloed repositories, skill asymmetries, governance complexity, and generative AI’s voracious data requirements—impede realization.
Contemporary data architectures position governance as the foundational core, ensuring security, compliance, and accessibility. Radiating outward are data warehouses, streaming analytics engines, and generative AI applications. This configuration systematically dismantles silos while satisfying instantaneous insight demands.
Confluent operationalizes this vision by providing real-time data integration across ingestion pipelines, data lakes, and batch processing systems. It delivers precisely contextualized information at the moment of need—prerequisite for effective generative AI deployment.
Amazon Bedrock complements this through managed access to foundation models from Anthropic, AI21 Labs, Cohere, Meta, Mistral AI, Stability AI, and Amazon. The service supports experimentation, fine-tuning, continued pre-training, and agent orchestration. Security architecture prohibits customer data incorporation into base models, maintains isolation for customized variants, implements encryption, enforces granular access controls, and complies with HIPAA, GDPR, SOC, ISO, and CSA STAR.
Proprietary data constitutes the primary differentiation vector. Three techniques leverage this advantage: RAG injects external knowledge into prompts; fine-tuning specializes models on domain corpora; continued pre-training expands comprehension using enterprise datasets.
\# Bedrock model customization (conceptual)
modelCustomization:
baseModel: anthropic.claude-3-sonnet
trainingData: s3://enterprise-corpus/
fineTuning:
epochs: 3
learningRate: 0.0001
Real-Time Contextual Injection and Flink Inference Orchestration
Confluent integrates directly with vector databases, ensuring conversational systems operate upon current, relevant information. This transcends mere data transport to deliver AI-actionable context.
Flink Inference enables real-time machine learning via Flink SQL, dramatically simplifying model integration into operational workflows. Configuration defines endpoints, authentication, prompts, and invocation patterns.
The architectural pipeline commences with document publication to Kafka topics. Documents undergo chunking for parallel processing, embedding generation via Bedrock/Anthropic, and indexing into MongoDB Atlas with original chunks. Quick-start templates deploy this workflow, incorporating structured data summarization through Claude for natural language querying.
Chatbot interactions initiate via API/Kafka, generate embeddings, retrieve documents, construct prompts with streaming context, and invoke Claude. Token optimization employs conversation summarization; enhanced vector queries via Claude-generated reformulations yield superior retrieval.
-- Flink model definition
CREATE MODEL claude_haiku WITH (
'connector' = 'anthropic',
'endpoint' = 'https://api.anthropic.com/v1/messages',
'api.key' = 'sk-ant-...',
'model' = 'claude-3-haiku-20240307'
);
-- Real-time inference
INSERT INTO responses
SELECT ML_PREDICT('claude_haiku', enriched_prompt) FROM interactions;
Flink provides cost-effective scaling, automatic elasticity, and native integration with AWS services, Snowflake, Databricks, and MongoDB. Anthropic models remain fully accessible via Bedrock.
Strategic Implications for Enterprise AI
The methodology transforms RAG from static knowledge injection to dynamic reasoning augmentation. Contextual retrieval and in-context learning mitigate hallucinations while enabling domain-specific differentiation.
Organizations achieve decision-making superiority through proprietary data in real-time contexts. Governance scales securely; challenges like data drift necessitate continuous refinement.
Future trajectories include declarative inference and hybrid vector-stream architectures for anticipatory intelligence.