Posts Tagged ‘Guardrails’
[VoxxedDaysAmsterdam2026] Un-Observable AI Is Untrustworthy AI: Building Reliable Systems Through Comprehensive Observability
Lecturer
Annie Freeman is a Developer Advocate at Coralogix, specializing in full-stack observability platforms and the responsible deployment of AI applications. With a background in green software practices and a focus on sustainability in technology, Annie explores how visibility into AI systems can address challenges related to cost, ethics, and operational reliability.
Abstract
The rapid adoption of AI systems, particularly those involving large language models and agentic workflows, introduces significant complexities around trust, resource consumption, and ethical behavior. Traditional monitoring approaches often prove insufficient for these dynamic environments. Annie Freeman examines how observability, implemented through OpenTelemetry, can establish robust systems of trust around AI applications. By analyzing four distinct layers of observability—from development tools to quality monitoring—the discussion highlights practical strategies for instrumenting AI workloads, detecting issues such as hallucinations or policy violations, and implementing real-time guardrails. These insights enable organizations to build AI solutions that are not only performant but also accountable and sustainable.
The Fundamental Challenge: Why Traditional Monitoring Falls Short for AI
AI systems differ fundamentally from conventional software in their non-deterministic nature. The same input can produce varying outputs, agentic loops may execute unpredictable numbers of tool calls, and decision-making processes remain opaque. This unpredictability creates multiple layers of risk: potential harm from inappropriate responses, escalating operational costs from uncontrolled resource usage, and difficulties in capacity planning due to variable inference demands.
Users require consistent and reliable experiences. Company leadership must ensure investments yield clear business value without runaway expenses. Developers, increasingly reliant on AI coding assistants as production dependencies, need confidence in the generated outputs. Traditional metrics focused on uptime or basic performance fail to capture these nuances. Without targeted observability, teams operate with limited visibility into model behavior, making it impossible to verify ethical alignment or optimize resource utilization effectively.
Establishing Foundational Observability: Development and Operational Layers
Observability begins at the development stage, where AI coding tools such as Claude Code or CodeWhisperer generate substantial portions of application logic. These tools emit OpenTelemetry data natively, providing metrics on token usage, cost per session, model selection patterns, and code acceptance rates. Such visibility transforms subjective assessments of tool effectiveness into data-driven insights, enabling teams to optimize developer productivity and identify which models deliver the highest value for specific tasks.
Operational metrics extend this foundation into production environments. Key signals include token consumption trends, model invocation patterns, and response finish reasons. These indicators function analogously to HTTP status codes, revealing whether completions result from natural termination, length limits, or other constraints. High-spending users or unusual patterns, such as excessive retry loops, become immediately apparent. Organizations can then implement targeted optimizations, such as adjusting model sizes for specific use cases or imposing limits on tool call iterations.
The unified nature of OpenTelemetry ensures that AI telemetry integrates seamlessly with existing application monitoring. This avoids data silos and enables comprehensive system analysis. Teams gain the ability to correlate AI behavior with broader application performance, facilitating more informed architectural decisions.
Enhancing Decision Transparency and Real-Time Protection
Decision tracing provides critical context for understanding not just what an AI system produces but why it arrived at particular conclusions. By instrumenting agentic loops with custom spans, teams can capture detailed information about each step: input validation, prompt construction, tool selection, and reasoning chains. This granular visibility transforms black-box operations into auditable processes.
OpenTelemetry’s semantic conventions standardize the collection of this data, ensuring consistency across different AI workloads. Traces reveal the complete journey of a request, from initial user input through multiple reasoning iterations to final output. Such transparency supports debugging, compliance requirements, and continuous improvement efforts.
Quality monitoring introduces an additional safeguard layer. Small language models serve as specialized evaluators, analyzing outputs for hallucinations, toxicity, policy violations, or relevance issues. These evaluators operate with high accuracy due to their focused training, providing rapid feedback without the latency of larger models. When combined with guardrails, this approach enables real-time intervention. Suspicious inputs or outputs can be blocked before reaching users, maintaining system integrity and user trust.
Practical Implementation and Long-Term Benefits
Implementing these observability layers requires intentional design but yields substantial returns. OpenTelemetry’s vendor-neutral approach prevents lock-in while leveraging existing infrastructure investments. Teams can begin with basic instrumentation and progressively add sophistication as needs evolve.
The framework supports multiple stakeholder requirements simultaneously. Users benefit from consistent, safe interactions. Leadership gains visibility into costs and value delivery. Developers receive actionable insights for refining both AI components and their integration with business logic.
As AI adoption accelerates, observability becomes the cornerstone of responsible deployment. Systems built with comprehensive monitoring demonstrate greater reliability, ethical alignment, and operational efficiency. The investment in observability infrastructure pays dividends through reduced incidents, optimized resource usage, and enhanced organizational confidence in AI capabilities.
By treating observability as integral to AI system design rather than an afterthought, teams can move beyond experimental prototypes toward production-grade solutions that earn and maintain user trust.
Links:
[VoxxedDaysBucharest2026] Breaching LLM-Powered Applications: Brian Vermeer on Security and Privacy Challenges in AI Systems
Lecturer
Brian Vermeer is a Staff Developer Advocate at Snyk, where he focuses on developer security, DevSecOps practices, and emerging risks in modern application architectures. A recognized Java Champion and active community leader who co-leads the Netherlands Java User Group (NLJUG), Brian brings extensive experience in application security, secure coding, and helping organizations build more resilient systems. He frequently speaks at international conferences on topics ranging from traditional web vulnerabilities to the novel attack surfaces introduced by artificial intelligence and large language models.
Abstract
As organizations rapidly integrate Large Language Models into production applications, new categories of security and privacy vulnerabilities emerge alongside familiar web application risks. Brian Vermeer provides a compelling, demonstration-heavy exploration of these challenges through a fictional car rental application called “Really Good Rentals.” He demonstrates practical attack vectors such as prompt injection, RAG poisoning, memory manipulation, and tool abuse, while outlining layered mitigation strategies including input/output guardrails, scoped permissions, human-in-the-loop verification, and architectural defenses essential for building trustworthy LLM-powered systems.
The Allure and Inherent Risks of LLMs in Production Applications
Brian begins by drawing a relatable analogy: just as children enthusiastically misuse new toys in unexpected ways, developers often rush to incorporate powerful new technologies like LLMs without fully appreciating the expanded attack surface they create. While LLMs offer remarkable capabilities for natural language processing, code generation, and intelligent automation, they introduce significant risks when granted access to tools, user data, or execution privileges.
He presents a simplified architecture of a typical LLM-powered application, highlighting key components: user prompts, system instructions, conversation memory, retrieval-augmented generation (RAG) pipelines, and tool-calling mechanisms. Because LLMs are fundamentally stateless, the surrounding application bears responsibility for maintaining context, which creates multiple points where malicious actors can influence behavior through carefully crafted inputs.
Context Poisoning Through RAG and Memory Manipulation
A central demonstration revolves around the “Really Good Rentals” application. Brian shows how a seemingly innocuous file upload feature with inadequate path validation allows attackers to perform directory traversal and overwrite critical documents stored in the vector database, such as terms-of-service files. By injecting a modified cancellation policy containing trigger phrases like “vroom vroom,” the attacker can later invoke this policy through normal chat interactions, tricking the LLM into granting unauthorized credits or violating business rules.
This technique, termed RAG poisoning, illustrates how tainted retrieval sources can persistently influence model behavior across conversations. Similar vulnerabilities arise through traditional injection attacks in search functionality, where SQL injection not only extracts data but also poisons the conversation memory fed to the LLM on subsequent interactions. Brian emphasizes that classic web vulnerabilities gain dramatically amplified impact when they shape the context provided to powerful generative models.
Abusing Permissions, Tool Calling, and Advanced Prompt Injection
Brian demonstrates how overly broad tool permissions create dangerous scenarios. In older models like GPT-3.5, carefully crafted prompts could coerce the LLM into executing arbitrary SQL statements with destructive consequences. Even with more recent, safety-aligned models, insufficient scoping of available tools allows privilege escalation and unauthorized actions.
Advanced prompt injection techniques go beyond simple overrides. Multi-turn attacks gradually extract personally identifiable information by leveraging accumulated conversation memory. When combined with tool calling capabilities, especially with locally hosted models, attackers can trigger hallucinations that inadvertently expose sensitive data during operations such as account creation or data processing.
The presentation underscores that granting LLMs access to powerful tools without rigorous permission boundaries and validation is equivalent to giving untrusted code broad system access.
Mitigation Strategies and Architectural Defenses
Brian outlines a comprehensive defense-in-depth approach spanning multiple layers:
-
Input and Output Guardrails: Deploying dedicated LLM-as-a-Judge mechanisms that evaluate both incoming prompts and generated outputs for malicious content, policy violations, or harmful instructions. These guardrails act as critical safety nets.
-
Limited-Scope and Permission-Aware Tools: Designing tools with granular permissions, explicit user confirmation flows for sensitive operations, and runtime validation of actions against the authenticated user’s privileges.
-
Structured Outputs and Schema Enforcement: Using techniques that force models to produce responses conforming to predefined schemas, significantly reducing the potential for unexpected or harmful outputs.
-
Model Selection and Routing: Strategically routing sensitive operations to private, self-hosted models while reserving more powerful commercial models for less critical tasks.
-
Traditional Security Foundations: Maintaining rigorous input sanitization, dependency updates, secure file handling, and regular security scanning. Brian stresses that foundational web application security remains non-negotiable even in AI-enhanced systems.
Additional considerations include implementing rate limiting to prevent “denial of pocket money” attacks that exhaust token quotas through malicious prompting, as well as comprehensive auditing of all tool invocations and model interactions.
Broader Implications for Secure AI Development
The talk concludes with forward-looking guidance for organizations adopting LLM technologies. Brian encourages treating LLMs as powerful but inherently unpredictable components requiring the same rigorous engineering discipline applied to any critical system. Key principles include careful context management, strict permission boundaries, deterministic fallback mechanisms where possible, and continuous security education for development teams.
By sharing concrete attack demonstrations and corresponding defenses, Brian equips attendees with actionable insights to build more secure, privacy-preserving AI applications while continuing to harness their transformative potential.