Posts Tagged ‘BrianVermeer’
[VoxxedDaysBucharest2026] Breaching LLM-Powered Applications: Brian Vermeer on Security and Privacy Challenges in AI Systems
Lecturer
Brian Vermeer is a Staff Developer Advocate at Snyk, where he focuses on developer security, DevSecOps practices, and emerging risks in modern application architectures. A recognized Java Champion and active community leader who co-leads the Netherlands Java User Group (NLJUG), Brian brings extensive experience in application security, secure coding, and helping organizations build more resilient systems. He frequently speaks at international conferences on topics ranging from traditional web vulnerabilities to the novel attack surfaces introduced by artificial intelligence and large language models.
Abstract
As organizations rapidly integrate Large Language Models into production applications, new categories of security and privacy vulnerabilities emerge alongside familiar web application risks. Brian Vermeer provides a compelling, demonstration-heavy exploration of these challenges through a fictional car rental application called “Really Good Rentals.” He demonstrates practical attack vectors such as prompt injection, RAG poisoning, memory manipulation, and tool abuse, while outlining layered mitigation strategies including input/output guardrails, scoped permissions, human-in-the-loop verification, and architectural defenses essential for building trustworthy LLM-powered systems.
The Allure and Inherent Risks of LLMs in Production Applications
Brian begins by drawing a relatable analogy: just as children enthusiastically misuse new toys in unexpected ways, developers often rush to incorporate powerful new technologies like LLMs without fully appreciating the expanded attack surface they create. While LLMs offer remarkable capabilities for natural language processing, code generation, and intelligent automation, they introduce significant risks when granted access to tools, user data, or execution privileges.
He presents a simplified architecture of a typical LLM-powered application, highlighting key components: user prompts, system instructions, conversation memory, retrieval-augmented generation (RAG) pipelines, and tool-calling mechanisms. Because LLMs are fundamentally stateless, the surrounding application bears responsibility for maintaining context, which creates multiple points where malicious actors can influence behavior through carefully crafted inputs.
Context Poisoning Through RAG and Memory Manipulation
A central demonstration revolves around the “Really Good Rentals” application. Brian shows how a seemingly innocuous file upload feature with inadequate path validation allows attackers to perform directory traversal and overwrite critical documents stored in the vector database, such as terms-of-service files. By injecting a modified cancellation policy containing trigger phrases like “vroom vroom,” the attacker can later invoke this policy through normal chat interactions, tricking the LLM into granting unauthorized credits or violating business rules.
This technique, termed RAG poisoning, illustrates how tainted retrieval sources can persistently influence model behavior across conversations. Similar vulnerabilities arise through traditional injection attacks in search functionality, where SQL injection not only extracts data but also poisons the conversation memory fed to the LLM on subsequent interactions. Brian emphasizes that classic web vulnerabilities gain dramatically amplified impact when they shape the context provided to powerful generative models.
Abusing Permissions, Tool Calling, and Advanced Prompt Injection
Brian demonstrates how overly broad tool permissions create dangerous scenarios. In older models like GPT-3.5, carefully crafted prompts could coerce the LLM into executing arbitrary SQL statements with destructive consequences. Even with more recent, safety-aligned models, insufficient scoping of available tools allows privilege escalation and unauthorized actions.
Advanced prompt injection techniques go beyond simple overrides. Multi-turn attacks gradually extract personally identifiable information by leveraging accumulated conversation memory. When combined with tool calling capabilities, especially with locally hosted models, attackers can trigger hallucinations that inadvertently expose sensitive data during operations such as account creation or data processing.
The presentation underscores that granting LLMs access to powerful tools without rigorous permission boundaries and validation is equivalent to giving untrusted code broad system access.
Mitigation Strategies and Architectural Defenses
Brian outlines a comprehensive defense-in-depth approach spanning multiple layers:
-
Input and Output Guardrails: Deploying dedicated LLM-as-a-Judge mechanisms that evaluate both incoming prompts and generated outputs for malicious content, policy violations, or harmful instructions. These guardrails act as critical safety nets.
-
Limited-Scope and Permission-Aware Tools: Designing tools with granular permissions, explicit user confirmation flows for sensitive operations, and runtime validation of actions against the authenticated user’s privileges.
-
Structured Outputs and Schema Enforcement: Using techniques that force models to produce responses conforming to predefined schemas, significantly reducing the potential for unexpected or harmful outputs.
-
Model Selection and Routing: Strategically routing sensitive operations to private, self-hosted models while reserving more powerful commercial models for less critical tasks.
-
Traditional Security Foundations: Maintaining rigorous input sanitization, dependency updates, secure file handling, and regular security scanning. Brian stresses that foundational web application security remains non-negotiable even in AI-enhanced systems.
Additional considerations include implementing rate limiting to prevent “denial of pocket money” attacks that exhaust token quotas through malicious prompting, as well as comprehensive auditing of all tool invocations and model interactions.
Broader Implications for Secure AI Development
The talk concludes with forward-looking guidance for organizations adopting LLM technologies. Brian encourages treating LLMs as powerful but inherently unpredictable components requiring the same rigorous engineering discipline applied to any critical system. Key principles include careful context management, strict permission boundaries, deterministic fallback mechanisms where possible, and continuous security education for development teams.
By sharing concrete attack demonstrations and corresponding defenses, Brian equips attendees with actionable insights to build more secure, privacy-preserving AI applications while continuing to harness their transformative potential.
Links:
[DevoxxUK2024] Breaking AI: Live Coding and Hacking Applications with Generative AI by Simon Maple and Brian Vermeer
Simon Maple and Brian Vermeer, both seasoned developer advocates with extensive experience at Snyk and other tech firms, delivered an electrifying live coding session at DevoxxUK2024, exploring the double-edged sword of generative AI in software development. Simon, recently transitioned to a stealth-mode startup, and Brian, a current Snyk advocate, demonstrate how tools like GitHub Copilot and ChatGPT can accelerate coding velocity while introducing significant security risks. Through a live-coded Spring Boot coffee shop application, they expose vulnerabilities such as SQL injection, directory traversal, and cross-site scripting, emphasizing the need for rigorous validation and security practices. Their engaging, demo-driven approach underscores the balance between innovation and caution, offering developers actionable insights for leveraging AI safely.
Accelerating Development with Generative AI
Simon and Brian kick off by highlighting the productivity boost offered by generative AI tools, citing studies that suggest a 55% increase in developer efficiency and a 27% higher likelihood of meeting project goals. They build a Spring Boot application with a Thymeleaf front end, using Copilot to generate a homepage with a banner and product table. The process showcases AI’s ability to rapidly produce code snippets, such as HTML fragments, based on minimal prompts. However, they caution that this speed comes with risks, as AI often prioritizes completion over correctness, potentially embedding vulnerabilities. Their live demo illustrates how Copilot’s suggestions evolve with context, but also how developers must critically evaluate outputs to ensure functionality and security.
Exposing SQL Injection Vulnerabilities
The duo dives into a search functionality for their coffee shop application, where Copilot generates a query to filter products by name or description. However, the initial code concatenates user input directly into an SQL query, creating a classic SQL injection vulnerability. Brian demonstrates an exploit by injecting malicious input to set product prices to zero, highlighting how unchecked AI-generated code can compromise a system. They then refactor the code using prepared statements, showing how parameterization separates user input from the query execution plan, effectively neutralizing the vulnerability. This example underscores the importance of understanding AI outputs and applying secure coding practices, as tools like Copilot may not inherently prioritize security.
Mitigating Directory Traversal Risks
Next, Simon and Brian tackle a profile picture upload feature, where Copilot generates code to save files to a directory. The initial implementation concatenates user-provided file names with a base path, opening the door to directory traversal attacks. Using Burp Suite, they demonstrate how an attacker could overwrite critical files by manipulating the file name with “../” sequences. To address this, they refine the code to normalize paths, ensuring files remain within the intended directory. The session highlights the limitations of AI in detecting complex vulnerabilities like path traversal, emphasizing the need for developer vigilance and tools like Snyk to catch issues early in the development cycle.
Addressing Cross-Site Scripting Threats
The final vulnerability explored is cross-site scripting (XSS) in a product page feature. The AI-generated code directly embeds user input (product names) into HTML without sanitization, allowing Brian to inject a malicious script that captures session cookies. They demonstrate both reflective and stored XSS, showing how attackers could exploit these to hijack user sessions. While querying ChatGPT for a code review fails to pinpoint the XSS issue, Simon and Brian advocate for using established libraries like Spring Utils for input sanitization. This segment reinforces the necessity of combining AI tools with robust security practices and automated scanning to mitigate risks that AI might overlook.
Balancing Innovation and Security
Throughout the session, Simon and Brian stress that generative AI, while transformative, demands a cautious approach. They liken AI tools to junior developers, capable of producing functional code but requiring oversight to avoid errors or vulnerabilities. Real-world examples, such as a Samsung employee leaking sensitive code via ChatGPT, underscore the risks of blindly trusting AI outputs. They advocate for education, clear guidelines, and security tooling to complement AI-assisted development. By integrating tools like Snyk for vulnerability scanning and fostering a culture of code review, developers can harness AI’s potential while safeguarding their applications against threats.