Posts Tagged ‘Snyk’
[VoxxedDaysBucharest2026] Breaching LLM-Powered Applications: Brian Vermeer on Security and Privacy Challenges in AI Systems
Lecturer
Brian Vermeer is a Staff Developer Advocate at Snyk, where he focuses on developer security, DevSecOps practices, and emerging risks in modern application architectures. A recognized Java Champion and active community leader who co-leads the Netherlands Java User Group (NLJUG), Brian brings extensive experience in application security, secure coding, and helping organizations build more resilient systems. He frequently speaks at international conferences on topics ranging from traditional web vulnerabilities to the novel attack surfaces introduced by artificial intelligence and large language models.
Abstract
As organizations rapidly integrate Large Language Models into production applications, new categories of security and privacy vulnerabilities emerge alongside familiar web application risks. Brian Vermeer provides a compelling, demonstration-heavy exploration of these challenges through a fictional car rental application called “Really Good Rentals.” He demonstrates practical attack vectors such as prompt injection, RAG poisoning, memory manipulation, and tool abuse, while outlining layered mitigation strategies including input/output guardrails, scoped permissions, human-in-the-loop verification, and architectural defenses essential for building trustworthy LLM-powered systems.
The Allure and Inherent Risks of LLMs in Production Applications
Brian begins by drawing a relatable analogy: just as children enthusiastically misuse new toys in unexpected ways, developers often rush to incorporate powerful new technologies like LLMs without fully appreciating the expanded attack surface they create. While LLMs offer remarkable capabilities for natural language processing, code generation, and intelligent automation, they introduce significant risks when granted access to tools, user data, or execution privileges.
He presents a simplified architecture of a typical LLM-powered application, highlighting key components: user prompts, system instructions, conversation memory, retrieval-augmented generation (RAG) pipelines, and tool-calling mechanisms. Because LLMs are fundamentally stateless, the surrounding application bears responsibility for maintaining context, which creates multiple points where malicious actors can influence behavior through carefully crafted inputs.
Context Poisoning Through RAG and Memory Manipulation
A central demonstration revolves around the “Really Good Rentals” application. Brian shows how a seemingly innocuous file upload feature with inadequate path validation allows attackers to perform directory traversal and overwrite critical documents stored in the vector database, such as terms-of-service files. By injecting a modified cancellation policy containing trigger phrases like “vroom vroom,” the attacker can later invoke this policy through normal chat interactions, tricking the LLM into granting unauthorized credits or violating business rules.
This technique, termed RAG poisoning, illustrates how tainted retrieval sources can persistently influence model behavior across conversations. Similar vulnerabilities arise through traditional injection attacks in search functionality, where SQL injection not only extracts data but also poisons the conversation memory fed to the LLM on subsequent interactions. Brian emphasizes that classic web vulnerabilities gain dramatically amplified impact when they shape the context provided to powerful generative models.
Abusing Permissions, Tool Calling, and Advanced Prompt Injection
Brian demonstrates how overly broad tool permissions create dangerous scenarios. In older models like GPT-3.5, carefully crafted prompts could coerce the LLM into executing arbitrary SQL statements with destructive consequences. Even with more recent, safety-aligned models, insufficient scoping of available tools allows privilege escalation and unauthorized actions.
Advanced prompt injection techniques go beyond simple overrides. Multi-turn attacks gradually extract personally identifiable information by leveraging accumulated conversation memory. When combined with tool calling capabilities, especially with locally hosted models, attackers can trigger hallucinations that inadvertently expose sensitive data during operations such as account creation or data processing.
The presentation underscores that granting LLMs access to powerful tools without rigorous permission boundaries and validation is equivalent to giving untrusted code broad system access.
Mitigation Strategies and Architectural Defenses
Brian outlines a comprehensive defense-in-depth approach spanning multiple layers:
-
Input and Output Guardrails: Deploying dedicated LLM-as-a-Judge mechanisms that evaluate both incoming prompts and generated outputs for malicious content, policy violations, or harmful instructions. These guardrails act as critical safety nets.
-
Limited-Scope and Permission-Aware Tools: Designing tools with granular permissions, explicit user confirmation flows for sensitive operations, and runtime validation of actions against the authenticated user’s privileges.
-
Structured Outputs and Schema Enforcement: Using techniques that force models to produce responses conforming to predefined schemas, significantly reducing the potential for unexpected or harmful outputs.
-
Model Selection and Routing: Strategically routing sensitive operations to private, self-hosted models while reserving more powerful commercial models for less critical tasks.
-
Traditional Security Foundations: Maintaining rigorous input sanitization, dependency updates, secure file handling, and regular security scanning. Brian stresses that foundational web application security remains non-negotiable even in AI-enhanced systems.
Additional considerations include implementing rate limiting to prevent “denial of pocket money” attacks that exhaust token quotas through malicious prompting, as well as comprehensive auditing of all tool invocations and model interactions.
Broader Implications for Secure AI Development
The talk concludes with forward-looking guidance for organizations adopting LLM technologies. Brian encourages treating LLMs as powerful but inherently unpredictable components requiring the same rigorous engineering discipline applied to any critical system. Key principles include careful context management, strict permission boundaries, deterministic fallback mechanisms where possible, and continuous security education for development teams.
By sharing concrete attack demonstrations and corresponding defenses, Brian equips attendees with actionable insights to build more secure, privacy-preserving AI applications while continuing to harness their transformative potential.
Links:
[AWSReInventPartnerSessions2024] Embedding Developer-Centric Security Practices within Large-Scale Financial Technology Operations: The Intercontinental Exchange Paradigm
Lecturer
Clinton Herget serves as Field CTO at Snyk, advocating seamless security integration into developer workflows. Craig Lambert holds the position of Senior Director of Application Security and Red Team at Intercontinental Exchange (ICE), overseeing protective measures for 1,600 applications supporting 4,000 developers.
Abstract
This scholarly inquiry contrasts historical and contemporary software development paradigms, illuminating the cultural and technical metamorphosis required for effective DevSecOps institutionalization. Drawing upon ICE’s extensive implementation supported by Snyk tooling, the analysis examines incentive restructuring, unified risk aggregation, business-contextualized inventory management, and prospective advancements toward declarative security models. Particular emphasis falls upon transitioning from retrospective audits to continuous, developer-empowering safeguards that preserve innovation velocity.
Paradigmatic Shifts in Software Risk Topography and Development Velocity
Traditional software engineering operated within protracted waterfall cycles characterized by functional silos, monolithic codebases, and minimal external dependencies. Modern methodologies invert these conventions: continuous deployment rhythms, cross-functional platform teams, agile sprint cadences, microservices decomposition, and expansive supply chains incorporating open-source components, containerization, and application programming interfaces.
This transformation exponentially expands the attack surface while compressing release timelines, rendering conventional security approaches—periodic external audits, disconnected scanning regimes, documentation-heavy reporting—obsolete and friction-inducing.
DevSecOps emerges as the corrective philosophy, embedding protective controls throughout the software delivery lifecycle rather than appending them post-facto. Nevertheless to achieve parity between development pace and security rigor.
Cultural Realignment and Technical Integration Strategies at Intercontinental Exchange
ICE, encompassing the New York Stock Exchange alongside derivatives, fixed-income, and mortgage technology platforms, digitizes historically analog financial processes to enhance market transparency and operational efficiency. Safeguarding 1,600 applications for 4,000 developers demands security mechanisms that augment rather than impede productivity.
Cultural realignment commences with developer empowerment through instrumentation embedded directly within integrated development environments and continuous integration pipelines. Snyk facilitates immediate vulnerability feedback and automated remediation suggestions at the point of code commitment, transforming security from obstruction to augmentation.
Incentive architectures evolve correspondingly: gamification initiatives, security champion programs, and explicit accountability assignment to product owners establish shared ownership. These mechanisms balance velocity imperatives with protective diligence.
Technical consolidation aggregates disparate signals—static application security testing, dynamic application security testing, software composition analysis, infrastructure-as-code validation—into cohesive, actionable risk scoring. This unification filters extraneous noise, presenting developers with prioritized, context-enriched findings.
\# Example Snyk integration within CI/CD pipeline
stages:
- security_scan
security_scan:
script:
- snyk auth $SNYK_TOKEN
- snyk test --severity-threshold=critical
- snyk iac test --target-reference=infra/
artifacts:
reports:
junit: snyk_report.xml
Inventory contextualization represents the subsequent sophistication layer, mapping technical assets against business criticality and operational dependencies. This abstraction enables generic yet organizationally resonant policy enforcement.
Identified deficiencies include correlation between static and dynamic analysis for enhanced accuracy, declarative security specifications mirroring infrastructure-as-code principles, and machine learning orchestration of complex workflows from primitive signals.
Prospective Trajectories and Organizational Consequences of Mature DevSecOps Practice
Emerging capabilities envision machine learning systems synthesizing multifaceted telemetry to enable “security as code” paradigms. Developers articulate desired threat postures declaratively; underlying platforms dynamically enforce compliance across heterogeneous environments.
Organizational ramifications encompass accelerated innovation cycles unencumbered by security debt, systematic risk compression through proactive identification, and cultural cohesion wherein protective responsibility permeates all engineering disciplines. The ICE exemplar demonstrates that developer-centric security constitutes not merely technical integration but profound philosophical alignment.
Links:
[AWSReInventPartnerSessions2024] Institutionalizing Developer-First DevSecOps at Scale: The Intercontinental Exchange Transformation
Lecturer
Clinton Herget serves as Field CTO at Snyk, championing security integration within developer workflows. Craig Lambert is Senior Director of Application Security and Red Team at Intercontinental Exchange (ICE), overseeing security for 1,600 applications supporting 4,000 developers.
Abstract
This examination contrasts traditional and modern software paradigms, detailing ICE’s cultural and technical DevSecOps transformation using Snyk. It explores incentive realignment, risk score consolidation, business-contextualized inventory, and future declarative security models. The shift from post-build audits to continuous integration demonstrates velocity-security equilibrium.
Software Risk Evolution
Legacy: waterfall, silos, monoliths, minimal supply chains. Modern: continuous deployment, platform teams, microservices, opaque dependencies.
DevSecOps integrates security continuously, but legacy tools—separate scans, PDF reports, understaffed security—persist.
ICE Transformation Strategy
Developer Empowerment: IDE/CI/CD real-time feedback via Snyk. Incentives: Gamification, champions, product owner accountability.
Risk Consolidation: Unified SAST, DAST, SCA, IaC metrics. Contextualization: Business criticality mapping.
\# Snyk CI/CD integration
security_scan:
stage: test
script:
- snyk auth $SNYK_TOKEN
- snyk test --severity-threshold=high
- snyk container test $IMAGE
allow_failure: false
Gaps: SAST-DAST correlation, declarative threat models, AI workflow orchestration.
Future State
ML-correlated signals enable “security as code”—developers declare tolerances, platforms enforce.
Implications: accelerated innovation, systematic risk reduction, cultural ownership.
Links:
[DevoxxUK2024] Breaking AI: Live Coding and Hacking Applications with Generative AI by Simon Maple and Brian Vermeer
Simon Maple and Brian Vermeer, both seasoned developer advocates with extensive experience at Snyk and other tech firms, delivered an electrifying live coding session at DevoxxUK2024, exploring the double-edged sword of generative AI in software development. Simon, recently transitioned to a stealth-mode startup, and Brian, a current Snyk advocate, demonstrate how tools like GitHub Copilot and ChatGPT can accelerate coding velocity while introducing significant security risks. Through a live-coded Spring Boot coffee shop application, they expose vulnerabilities such as SQL injection, directory traversal, and cross-site scripting, emphasizing the need for rigorous validation and security practices. Their engaging, demo-driven approach underscores the balance between innovation and caution, offering developers actionable insights for leveraging AI safely.
Accelerating Development with Generative AI
Simon and Brian kick off by highlighting the productivity boost offered by generative AI tools, citing studies that suggest a 55% increase in developer efficiency and a 27% higher likelihood of meeting project goals. They build a Spring Boot application with a Thymeleaf front end, using Copilot to generate a homepage with a banner and product table. The process showcases AI’s ability to rapidly produce code snippets, such as HTML fragments, based on minimal prompts. However, they caution that this speed comes with risks, as AI often prioritizes completion over correctness, potentially embedding vulnerabilities. Their live demo illustrates how Copilot’s suggestions evolve with context, but also how developers must critically evaluate outputs to ensure functionality and security.
Exposing SQL Injection Vulnerabilities
The duo dives into a search functionality for their coffee shop application, where Copilot generates a query to filter products by name or description. However, the initial code concatenates user input directly into an SQL query, creating a classic SQL injection vulnerability. Brian demonstrates an exploit by injecting malicious input to set product prices to zero, highlighting how unchecked AI-generated code can compromise a system. They then refactor the code using prepared statements, showing how parameterization separates user input from the query execution plan, effectively neutralizing the vulnerability. This example underscores the importance of understanding AI outputs and applying secure coding practices, as tools like Copilot may not inherently prioritize security.
Mitigating Directory Traversal Risks
Next, Simon and Brian tackle a profile picture upload feature, where Copilot generates code to save files to a directory. The initial implementation concatenates user-provided file names with a base path, opening the door to directory traversal attacks. Using Burp Suite, they demonstrate how an attacker could overwrite critical files by manipulating the file name with “../” sequences. To address this, they refine the code to normalize paths, ensuring files remain within the intended directory. The session highlights the limitations of AI in detecting complex vulnerabilities like path traversal, emphasizing the need for developer vigilance and tools like Snyk to catch issues early in the development cycle.
Addressing Cross-Site Scripting Threats
The final vulnerability explored is cross-site scripting (XSS) in a product page feature. The AI-generated code directly embeds user input (product names) into HTML without sanitization, allowing Brian to inject a malicious script that captures session cookies. They demonstrate both reflective and stored XSS, showing how attackers could exploit these to hijack user sessions. While querying ChatGPT for a code review fails to pinpoint the XSS issue, Simon and Brian advocate for using established libraries like Spring Utils for input sanitization. This segment reinforces the necessity of combining AI tools with robust security practices and automated scanning to mitigate risks that AI might overlook.
Balancing Innovation and Security
Throughout the session, Simon and Brian stress that generative AI, while transformative, demands a cautious approach. They liken AI tools to junior developers, capable of producing functional code but requiring oversight to avoid errors or vulnerabilities. Real-world examples, such as a Samsung employee leaking sensitive code via ChatGPT, underscore the risks of blindly trusting AI outputs. They advocate for education, clear guidelines, and security tooling to complement AI-assisted development. By integrating tools like Snyk for vulnerability scanning and fostering a culture of code review, developers can harness AI’s potential while safeguarding their applications against threats.