Posts Tagged ‘DevSecOps’
[VoxxedDaysBucharest2026] Breaching LLM-Powered Applications: Brian Vermeer on Security and Privacy Challenges in AI Systems
Lecturer
Brian Vermeer is a Staff Developer Advocate at Snyk, where he focuses on developer security, DevSecOps practices, and emerging risks in modern application architectures. A recognized Java Champion and active community leader who co-leads the Netherlands Java User Group (NLJUG), Brian brings extensive experience in application security, secure coding, and helping organizations build more resilient systems. He frequently speaks at international conferences on topics ranging from traditional web vulnerabilities to the novel attack surfaces introduced by artificial intelligence and large language models.
Abstract
As organizations rapidly integrate Large Language Models into production applications, new categories of security and privacy vulnerabilities emerge alongside familiar web application risks. Brian Vermeer provides a compelling, demonstration-heavy exploration of these challenges through a fictional car rental application called “Really Good Rentals.” He demonstrates practical attack vectors such as prompt injection, RAG poisoning, memory manipulation, and tool abuse, while outlining layered mitigation strategies including input/output guardrails, scoped permissions, human-in-the-loop verification, and architectural defenses essential for building trustworthy LLM-powered systems.
The Allure and Inherent Risks of LLMs in Production Applications
Brian begins by drawing a relatable analogy: just as children enthusiastically misuse new toys in unexpected ways, developers often rush to incorporate powerful new technologies like LLMs without fully appreciating the expanded attack surface they create. While LLMs offer remarkable capabilities for natural language processing, code generation, and intelligent automation, they introduce significant risks when granted access to tools, user data, or execution privileges.
He presents a simplified architecture of a typical LLM-powered application, highlighting key components: user prompts, system instructions, conversation memory, retrieval-augmented generation (RAG) pipelines, and tool-calling mechanisms. Because LLMs are fundamentally stateless, the surrounding application bears responsibility for maintaining context, which creates multiple points where malicious actors can influence behavior through carefully crafted inputs.
Context Poisoning Through RAG and Memory Manipulation
A central demonstration revolves around the “Really Good Rentals” application. Brian shows how a seemingly innocuous file upload feature with inadequate path validation allows attackers to perform directory traversal and overwrite critical documents stored in the vector database, such as terms-of-service files. By injecting a modified cancellation policy containing trigger phrases like “vroom vroom,” the attacker can later invoke this policy through normal chat interactions, tricking the LLM into granting unauthorized credits or violating business rules.
This technique, termed RAG poisoning, illustrates how tainted retrieval sources can persistently influence model behavior across conversations. Similar vulnerabilities arise through traditional injection attacks in search functionality, where SQL injection not only extracts data but also poisons the conversation memory fed to the LLM on subsequent interactions. Brian emphasizes that classic web vulnerabilities gain dramatically amplified impact when they shape the context provided to powerful generative models.
Abusing Permissions, Tool Calling, and Advanced Prompt Injection
Brian demonstrates how overly broad tool permissions create dangerous scenarios. In older models like GPT-3.5, carefully crafted prompts could coerce the LLM into executing arbitrary SQL statements with destructive consequences. Even with more recent, safety-aligned models, insufficient scoping of available tools allows privilege escalation and unauthorized actions.
Advanced prompt injection techniques go beyond simple overrides. Multi-turn attacks gradually extract personally identifiable information by leveraging accumulated conversation memory. When combined with tool calling capabilities, especially with locally hosted models, attackers can trigger hallucinations that inadvertently expose sensitive data during operations such as account creation or data processing.
The presentation underscores that granting LLMs access to powerful tools without rigorous permission boundaries and validation is equivalent to giving untrusted code broad system access.
Mitigation Strategies and Architectural Defenses
Brian outlines a comprehensive defense-in-depth approach spanning multiple layers:
-
Input and Output Guardrails: Deploying dedicated LLM-as-a-Judge mechanisms that evaluate both incoming prompts and generated outputs for malicious content, policy violations, or harmful instructions. These guardrails act as critical safety nets.
-
Limited-Scope and Permission-Aware Tools: Designing tools with granular permissions, explicit user confirmation flows for sensitive operations, and runtime validation of actions against the authenticated user’s privileges.
-
Structured Outputs and Schema Enforcement: Using techniques that force models to produce responses conforming to predefined schemas, significantly reducing the potential for unexpected or harmful outputs.
-
Model Selection and Routing: Strategically routing sensitive operations to private, self-hosted models while reserving more powerful commercial models for less critical tasks.
-
Traditional Security Foundations: Maintaining rigorous input sanitization, dependency updates, secure file handling, and regular security scanning. Brian stresses that foundational web application security remains non-negotiable even in AI-enhanced systems.
Additional considerations include implementing rate limiting to prevent “denial of pocket money” attacks that exhaust token quotas through malicious prompting, as well as comprehensive auditing of all tool invocations and model interactions.
Broader Implications for Secure AI Development
The talk concludes with forward-looking guidance for organizations adopting LLM technologies. Brian encourages treating LLMs as powerful but inherently unpredictable components requiring the same rigorous engineering discipline applied to any critical system. Key principles include careful context management, strict permission boundaries, deterministic fallback mechanisms where possible, and continuous security education for development teams.
By sharing concrete attack demonstrations and corresponding defenses, Brian equips attendees with actionable insights to build more secure, privacy-preserving AI applications while continuing to harness their transformative potential.
Links:
[AWSReInvent2025] Revolutionizing DevSecOps: How Cathay Pacific Achieved 75% Faster Security with Agentic AI
Lecturer
Mike Markell is a Practice Manager for AWS Professional Services in Hong Kong, where he leads digital transformation and security initiatives for major enterprises across Asia. Naresh Sharma is a senior technology leader at Cathay Pacific Airways, overseeing the airline’s global application security and DevSecOps strategy. Tony Leong is a Senior Security Architect at Cathay, specialized in building AI-powered security tooling and integrating AppSec-as-Code into high-velocity deployment pipelines.
Abstract
In the highly regulated and high-stakes environment of global aviation, managing security across more than 4,000 annual deployments presents a massive operational challenge. This article details how Cathay Pacific Airways revolutionized its “security-first” culture by moving beyond traditional security scanning to a comprehensive DevSecOps model. The core methodology centers on the implementation of Agentic AI and a RAG-based (Retrieval-Augmented Generation) assistant to solve the industry’s “false positive crisis.” By deploying “AI-powered security champions” and customized scanning rules, Cathay achieved a 75% reduction in vulnerability remediation time and a 50% reduction in security operations costs. The analysis explores the technical and cultural shifts required to empower over 1,000 developers to become proactive security practitioners while maintaining the airline’s rapid pace of innovation.
Context: The Bottleneck of Manual Security Reviews
For a global leader like Cathay Pacific, the pace of digital innovation is essential for maintaining a competitive edge in the aviation industry. However, this speed was being severely hindered by the limitations of traditional security scanning tools. The primary conflict centered on a high noise-to-signal ratio, where approximately 78% of the vulnerabilities identified by standard tools were determined to be false positives. This created a crisis where security teams were overwhelmed by alerts, leading to significant delays in the deployment of features for the airline’s fleet.
Furthermore, the manual review process required to validate these alerts created significant friction between the security and development teams. Developers often viewed security requirements as a hurdle that slowed down their ability to deliver value, while security professionals struggled to keep up with the volume of code being produced. To overcome these challenges, Cathay needed a solution that could scale with their deployment frequency—which covers everything from customer-facing apps to critical flight operation systems—without compromising on the rigorous safety standards that define the brand.
Methodology: Implementing Shift-Left Security with AI
The solution implemented by Cathay Pacific and AWS Professional Services involved a comprehensive “shift-left” strategy, which integrates security at the very beginning of the software development lifecycle. The cornerstone of this methodology is the use of Agentic AI. Unlike traditional static scanners, these AI agents act as “security champions” that provide real-time, context-aware guidance to developers as they write code. This allows for the identification of security anti-patterns and the suggestion of defensive coding practices before the code is even committed to a repository.
Another critical component of the methodology is the AppSec-as-Code library. This centralized knowledge base translates complex security policies into programmatic requirements that can be automatically enforced within CI/CD pipelines. To make this information accessible to developers, the team developed a RAG-based (Retrieval-Augmented Generation) assistant. This tool allows developers to query internal security standards using natural language, receiving accurate and context-specific advice instantly. Finally, the team moved away from “out of the box” tool configurations in favor of highly customized scanning rules. This technical fine-tuning was essential for drastically reducing the false-positive rate and ensuring that the security team only focused on legitimate threats.
Technical Analysis of Operational Gains
The implementation of AI-driven DevSecOps has yielded remarkable quantitative results for Cathay Pacific. The most significant outcome is a 75% reduction in the time required to remediate vulnerabilities. Because the AI agents filter out the vast majority of false positives and provide developers with clear, actionable fix suggestions, the entire security lifecycle has been compressed. Qualitatively, this has led to a 70% improvement in developer security capability, as the tools effectively serve as an automated, on-the-job training system that reinforces secure coding habits.
From a financial perspective, the automation of manual reviews and the reduction in wasted engineering time have led to a 50% cost reduction in security operations. The airline is now able to manage over 4,000 deployments annually with a higher level of confidence and lower overhead than was previously possible. A critical technical lesson learned during the journey was that “by default, no tool is perfect.” Success required a commitment to continuous customization and a willingness to collaborate with product vendors to tune their tools to the specific needs of the aviation industry. This iterative feedback loop was the key to moving from “human-in-the-loop” automation to a more efficient “AI-informed” model.
Consequences: A Cultural and Technical Transformation
The transformation at Cathay Pacific extended far beyond the technical architecture; it required a fundamental shift in the organization’s culture. The success of the project was predicated on a “can-do” spirit and the setting of ambitious targets that challenged the status quo. By providing developers with the tools to take ownership of security, the organization has fostered a culture where security is seen as a shared responsibility rather than an external constraint.
The implications for the global aviation and enterprise sectors are significant. Cathay has proven that it is possible to maintain a high-velocity deployment schedule in a safety-critical environment by leveraging the power of generative AI. Looking forward, the organization plans to develop even more insightful dashboards to provide security leaders with real-time visibility into the health of the application portfolio. The journey serves as a powerful testament to how Agentic AI can bridge the gap between agility and security, turning a potential bottleneck into a powerful competitive advantage.
Links:
[AWSReInventPartnerSessions2024] Embedding Developer-Centric Security Practices within Large-Scale Financial Technology Operations: The Intercontinental Exchange Paradigm
Lecturer
Clinton Herget serves as Field CTO at Snyk, advocating seamless security integration into developer workflows. Craig Lambert holds the position of Senior Director of Application Security and Red Team at Intercontinental Exchange (ICE), overseeing protective measures for 1,600 applications supporting 4,000 developers.
Abstract
This scholarly inquiry contrasts historical and contemporary software development paradigms, illuminating the cultural and technical metamorphosis required for effective DevSecOps institutionalization. Drawing upon ICE’s extensive implementation supported by Snyk tooling, the analysis examines incentive restructuring, unified risk aggregation, business-contextualized inventory management, and prospective advancements toward declarative security models. Particular emphasis falls upon transitioning from retrospective audits to continuous, developer-empowering safeguards that preserve innovation velocity.
Paradigmatic Shifts in Software Risk Topography and Development Velocity
Traditional software engineering operated within protracted waterfall cycles characterized by functional silos, monolithic codebases, and minimal external dependencies. Modern methodologies invert these conventions: continuous deployment rhythms, cross-functional platform teams, agile sprint cadences, microservices decomposition, and expansive supply chains incorporating open-source components, containerization, and application programming interfaces.
This transformation exponentially expands the attack surface while compressing release timelines, rendering conventional security approaches—periodic external audits, disconnected scanning regimes, documentation-heavy reporting—obsolete and friction-inducing.
DevSecOps emerges as the corrective philosophy, embedding protective controls throughout the software delivery lifecycle rather than appending them post-facto. Nevertheless to achieve parity between development pace and security rigor.
Cultural Realignment and Technical Integration Strategies at Intercontinental Exchange
ICE, encompassing the New York Stock Exchange alongside derivatives, fixed-income, and mortgage technology platforms, digitizes historically analog financial processes to enhance market transparency and operational efficiency. Safeguarding 1,600 applications for 4,000 developers demands security mechanisms that augment rather than impede productivity.
Cultural realignment commences with developer empowerment through instrumentation embedded directly within integrated development environments and continuous integration pipelines. Snyk facilitates immediate vulnerability feedback and automated remediation suggestions at the point of code commitment, transforming security from obstruction to augmentation.
Incentive architectures evolve correspondingly: gamification initiatives, security champion programs, and explicit accountability assignment to product owners establish shared ownership. These mechanisms balance velocity imperatives with protective diligence.
Technical consolidation aggregates disparate signals—static application security testing, dynamic application security testing, software composition analysis, infrastructure-as-code validation—into cohesive, actionable risk scoring. This unification filters extraneous noise, presenting developers with prioritized, context-enriched findings.
\# Example Snyk integration within CI/CD pipeline
stages:
- security_scan
security_scan:
script:
- snyk auth $SNYK_TOKEN
- snyk test --severity-threshold=critical
- snyk iac test --target-reference=infra/
artifacts:
reports:
junit: snyk_report.xml
Inventory contextualization represents the subsequent sophistication layer, mapping technical assets against business criticality and operational dependencies. This abstraction enables generic yet organizationally resonant policy enforcement.
Identified deficiencies include correlation between static and dynamic analysis for enhanced accuracy, declarative security specifications mirroring infrastructure-as-code principles, and machine learning orchestration of complex workflows from primitive signals.
Prospective Trajectories and Organizational Consequences of Mature DevSecOps Practice
Emerging capabilities envision machine learning systems synthesizing multifaceted telemetry to enable “security as code” paradigms. Developers articulate desired threat postures declaratively; underlying platforms dynamically enforce compliance across heterogeneous environments.
Organizational ramifications encompass accelerated innovation cycles unencumbered by security debt, systematic risk compression through proactive identification, and cultural cohesion wherein protective responsibility permeates all engineering disciplines. The ICE exemplar demonstrates that developer-centric security constitutes not merely technical integration but profound philosophical alignment.
Links:
[AWSReInventPartnerSessions2024] Institutionalizing Developer-First DevSecOps at Scale: The Intercontinental Exchange Transformation
Lecturer
Clinton Herget serves as Field CTO at Snyk, championing security integration within developer workflows. Craig Lambert is Senior Director of Application Security and Red Team at Intercontinental Exchange (ICE), overseeing security for 1,600 applications supporting 4,000 developers.
Abstract
This examination contrasts traditional and modern software paradigms, detailing ICE’s cultural and technical DevSecOps transformation using Snyk. It explores incentive realignment, risk score consolidation, business-contextualized inventory, and future declarative security models. The shift from post-build audits to continuous integration demonstrates velocity-security equilibrium.
Software Risk Evolution
Legacy: waterfall, silos, monoliths, minimal supply chains. Modern: continuous deployment, platform teams, microservices, opaque dependencies.
DevSecOps integrates security continuously, but legacy tools—separate scans, PDF reports, understaffed security—persist.
ICE Transformation Strategy
Developer Empowerment: IDE/CI/CD real-time feedback via Snyk. Incentives: Gamification, champions, product owner accountability.
Risk Consolidation: Unified SAST, DAST, SCA, IaC metrics. Contextualization: Business criticality mapping.
\# Snyk CI/CD integration
security_scan:
stage: test
script:
- snyk auth $SNYK_TOKEN
- snyk test --severity-threshold=high
- snyk container test $IMAGE
allow_failure: false
Gaps: SAST-DAST correlation, declarative threat models, AI workflow orchestration.
Future State
ML-correlated signals enable “security as code”—developers declare tolerances, platforms enforce.
Implications: accelerated innovation, systematic risk reduction, cultural ownership.
Links:
[NodeCongress2021] Security Testing for JS Apps, Node Congress – Ryan Severns
Application security need not impede developer agility; instead, it can integrate seamlessly into workflows. Ryan Severns, co-founder of StackHawk, presents a streamlined approach to vulnerability detection in JavaScript ecosystems, leveraging automation to unearth issues pre-production.
StackHawk automates dynamic analysis against JS apps and APIs—REST, GraphQL—flagging SQL injections or data leaks via CI/CD scans. On pull requests, scans mimic attacks, surfacing flaws with request/response evidence, expediting triages.
Automating Scans with ZAP Foundations
Built atop OWASP ZAP, StackHawk configures effortlessly for Node.js stacks, scanning SPAs or backends sans code mods. Post-scan, dashboards highlight exploits, with remediation docs and Jira integrations deferring low-risks, respecting only novel threats.
Integrating into DevSecOps Pipelines
Ryan emphasizes workflow harmony: GitHub Actions triggers validate endpoints, blocking merges on criticals while queuing fixes. Free tiers invite experimentation, blending security into Node.js velocity without friction.