Posts Tagged ‘AISecurity’
[DefCon32] Threat Modeling in the Age of AI
As artificial intelligence (AI) reshapes technology, Adam Shostack, a renowned threat modeling expert, explores its implications for security. Speaking at the AppSec Village, Adam examines how traditional threat modeling adapts to large language models (LLMs), addressing real-world risks like biased hiring algorithms and deepfake misuse. His practical approach demystifies AI security, offering actionable strategies for researchers and developers to mitigate vulnerabilities in an AI-driven world.
Foundations of Threat Modeling
Adam introduces threat modeling’s four-question framework: what are we working on, what can go wrong, what are we going to do about it, and did we do a good job? This structured approach, applicable to any system, helps identify vulnerabilities in LLMs. By creating simplified system models, researchers can map AI components, such as training data and inference pipelines, to pinpoint potential failure points, ensuring a proactive stance against emerging threats.
AI-Specific Security Challenges
Delving into LLMs, Adam highlights unique risks stemming from their design, particularly the mingling of code and data. This architecture complicates secure deployment, as malicious inputs can exploit model behavior. Real-world issues, such as AI-driven resume screening biases or facial recognition errors leading to wrongful arrests, underscore the urgency of robust threat modeling. Adam notes that while LLMs excel at specific mitigation tasks, broad security questions yield poor results, necessitating precise queries.
Leveraging AI for Security Solutions
Adam explores how LLMs can enhance security practices. By generating mitigation code or test cases for specific vulnerabilities, AI can assist developers in fortifying systems. However, he cautions against over-reliance, as generic queries produce unreliable outcomes. His approach involves using AI to streamline threat identification while maintaining human oversight, ensuring that mitigations address tangible risks like data leaks or model poisoning.
Future Directions and Real-World Impact
Concluding, Adam dismisses apocalyptic AI fears but stresses immediate concerns, such as deepfake proliferation and biased decision-making. He advocates integrating threat modeling into AI development to address these issues early. By fostering a collaborative community effort, Adam encourages researchers to refine AI security practices, ensuring that LLMs serve as tools for progress rather than vectors for harm.
Links:
[DefCon32] Your AI Assistant Has a Big Mouth: A New Side-Channel Attack
As AI assistants like ChatGPT reshape human-technology interactions, their security gaps pose alarming risks. Yisroel Mirsky, a Zuckerman Faculty Scholar at Ben-Gurion University, alongside graduate students Daniel Eisenstein and Roy Weiss, unveils a novel side-channel attack exploiting token length in encrypted AI responses. Their research exposes vulnerabilities in major platforms, including OpenAI, Microsoft, and Cloudflare, threatening the confidentiality of personal and sensitive communications.
Yisroel’s Offensive AI Research Lab focuses on adversarial techniques, and this discovery highlights how subtle data leaks can undermine encryption. By analyzing network traffic, they intercept encrypted responses, reconstructing conversations from medical queries to document edits. Their findings, disclosed responsibly, prompted swift vendor patches, underscoring the urgency of securing AI integrations.
The attack leverages predictable token lengths in JSON responses, allowing adversaries to infer content despite encryption. Demonstrations reveal real-world impacts, from exposing personal advice to compromising corporate data, urging a reevaluation of AI security practices.
Understanding the Side-Channel Vulnerability
Yisroel explains the attack’s mechanics: AI assistants transmit responses as JSON objects, with token lengths correlating to content size. By sniffing HTTPS traffic, attackers deduce these lengths, mapping them to probable outputs. For instance, a query about a medical rash yields distinct packet sizes, enabling reconstruction.
Vulnerable vendors, unaware of this flaw until February 2025, included OpenAI and Quora. The team’s tool, GPTQ Logger, automates traffic analysis, highlighting the ease of exploitation in unpatched systems.
Vendor Responses and Mitigations
Post-disclosure, vendors acted decisively. OpenAI implemented padding to the nearest 32-byte value, obscuring token lengths. Cloudflare adopted random padding, further disrupting patterns. By March 2025, patches neutralized the threat, with five vendors offering bug bounties.
Yisroel emphasizes simple defenses: random padding, fixed-size packets, or increased buffering. These measures, easily implemented, prevent length-based inference, safeguarding user privacy.
Implications for AI Security
The discovery underscores a broader issue: AI services, despite their sophistication, inherit historical encryption pitfalls. Yisroel draws parallels to past side-channel attacks, where minor details like timing betrayed secrets. AI’s integration into sensitive domains demands rigorous security, akin to traditional software.
The work encourages offensive research to uncover similar weaknesses, advocating AI’s dual role in identifying and mitigating vulnerabilities. As new services emerge, proactive design is critical to prevent data exposure.
Broader Call to Action
Yisroel’s team urges the community to explore additional side channels, from compression ratios to processing delays. Their open-source tools invite further scrutiny, fostering a collaborative defense against evolving threats.
This research redefines AI assistant security, emphasizing meticulous data handling to protect user trust.
Links:
[DefCon32] On Your Ocean’s 11 Team, I’m the AI Guy (Technically Girl)
Blending the allure of high-stakes gambles with cutting-edge threats, Harriet Farlow, an AI security specialist, navigates the intersection of adversarial machine learning and casino operations. Targeting Canberra Casino, she exposes frailties in emerging AI integrations for surveillance and monitoring. Her exploits disrupt facial recognition, evade detection, and manipulate gameplay, illustrating broader perils in sectors reliant on such systems.
Harriet’s background spans physics, data science, and government intelligence, culminating in founding ML Security Labs. Her focus: deceiving AI to reveal weaknesses, akin to cyber intrusions but tailored to models’ statistical natures.
Casinos, epitomizing surveillance-heavy environments, increasingly adopt AI for identifying threats and optimizing play. Canberra, though modest, mirrors global trends where a few providers dominate, ripe for widespread impacts.
Adversarial attacks perturb inputs subtly, fooling models without human notice. Harriet employs techniques like fast gradient sign methods, crafting perturbations that reduce classification confidence.
Targeting Facial Recognition
Facial systems, crucial for barring excluded patrons, succumb to perturbations. Harriet generates adversarial examples via libraries like Foolbox, adding noise that misclassifies faces.
Tests show 40.4% success in evading matches, but practical adaptations ensure consistent bypasses. This equates to denial-of-service equivalents in AI, disrupting reliability.
Broader implications span medical diagnostics to autonomous navigation, where minor alterations yield catastrophic errors.
Evading Surveillance and Gameplay Monitoring
Surveillance AI detects anomalies; Harriet’s perturbations obscure actions, mimicking wild exploits.
Gameplay AI monitors for advantages; adversarial inputs confuse chip recognition or behavior analysis, enabling undetected strategies.
Interviews with casino personnel reveal heavy reliance on human oversight, despite AI promises. Only 8% of surveyed organizations secure AI effectively, versus 94% using it.
Lessons from the Inflection Point
Casinos transition to AI amid regulatory voids, amplifying risks. Harriet advocates integrating cyber lessons: robust testing beyond accuracy, incorporating security metrics.
Her findings stress governance: people and processes remain vital, yet overlooked. As societies embrace AI surveillance, vulnerabilities threaten equity and safety.
Harriet’s work urges cross-disciplinary approaches, blending cyber expertise with AI defenses to mitigate emerging dangers.
Links:
EN_DEFCON32MainStageTalks_011_013.md