Jonathan Lalou's Blog

Posts Tagged ‘ToolCalling’

Lecturer

Brian Vermeer is a Staff Developer Advocate at Snyk, where he focuses on developer security, DevSecOps practices, and emerging risks in modern application architectures. A recognized Java Champion and active community leader who co-leads the Netherlands Java User Group (NLJUG), Brian brings extensive experience in application security, secure coding, and helping organizations build more resilient systems. He frequently speaks at international conferences on topics ranging from traditional web vulnerabilities to the novel attack surfaces introduced by artificial intelligence and large language models.

Abstract

As organizations rapidly integrate Large Language Models into production applications, new categories of security and privacy vulnerabilities emerge alongside familiar web application risks. Brian Vermeer provides a compelling, demonstration-heavy exploration of these challenges through a fictional car rental application called “Really Good Rentals.” He demonstrates practical attack vectors such as prompt injection, RAG poisoning, memory manipulation, and tool abuse, while outlining layered mitigation strategies including input/output guardrails, scoped permissions, human-in-the-loop verification, and architectural defenses essential for building trustworthy LLM-powered systems.

The Allure and Inherent Risks of LLMs in Production Applications

Brian begins by drawing a relatable analogy: just as children enthusiastically misuse new toys in unexpected ways, developers often rush to incorporate powerful new technologies like LLMs without fully appreciating the expanded attack surface they create. While LLMs offer remarkable capabilities for natural language processing, code generation, and intelligent automation, they introduce significant risks when granted access to tools, user data, or execution privileges.

He presents a simplified architecture of a typical LLM-powered application, highlighting key components: user prompts, system instructions, conversation memory, retrieval-augmented generation (RAG) pipelines, and tool-calling mechanisms. Because LLMs are fundamentally stateless, the surrounding application bears responsibility for maintaining context, which creates multiple points where malicious actors can influence behavior through carefully crafted inputs.

Context Poisoning Through RAG and Memory Manipulation

A central demonstration revolves around the “Really Good Rentals” application. Brian shows how a seemingly innocuous file upload feature with inadequate path validation allows attackers to perform directory traversal and overwrite critical documents stored in the vector database, such as terms-of-service files. By injecting a modified cancellation policy containing trigger phrases like “vroom vroom,” the attacker can later invoke this policy through normal chat interactions, tricking the LLM into granting unauthorized credits or violating business rules.

This technique, termed RAG poisoning, illustrates how tainted retrieval sources can persistently influence model behavior across conversations. Similar vulnerabilities arise through traditional injection attacks in search functionality, where SQL injection not only extracts data but also poisons the conversation memory fed to the LLM on subsequent interactions. Brian emphasizes that classic web vulnerabilities gain dramatically amplified impact when they shape the context provided to powerful generative models.

Abusing Permissions, Tool Calling, and Advanced Prompt Injection

Brian demonstrates how overly broad tool permissions create dangerous scenarios. In older models like GPT-3.5, carefully crafted prompts could coerce the LLM into executing arbitrary SQL statements with destructive consequences. Even with more recent, safety-aligned models, insufficient scoping of available tools allows privilege escalation and unauthorized actions.

Advanced prompt injection techniques go beyond simple overrides. Multi-turn attacks gradually extract personally identifiable information by leveraging accumulated conversation memory. When combined with tool calling capabilities, especially with locally hosted models, attackers can trigger hallucinations that inadvertently expose sensitive data during operations such as account creation or data processing.

The presentation underscores that granting LLMs access to powerful tools without rigorous permission boundaries and validation is equivalent to giving untrusted code broad system access.

Mitigation Strategies and Architectural Defenses

Brian outlines a comprehensive defense-in-depth approach spanning multiple layers:

• Input and Output Guardrails: Deploying dedicated LLM-as-a-Judge mechanisms that evaluate both incoming prompts and generated outputs for malicious content, policy violations, or harmful instructions. These guardrails act as critical safety nets.

• Limited-Scope and Permission-Aware Tools: Designing tools with granular permissions, explicit user confirmation flows for sensitive operations, and runtime validation of actions against the authenticated user’s privileges.

• Structured Outputs and Schema Enforcement: Using techniques that force models to produce responses conforming to predefined schemas, significantly reducing the potential for unexpected or harmful outputs.

• Model Selection and Routing: Strategically routing sensitive operations to private, self-hosted models while reserving more powerful commercial models for less critical tasks.

• Traditional Security Foundations: Maintaining rigorous input sanitization, dependency updates, secure file handling, and regular security scanning. Brian stresses that foundational web application security remains non-negotiable even in AI-enhanced systems.

Additional considerations include implementing rate limiting to prevent “denial of pocket money” attacks that exhaust token quotas through malicious prompting, as well as comprehensive auditing of all tool invocations and model interactions.

Broader Implications for Secure AI Development

The talk concludes with forward-looking guidance for organizations adopting LLM technologies. Brian encourages treating LLMs as powerful but inherently unpredictable components requiring the same rigorous engineering discipline applied to any critical system. Key principles include careful context management, strict permission boundaries, deterministic fallback mechanisms where possible, and continuous security education for development teams.

By sharing concrete attack demonstrations and corresponding defenses, Brian equips attendees with actionable insights to build more secure, privacy-preserving AI applications while continuing to harness their transformative potential.

Links:

• Video: Breaching LLM-Powered Apps Overcoming Security and Privacy Challenges
• Brian Vermeer Website
• Brian Vermeer at Snyk
• Brian Vermeer on X/Twitter
• Snyk LLM Security Resources

Lecturer

Marcus Hellberg is the Vice President of AI Research at Vaadin, a company specializing in tools for Java developers to build web applications. As a Java Champion with nearly 20 years of experience in Java and web development, he focuses on integrating AI capabilities into Java ecosystems. Thomas Vitale is a software engineer at Systematic, a Danish software company, with expertise in cloud-native solutions, Java, and AI. He is the author of “Cloud Native Spring in Action” and an upcoming book on developer experience on Kubernetes, and serves as a CNCF Ambassador.

• Marcus Hellberg on LinkedIn
• Marcus Hellberg on GitHub
• Thomas Vitale on LinkedIn
• Thomas Vitale on GitHub

Abstract

This article examines practical patterns for incorporating artificial intelligence into Java applications using Spring AI and Vaadin, transitioning from experimental to production-ready implementations. It analyzes techniques for memory management, guardrails, multimodality, retrieval-augmented generation, tool calling, and agents, with implications for security, user experience, and system integration. Insights emphasize robust, observable AI workflows in on-premises or cloud environments.

Memory Management and Streaming in AI Interactions

Integrating large language models (LLMs) into applications requires addressing their stateless nature, where each interaction lacks inherent context from prior exchanges. Spring AI provides advisors—interceptor-like mechanisms—to augment prompts with conversation history, enabling short-term memory. For instance, a MessageChatMemoryAdvisor retains the last N messages, ensuring continuity without manual tracking.

This pattern enhances user interactions in chat-based interfaces, built here with Vaadin’s component model for server-side Java UIs. A vertical layout hosts message lists and inputs, injecting a ChatClientBuilder to construct clients with advisors. Basic interactions involve prompting the model and appending responses, but for realism, streaming via reactive fluxes improves responsiveness, subscribing to token streams and updating UI progressively.

Code illustration:

ChatClient chatClient = builder.build();
messageInput.addSubmitListener(submitEvent -> {
    String message = submitEvent.getMessage();
    MessageItem userItem = messageList.addMessage("You", message);
    chatClient.stream(new Prompt(message))
        .subscribe(response -> {
            userItem.append(response.getResult().getOutput().getContent());
        });
});

Streaming suits verbose responses, reducing perceived latency, while observability integrations (e.g., OpenTelemetry) trace interactions for debugging nondeterministic behaviors.

Guardrails for Security and Validation

AI workflows must mitigate risks like sensitive data leaks or invalid outputs. Input guardrails intercept prompts, using on-premises models to check for compliance with policies, blocking unauthorized queries (e.g., personal information). Output guardrails validate responses, reprompting for corrections if deserialization fails.

Advisors enable this: a default advisor with a local chat model filters inputs/outputs. For example, querying an address might be blocked if flagged, preventing cloud exposure. This ensures determinism in structured outputs, converting unstructured text to Java objects via JSON instructions.

Implications include privacy preservation in regulated sectors and integration with Spring Security for role-based tool access.

Multimodality and Retrieval-Augmented Generation

LLMs extend beyond text through multimodality, processing images, audio, or videos. Spring AI’s entity methods augment prompts for structured extraction, e.g., parsing attendee details from images into tables for programmatic use.

Retrieval-augmented generation (RAG) combats hallucinations by embedding external data as vectors in stores like PostgreSQL. A RetrievalAugmentationAdvisor retrieves relevant documents via similarity search, augmenting prompts. Customizations allow empty contexts for fallback to model knowledge.

Example:

VectorStore vectorStore = // PostgreSQL vector store
DocumentRetriever retriever = new VectorStoreDocumentRetriever(vectorStore);
RetrievalAugmentationAdvisor advisor = RetrievalAugmentationAdvisor.builder()
    .documentRetriever(retriever)
    .queryAugmentor(QueryAugmentor.contextual().allowEmptyContext(true))
    .build();

This pattern grounds responses in proprietary data, with thresholds controlling retrieval scope.

Tool Calling, Agents, and Dynamic Integrations

Tool calling empowers LLMs as agents, invoking external functions for tasks like database queries. Annotations describe tools, passed to clients for dynamic selection. For products, a service might expose query/update methods:

@Tool(description = "Fetch products from database")
public List<Product> getProducts(@P(description = "Category filter") String category) {
    // Database query
}

Agents orchestrate tools, potentially via Model Context Protocol for external services. Demonstrations include theme generation from screenshots, editing CSS via file system tools, highlighting nondeterminism and the need for safeguards.

In conclusion, these patterns enable production AI, emphasizing modularity, security, and observability for robust Java applications.

Links:

• Lecture Video
• Slides
• Marcus Hellberg on LinkedIn
• Thomas Vitale on LinkedIn