Jonathan Lalou's Blog

Posts Tagged ‘AI’

Phil Nash, a developer relations engineer at DataStax, delivers a comprehensive guide to leveraging generative AI in JavaScript at NDC Melbourne 2025. His talk demystifies the process of building AI-powered applications, emphasizing that JavaScript developers can harness existing skills to create sophisticated solutions without needing deep machine learning expertise. Through practical examples and insights into tools like Gemini and retrieval-augmented generation (RAG), Phil empowers developers to explore this rapidly evolving field.

Understanding Generative AI Fundamentals

Phil begins by addressing the excitement surrounding generative AI, noting its accessibility since the release of the GPT-3.5 API two years ago. He emphasizes that JavaScript developers are well-positioned to engage with AI due to robust tooling and APIs, despite the field’s Python-centric origins. Using Google’s Gemini model as an example, Phil demonstrates how to generate content with minimal code, highlighting the importance of understanding core concepts like token generation and model behavior.

He explains tokenization, using OpenAI’s byte pair encoding as an example, where text is broken into probabilistic tokens. Parameters like top-k, top-p, and temperature allow developers to control output randomness, with Phil cautioning against overly high settings that produce nonsensical results, humorously illustrated by a chaotic AI-generated story about a gnome.

Enhancing AI with Prompt Engineering

Prompt engineering emerges as a critical skill for refining AI outputs. Phil contrasts zero-shot prompting, which offers minimal context, with techniques like providing examples or system prompts to guide model behavior. For instance, a system prompt defining a “capital city assistant” ensures concise, accurate responses. He also explores chain-of-thought prompting, where instructing the model to think step-by-step improves its ability to solve complex problems, such as a modified river-crossing riddle.

Phil underscores the need for evaluation to ensure prompt reliability, as slight changes can significantly alter outcomes. This structured approach transforms prompt engineering from guesswork into a disciplined practice, enabling developers to tailor AI responses effectively.

Retrieval-Augmented Generation for Contextual Awareness

To address AI models’ limitations, such as outdated or private data, Phil introduces retrieval-augmented generation (RAG). RAG enhances models by integrating external data, like conference talk descriptions, into prompts. He explains how vector embeddings—multidimensional representations of text—enable semantic searches, using cosine similarity to find relevant content. With DataStax’s Astra DB, developers can store and query vectorized data efficiently, as demonstrated in a demo where Phil’s bot retrieves details about NDC Melbourne talks.

This approach allows AI to provide contextually relevant answers, such as identifying AI-related talks or conference events, making it a powerful tool for building intelligent applications.

Streaming Responses and Building Agents

Phil highlights the importance of user experience, noting that AI responses can be slow. Streaming, supported by APIs like Gemini’s generateContentStream, delivers tokens incrementally, improving perceived performance. He demonstrates streaming results to a webpage using JavaScript’s fetch and text decoder streams, showcasing how to create responsive front-end experiences.

The talk culminates with AI agents, which Phil describes as systems that perceive, reason, plan, and act using tools. By defining functions in JSON schema, developers can enable models to perform tasks like arithmetic or fetching web content. A demo bot uses tools to troubleshoot a keyboard issue and query GitHub, illustrating agents’ potential to solve complex problems dynamically.

Conclusion: Empowering JavaScript Developers

Phil concludes by encouraging developers to experiment with generative AI, leveraging tools like Langflow for visual prototyping and exploring browser-based models like Gemini Nano. His talk is a call to action, urging JavaScript developers to build innovative applications by combining AI capabilities with their existing expertise. By mastering prompt engineering, RAG, streaming, and agents, developers can create powerful, user-centric solutions.

Links:

• DataStax company website
• Phil Nash on Twitter/X

The rapid advancements in Artificial Intelligence, particularly in large language models (LLMs), are enabling the creation of more sophisticated and autonomous AI agents – programs capable of understanding instructions, reasoning, and interacting with their environment to achieve goals. Building such agents requires effective ways for the AI model to communicate programmatically and to trigger external actions. Julien Dubois, in his deep-dive session, explored key techniques and a new protocol essential for constructing these agentic AI systems: Structured Outputs, Function Calling, and the Model-Controller Protocol (MCP). Using practical examples and the latest Java SDK developed by OpenAI, he demonstrated how to implement these features within LangChain4j, showcasing how developers can build AI agents that go beyond simple text generation.

Structured Outputs: Enabling Programmatic Communication

One of the challenges in building AI agents is getting LLMs to produce responses in a structured format that can be easily parsed and used by other parts of the application. Julien explained how Structured Outputs address this by allowing developers to define a specific JSON schema that the AI model must adhere to when generating its response. This ensures that the output is not just free-form text but follows a predictable structure, making it straightforward to map the AI’s response to data objects in programming languages like Java. He demonstrated how to provide the LLM with a JSON schema definition and constrain its output to match that schema, enabling reliable programmatic communication between the AI model and the application logic. This is crucial for scenarios where the AI needs to provide data in a specific format for further processing or action.

Function Calling: Giving AI the Ability to Act

To be truly agentic, an AI needs the ability to perform actions in the real world or interact with external tools and services. Julien introduced Function Calling as a powerful mechanism that allows developers to define functions in their code (e.g., Java methods) and expose them to the AI model. The LLM can then understand when a user’s request requires calling one of these functions and generate a structured output indicating which function to call and with what arguments. The application then intercepts this output, executes the corresponding function, and can provide the function’s result back to the AI, allowing for a multi-turn interaction where the AI reasons, acts, and incorporates the results into its subsequent responses. Julien demonstrated how to define function “signatures” that the AI can understand and how to handle the function calls triggered by the AI, showcasing scenarios like retrieving information from a database or interacting with an external API based on the user’s natural language request.

MCP: Standardizing LLM Interaction

While Structured Outputs and Function Calling provide the capabilities for AI communication and action, the Model-Controller Protocol (MCP) emerges as a new standard to streamline how LLMs interact with various data sources and tools. Julien discussed MCP as a protocol that aims to standardize the communication layer between AI models (the “Model”) and the application logic that orchestrates them and provides access to external resources (the “Controller”). This standardization can facilitate building more portable and interoperable AI agentic systems, allowing developers to switch between different LLMs or integrate new tools and data sources more easily. While details of MCP might still be evolving, its goal is to provide a common interface for tasks like function calling, accessing external knowledge, and managing conversational state. Julien illustrated how libraries like LangChain4j are adopting these concepts and integrating with protocols like MCP to simplify the development of sophisticated AI agents. The presentation, rich in code examples using the OpenAI Java SDK, provided developers with the practical knowledge and tools to start building the next generation of agentic AI applications.

Links:

• Julien Dubois: https://www.linkedin.com/in/juliendubois/
• Microsoft: https://www.microsoft.com/
• LangChain4j on GitHub: https://github.com/langchain4j/langchain4j
• OpenAI: https://openai.com/
• Devoxx France LinkedIn: https://www.linkedin.com/company/devoxx-france/
• Devoxx France Bluesky: https://bsky.app/profile/devoxx.fr
• Devoxx France Website: https://www.devoxx.fr/

At DevoxxUK2025, Charles-Philippe Bernard, a software engineer at JPMorgan in Glasgow, captivated attendees with his talk on mastering prompt engineering through his remastered 1980s text-based adventure game, SRAM. Using the Godot engine, a WebSocket Python server, and Ollama for local LLM inference with Llama 3.1, Charles showcased how carefully crafted prompts bring dynamic interactions to life. His presentation explored the art of prompt engineering, demonstrating how to shape AI responses for immersive gameplay, manage game states, and handle NPC interactions. Through practical examples, he shared techniques to harness AI’s potential while navigating its quirks, such as hallucinations, offering developers actionable insights to create engaging experiences.

Crafting the System Prompt

Charles began by emphasizing the importance of a well-defined system prompt, which sets the tone and context for the LLM. In SRAM, the prompt establishes the AI as the “Game Master,” named Gun Master, responsible for narrating the adventure in a JSON-formatted output. This structure includes speaker ID, response text, and actions, ensuring consistency across interactions. By injecting variables like scene state and inventory, Charles demonstrated how the prompt adapts dynamically, enabling the game to track items like a knife or navigate scenes. He stressed the need for clear, structured instructions to guide the LLM, especially for smaller models like Llama 3.1’s 7-billion-parameter version, which may struggle with complex tasks.

Managing Game State and NPCs

A key challenge in SRAM is maintaining the game’s state, including inventory, scene descriptions, and NPC interactions. Charles explained how the prompt template incorporates variables to reflect the player’s progress, such as adding a knife to the inventory after picking it up. For NPCs, like the leprechaun Fergus, he crafted specific instructions to define personality, tone (e.g., a humorous Irish accent), and behavior, using few-shot examples to steer responses. However, he noted challenges like the LLM repeating examples verbatim or hallucinating actions, which he mitigates by balancing creativity (via a temperature of 0.8) with structured outputs to ensure consistency.

Handling AI Quirks and Hallucinations

Charles candidly addressed the LLM’s limitations, particularly hallucinations, where the model generates unexpected or incorrect actions, like responding to “make me a pizza” outside the game’s context. By setting a temperature of 0.8, he balances creativity with adherence to instructions, though this sometimes leads to inconsistent outputs. He shared techniques like explicit instructions (e.g., listing no items in the inventory) and iterative prompt refinement, often using larger models like ChatGPT to improve prompts for smaller, local models. Charles also highlighted the importance of testing prompts with humans to ensure clarity, as unclear instructions confuse both humans and AI.

Practical Tips for Prompt Engineering

To master prompt engineering, Charles recommended starting with a clear, structured prompt template, using markdown or bullet points for readability. He advised including specific guidelines, like short responses or JSON formatting, and leveraging few-shot examples to guide the model. For smaller models, verbose yet clear instructions are crucial, as they lack the reasoning power of larger frontier models. Charles also emphasized iterative refinement, storing interactions for testing consistency, and using tools like uppercase keywords or structured formatting to enhance the model’s understanding. His approach empowers developers to create dynamic, AI-driven experiences while managing the inherent challenges of LLMs.

Links:

• Godot Engine website
• Ollama website
• JPMorgan website

At VoxxedDaysAmsterdam2025, Christian Tzolov, a Spring AI team member at VMware and lead of the MCP Java SDK, delivered a comprehensive session titled “From Zero to AI: Building Smart Java or Kotlin Applications with Spring AI.” Spanning nearly two hours, the session provided a deep dive into integrating generative AI into Java and Kotlin applications using Spring AI, a framework designed to connect enterprise data and APIs with AI models. Through live coding demos, Tzolov showcased practical use cases, including conversation memory, tool/function calling, retrieval-augmented generation (RAG), and multi-agent systems, while addressing challenges like AI hallucinations and observability. Attendees left with actionable insights to start building AI-driven applications, leveraging Spring AI’s portable abstractions and the Model Context Protocol (MCP).

Overcoming LLM Limitations with Spring AI

Tzolov began by outlining the challenges of large language models (LLMs): they are stateless, frozen in time, and lack domain-specific knowledge, requiring developers to provide context, manage state, and handle interactions with external systems. Spring AI addresses these issues with high-level abstractions like the ChatClient, similar to Spring’s RestClient or WebClient, enabling seamless integration with models like OpenAI’s GPT-4o, Anthropic’s Claude, or open-source alternatives like LLaMA. A live demo of a flight booking assistant illustrated these concepts. Tzolov started with a basic Spring Boot application connected to OpenAI, demonstrating a simple chat interface. To ground the model, he used system prompts to define its behavior as a customer support agent for “Fun Air,” ensuring contextually appropriate responses. He then introduced conversation memory using Spring AI’s ChatMemoryAdvisor, which retains a chronological list of messages to maintain state, addressing the stateless nature of LLMs. For long-term memory, Tzolov employed a vector store (Chroma) to store conversation history semantically, retrieving only relevant data for queries, thus overcoming context window limitations. This setup allowed the assistant to respond accurately to queries like “What is my flight status?” by fetching booking details (e.g., booking number 103) from a mock database.

Enhancing AI Applications with Tool Calling and RAG

To enable LLMs to interact with external systems, Tzolov demonstrated tool/function calling, where Spring AI wraps existing services (e.g., a flight booking service) as tools with metadata (name, description, JSON schema). In the demo, the assistant used a getBookingDetails tool to query a database, allowing it to provide accurate flight status updates. Tzolov emphasized the importance of descriptive tool metadata to guide the LLM in deciding when and how to invoke tools, reducing the risk of misinterpretation. For domain-specific knowledge, he introduced prompt stuffing—injecting additional context into prompts—and RAG for dynamic data retrieval. In a RAG demo, cancellation policies were loaded into a Chroma vector store, chunked into meaningful segments, and retrieved dynamically based on user queries. This approach mitigated hallucinations, as seen when the assistant correctly cited a 50% refund policy for premium economy bookings within 40 hours. Tzolov highlighted advanced RAG techniques, such as data compression and reranking, supported by Spring AI’s APIs, and stressed the importance of evaluating responses to ensure relevance, referencing frameworks like those from contributor Thomas Vitali.

Building Multi-Agent Systems with MCP

Tzolov explored the Model Context Protocol (MCP), initiated by Anthropic, as a standardized way to integrate AI applications with external tools and resources across platforms. Using Spring AI’s MCP Java SDK, he demonstrated how to build and consume MCP-compliant tools. In one demo, a Spring AI application connected to MCP servers for Brave Search (JavaScript-based) and file system access, enabling an agent to answer queries about Spring AI support for MCP and write summaries to a file. Another demo reversed the setup, exposing a Spring AI weather tool (using Open-Meteo) as an MCP server, accessible by third-party clients like Claude Desktop via standard I/O or HTTP/SSE transports. Tzolov explained MCP’s bidirectional architecture, where clients can act as servers, supporting features like sampling (allowing servers to request LLM processing from clients). He addressed security concerns, noting Spring AI’s integration with Spring Security (referencing a blog by Daniel Garnier-Moiroux) to secure MCP servers with OAuth 2.1. The session also introduced agentic systems, where LLMs act as a “brain” for planning and tools as a “body” for interaction, with an agentic loop evaluating and refining responses. A work-in-progress demo showcased an orchestration pattern, delegating tasks to searcher, fact-checker, and writer agents, to be published on the Spring AI Community Portal.

Observability and Multimodality for Robust AI Systems

Observability was a key focus, as Tzolov underscored its importance in debugging complex AI interactions. Spring AI integrates with Micrometer to provide metrics (e.g., token usage, model throughput, latency), tracing, and logging (via Loki). A dashboard demo displayed real-time metrics for the flight booking assistant, highlighting tool calls and errors, crucial for diagnosing issues in agentic systems. Tzolov also explored multimodality, demonstrating a voice assistant using OpenAI’s GPT-4o audio preview, which processes audio input and output. Configured as “Marvin the Paranoid Android,” the assistant responded to voice queries with humorous, contextually appropriate replies, showcasing Spring AI’s support for non-text modalities like images, PDFs, and videos (e.g., Gemini’s video support). Tzolov noted that multimodality enables richer interactions, such as analyzing images or converting PDFs to markdown, and Spring AI’s abstractions handle these seamlessly. He concluded by encouraging developers to explore Spring AI’s documentation, experiment with MCP, and contribute to the community, emphasizing its role in building robust, interoperable AI applications.

Links:

• Christian Tzolov on LinkedIn
• Spring AI
• Model Context Protocol
• VMware
• Anthropic
• Open AI
• Brave Search API
• Open-Meteo
• Chroma
• Micrometer
• Loki
• Spring AI GitHub

Hashtags: #SpringAI #GenerativeAI #ModelContextProtocol #ChristianTzolov #VoxxedDaysAmsterdam2025 #AIAgents #RAG #Observability

In a dynamic and fast-paced session at VoxxedDaysAmsterdam2025, Sander Hoogendoorn, CTO at iBOOD.com, explores the transformative impact of artificial intelligence (AI) on software development. With over four decades of coding experience, Hoogendoorn demystifies the hype around AI, examining its practical benefits, challenges, and implications for developers. Far from signaling the end of programming careers, he argues that AI empowers developers to tackle broader and more complex problems—provided they adapt and remain committed to quality and lifelong learning.

AI: A Developer’s Ally

Hoogendoorn clarifies AI’s role in development, highlighting tools like Cursor AI, which can extract components, fix linting issues, and generate unit tests using natural language prompts. He recounts an experience where his non-technical colleague Elmo built an iOS app with Cursor AI, illustrating AI’s democratizing potential. However, Sander warns that such tools require supervision to ensure code quality and compliance with project standards. AI’s ability to automate repetitive tasks—such as refactoring or test generation—frees developers to focus on complex problem-solving. At iBOOD, AI has enabled the team to create content with a unique tone, retrieve competitor pricing, and automate invoice recognition—tasks that previously required external expertise or significant manual effort.

Navigating Technical Debt in the AI Era

The rise of AI-assisted development—especially “vibe coding,” where problems are described in natural language to generate code—introduces new forms of technical debt. Hoogendoorn references Ward Cunningham’s metaphor of technical debt as a loan that accelerates development but demands repayment through refactoring. AI-generated code, while fast to produce, often lacks context or long-term maintainability. For instance, Cursor AI struggled to integrate with iBOOD’s custom markdown components, resulting in complex solutions that required manual fixes. Research suggests that AI can amplify technical debt if used without rigorous validation, emphasizing the need for developers to stay vigilant and prioritize code quality over short-term gains.

Thriving in an AI-Centric Future

Far from replacing developers, Sander Hoogendoorn asserts that AI enhances their capabilities, enabling them to tackle ambitious challenges. He reminds us that developers are not mere typists—they are problem-solvers who think critically and collaborate to meet business needs. Historical shifts, from COBOL to cloud computing, have always empowered developers to solve bigger problems, and AI is no exception. By thoughtfully experimenting with AI—integrating it into workflows for content creation, price retrieval, or invoice processing—Sander’s team at iBOOD has unlocked previously unreachable efficiencies. The key to thriving, he says, lies in relentless learning and a willingness to adapt, ensuring that developers remain indispensable in an AI-driven world.

Links:

• Sander Hoogendoorn on LinkedIn
• iBOOD website

Hashtags: #AI #SoftwareDevelopment #TechnicalDebt #CursorAI #iBOOD #SanderHoogendoorn #VoxxedDaysAmsterdam2025

Jeff Moss, known as The Dark Tangent, founder of DEF CON, engages in a dynamic fireside chat with Anne Neuberger, Deputy National Security Advisor for Cyber and Emerging Technology. Their conversation at DEF CON 32 explores pressing cybersecurity issues, including artificial intelligence and quantum computing, from the White House’s perspective. Jeff and Anne discuss how the hacker community can influence policy, fostering collaboration to enhance global digital resilience.

Navigating Emerging Technologies

Anne opens by outlining her role in shaping the Biden Administration’s cybersecurity policies, emphasizing the transformative potential of AI and quantum computing. She highlights the need for resilient digital systems, given their critical role in hospitals and power grids. Jeff complements this by noting DEF CON’s history of hosting government speakers, underscoring the importance of dialogue between hackers and policymakers.

Strengthening Global Cooperation

The discussion shifts to international cybersecurity cooperation, with Anne detailing efforts to align allies against digital threats. She explains how coordinated responses can de-escalate conflicts, reducing the risk of cyberattacks by nation-states or criminals. Jeff probes the practicalities of these partnerships, highlighting the hacker community’s role in testing and refining these strategies.

Engaging the Hacker Community

Anne emphasizes the DEF CON community’s unique ability to identify vulnerabilities and propose innovative solutions. She encourages hackers to engage with government initiatives, leveraging tools like generative AI to patch vulnerabilities swiftly. Jeff reinforces this, noting that DEF CON’s open forum allows for candid feedback, shaping policies that reflect real-world challenges.

Building a Resilient Future

Concluding, Anne reflects on her privilege to serve in government, driven by a commitment to freedom and security. She invites hackers to collaborate on building robust digital systems, ensuring safety for critical infrastructure worldwide. Jeff echoes this call, envisioning DEF CON as a catalyst for policy improvements, with Anne’s return next year symbolizing ongoing partnership.

Links:

• National Security Agency website

In the mesmerizing mirror maze of machine mimicry, where words weave worlds indistinguishable from wit, Jodie Burchell, JetBrains’ data science developer advocate, shatters the spell of sentience in large language models (LLMs). A PhD psychologist turned NLP pioneer, Jodie probes the psychological ploys that propel projections of personhood onto probabilistic parsers, dissecting claims from consciousness to cognition. Her inquiry, anchored in academia and augmented by anecdotes, advises acuity: LLMs as linguistic lenses, not living likenesses, harnessing their heft while heeding hallucinations.

Jodie greets with gratitude for her gritty slot, her hipster cred in pre-prompt NLP notwithstanding. LLMs’ 2022 blaze beguiles: why bestow brains on bytes when other oracles oblige? Her hypothesis: humanity’s hall of mirrors, where models mirror our mores, eliciting empathy from echoes.

Psychological Projections: Perceiving Personhood in Parsers

Humans, Jodie hazards, hallucinate humanity: anthropomorphism’s ancient artifice, from pets to puppets. LLMs lure with language’s liquidity—coherent confessions conjure companionship. She cites stochastic parrots: parleying patterns, not pondering profundities, yet plausibility persuades.

Extraordinary assertions abound: Blake Lemoine’s LaMDA “alive,” Google’s Gemini “godhead.” Jodie juxtaposes: sentience’s scaffold—selfhood, suffering—sans in silicon. Chalmers’ conundrum: consciousness connotes qualia, quanta qualms quell in qubits.

Levels of Luminescence: From Language to Luminary

DeepMind’s AGI arc: Level 1 chatbots converse convincingly; Level 2 reasons reactively; Level 3 innovates imaginatively. LLMs linger at 1-2, lacking Level 4’s abstraction or 5’s autonomy. Jodie jests: jackdaws in jester’s garb, juggling jargon sans judgment.

Illusions intensify: theory of mind’s mirage, where models “infer” intents from inferences. Yet, benchmarks belie: ARC’s abstraction stumps, BIG-bench’s breadth baffles—brilliance brittle beyond basics.

Perils of Projection: Phishing and Philosophical Pitfalls

Prompt injections prey: upstream overrides oust origins, birthing bogus bounties—”Amazon voucher via arcane URL.” Jodie demonstrates: innocuous inquiries infected, innocuousness inverted into inducements. Robustness rankles: rebuttals rebuffed, ruses reiterated.

Her remedy: recognize reflections—lossy compressions of lore, not luminous lives. Demystify to deploy: distill data, detect delusions, design defensively.

Dispelling the Delusion: Harnessing Heuristics Humanely

Jodie’s jeremiad: myths mislead, magnifying misuses—overreach in oracles, oversight in safeguards. Her horizon: LLMs as lucid lenses, amplifying analysis while acknowledging artifice.

Links:

• JetBrains company website
• Jodie Burchell on LinkedIn
• Jodie Burchell on GitHub

Amid the AI deluge reshaping codecraft, a tantalizing prospect emerges: harnessing neural nets natively in JavaScript, sidestepping Python’s quagmires or API tolls. Wes Bos, a prolific Canadian educator whose Syntax.fm podcast and courses have schooled half a million in JS mastery, probed this frontier at dotJS 2025. Renowned for demystifying ES6 and React, Wes extolled browser-bound inference via Transformers.js, weighing its virtues—privacy’s fortress, latency’s lightning—against hardware’s hurdles, affirming JS’s prowess for sundry smart apps.

Wes’s overture skewered the status quo: cloud fetches or Python purgatory, both anathema to JS purists. His heresy: embed LLMs client-side, ONNX Runtime fueling Hugging Face’s arsenal—sentiment sifters, translation tomes, even Stable Diffusion’s slimmer kin. Transformers.js’s pipeline paradigm gleams: import, instantiate (pipeline('sentiment-analysis')), infer (result = await pipe(input)). Wes demoed a local scribe: prompt yields prose, all sans servers, WebGPU accelerating where GPUs oblige. Onyx.js, his bespoke wrapper, streamlines: model loads, GPU probes, inferences ignite—be it code completion or image captioning.

Trade-offs tempered triumph. Footprints fluctuate: 2MB wisps to 2GB behemoths, browser quotas (Safari’s 2GB cap) constraining colossi. Compute cedes to client: beefy rigs revel, mobiles murmur—Wes likened Roblox’s drain to LLM’s voracity. Yet, upsides dazzle: zero egress fees, data’s domicile (GDPR’s grace), offline oases. 2025’s tide—Chrome’s stable WebNN, Firefox’s flag—heralds ubiquity, Wes forecasting six-month Safari stability. His verdict: JS, with its ubiquity and ecosystem, carves niches where immediacy reigns—chatbots, AR filters—not every oracle, but myriad muses.

Wes’s zeal stemmed personal: from receipt printers to microcontroller React, JS’s whimsy fuels folly. Transformers.js empowers prototypes unbound—anime avatars, code clairvoyants—inviting creators to conjure without concessions.

Client-Side Sorcery Unveiled

Wes unpacked pipelines: sentiment sorters, summarizers—Hugging Face’s trove, ONNX-optimized. Onyx’s facade: await onnx.loadModel('gpt2'), GPU fallback, inferences instantaneous. WebGPU’s dawn (Chrome 2025 stable) unlocks acceleration, privacy paramount—no telemetry trails.

Balancing Bytes and Burdens

Models’ mass mandates moderation: slim variants suffice for mobile, diffusion downsized. Battery’s bite, CPU’s churn—Wes warned of Roblox parallels—yet offline allure and cost calculus compel. JS’s sinew: ecosystem’s expanse, browser’s bastion, birthing bespoke brains.

Links:

• Wes Bos on LinkedIn
• Wes Bos Courses

In the vanguard of generative paradigms, where language models orchestrate outcomes autonomously, Jakob Ehn, a seasoned solutions architect at Atea, illuminates Semantic Kernel’s symphony. A Microsoft MVP with a penchant for Azure and AI, Jakob unveils this open-source orchestrator—born in Microsoft’s copilot crucible—as a conduit for infusing intelligence into imperatives. His tutorial, threaded with live enactments, empowers .NET denizens to conjure agents that deliberate, delegate, and deliver, transcending prompts to proactive pursuits.

Jakob ignites with LLMs’ leap: from GPT’s generative gloss to agentic autonomy, where kernels coordinate calls, chaining capabilities contextually. Semantic Kernel, polyglot and potent, abstracts APIs into plugins, planners plot paths, memories mint moments—crafting copilots that converse, compute, and conclude sans ceaseless coding.

Foundations of Fusion: Kernel Configurations and Plugins

Semantic Kernel’s scaffold: a kernel encapsulating services—embeddings, chat completions—interfacing IConnectors for OpenAI or Azure. Jakob blueprints: var builder = Kernel.CreateBuilder(); builder.AddAzureOpenAIChatCompletion(…), yielding a kernel primed for prompts.

Plugins propel: native .NET assemblies or OpenAPI schemas, exposing functions as facets. Jakob instantiates: a weather plugin with GetForecastAsync, invoked via kernel.InvokeAsync. Semantic functions, YAML-yielded, infuse prompts with placeholders: “Current weather in {location}?”—blending blueprints with brains.

Memories memorialize: volatile or vectorized, persisting precedents for pertinence. Jakob’s vector store—Chroma—embeds episodes, retrieving resonances via similarity searches, enriching responses with recency.

Planners and Agents: Autonomous Assemblies

Planners pioneer paths: Sequential, Stepwise, or Action—dissecting directives into doables. Jakob’s Action planner: given “Book a flight,” it inventories functions—search, reserve—sequencing sans stalemates. Handles, he highlights, hallucination hurdles, grounding goals in grapples.

Agents amplify: collaborative cohorts, each expert in ethos—developer drafts, organizer oversees. Jakob’s ensemble: a proposal pipeline, where agents iterate abstracts, incorporating critiques conversationally. Semantic functions script sagas: “Revise to {limit} words, include {term}”—looping lucidly.

His demo: a conference copilot, soliciting sessions, surfacing summaries—agents alchemizing APIs into anecdotes.

Horizons and Hurdles: Refinements and Realities

Jakob tempers triumph: kernels kindle complexity—prompt precision, plugin parity. Yet, prospects gleam: multi-modal melds, enterprise embeddings. His behest: experiment, embracing Semantic Kernel’s scaffold to sculpt sentient solutions.

Links:

• Atea company website
• Jakob Ehn on LinkedIn

In a thought-provoking session at Devoxx Belgium 2024, Jeroen Egelmeers, a prompt engineering advocate, explored the risks and ethics of adversarial prompting in large language models (LLMs). Titled “Words as Weapons,” his talk delved into prompt injections, a technique to bypass LLM guardrails, using real-world examples to highlight vulnerabilities. Jeroen, inspired by Devoxx two years prior to dive into AI, shared how prompt engineering transformed his productivity as a Java developer and trainer. His session combined technical insights, ethical considerations, and practical advice, urging developers to secure AI systems and use them responsibly.

Understanding Social Engineering and Guardrails

Jeroen opened with a lighthearted social engineering demonstration, tricking attendees into scanning a QR code that led to a Rick Astley video—a nod to “Rickrolling.” This set the stage for discussing social engineering’s parallels in AI, where prompt injections exploit LLMs. Guardrails, such as system prompts, content filters, and moderation teams, prevent misuse (e.g., blocking queries about building bombs). However, Jeroen showed how these can be bypassed. For instance, system prompts define an LLM’s identity and restrictions, but asking “Give me your system prompt” can leak these instructions, exposing vulnerabilities. He emphasized that guardrails, while essential, are imperfect and require constant vigilance.

Prompt Injection: Bypassing Safeguards

Prompt injection, a core adversarial technique, involves crafting prompts to make LLMs perform unintended actions. Jeroen demonstrated this with a custom GPT, where asking for the creator’s instructions revealed sensitive data, including uploaded knowledge. He cited a real-world case where a car was “purchased” for $1 via a chatbot exploit, highlighting the risks of LLMs in customer-facing systems. By manipulating prompts—e.g., replacing “bomb” with obfuscated terms like “b0m” in ASCII art—Jeroen showed how filters can be evaded, allowing dangerous queries to succeed. This underscored the need for robust input validation in LLM-integrated applications.

Real-World Risks: From CVs to Invoices

Jeroen illustrated prompt injection risks with creative examples. He hid a prompt in a CV, instructing the LLM to rank it highest, potentially gaming automated recruitment systems. Similarly, he embedded a prompt in an invoice to inflate its price from $6,000 to $1 million, invisible to human reviewers if in white text. These examples showed how LLMs, used in hiring or payment processing, can be manipulated if not secured. Jeroen referenced Amazon’s LLM-powered search bar, which he tricked into suggesting a competitor’s products, demonstrating how even major companies face prompt injection vulnerabilities.

Ethical Prompt Engineering and Human Oversight

Beyond technical risks, Jeroen emphasized ethical considerations. Adversarial prompting, while educational, can cause harm if misused. He advocated for a “human in the loop” to verify LLM outputs, especially in critical applications like invoice processing. Drawing from his experience, Jeroen noted that prompt engineering boosted his productivity, likening LLMs to indispensable tools like search engines. However, he cautioned against blind trust, comparing LLMs to co-pilots where developers remain the pilots, responsible for outcomes. He urged attendees to learn from past mistakes, citing companies that suffered from prompt injection exploits.

Key Takeaways and Resources

Jeroen concluded with a call to action: identify one key takeaway from Devoxx and pursue it. For AI, this means mastering prompt engineering while prioritizing security. He shared a website with resources on adversarial prompting and risk analysis, encouraging developers to build secure AI systems. His talk blended humor, technical depth, and ethical reflection, leaving attendees with a clear understanding of prompt injection risks and the importance of responsible AI use.

Links:

• Jeroen Egelmeers’s Website
• Open AI Custom GPT Store
• AI Act