Jonathan Lalou's Blog

Posts Tagged ‘Snowflake’

Lecturer

Santiago Giraldo serves as Senior Director of Product Marketing for Artificial Intelligence at Snowflake. With over 15 years of experience in data and AI technology, Santiago specializes in bridging business needs with advanced technical solutions, focusing on generative AI and enterprise data platforms. He holds a background from Parsons School of Design – The New School and is based in the Denver Metropolitan Area.

Brian Neumann is Senior Vice President of Engineering at Decile, an e-commerce analytics platform. Brian leads engineering efforts to develop innovative data solutions for brands, emphasizing multi-tenant architectures and AI integration to enhance customer insights.

Abstract

This presentation explores the transformative potential of Snowflake Intelligence, a generative AI-powered feature set designed to enable natural language interactions with enterprise data. Santiago introduces the core principles of Snowflake Intelligence, addressing longstanding challenges in data accessibility and decision-making velocity. Brian then details Decile’s implementation, showcasing how the platform powers Luma, a custom AI analyst that democratizes e-commerce insights across organizational roles. The discussion highlights architectural strategies, trust mechanisms, and practical outcomes, illustrating how agentic AI can shift enterprises from reactive reporting to proactive, reasoned action.

Bridging the Gap Between Business and Data Teams

Enterprises often grapple with disparities in how business users and data teams interact with information. Business stakeholders require timely, actionable insights to drive decisions, yet data teams frequently dedicate substantial effort to producing static reports or dashboards. By the time these deliverables reach decision-makers, opportunities may have diminished, as insights arrive too late for effective intervention.

Snowflake Intelligence addresses this divide by empowering users—from executives to frontline employees—to pose complex questions in natural language and receive reasoned responses. Unlike traditional tools limited to surface-level queries (e.g., “What were sales last week?”), this innovation facilitates deeper inquiry, such as identifying underlying causes or forecasting future trends. It integrates data from disparate sources, including databases, customer platforms like Salesforce, and third-party enrichments, all within a secure, governed environment.

A key advantage lies in its enterprise readiness: features are native to the Snowflake platform, ensuring robust governance, security, and data quality. This approach fosters a “reasoning partner” dynamic, where AI not only retrieves data but also provides explanatory context, enabling high-confidence decisions in real time.

Core Principles and Capabilities of Snowflake Intelligence

Snowflake Intelligence rests on three foundational pillars: deep analysis, trust, and enterprise-grade security.

Deep analysis extends beyond descriptive reporting to prescriptive and predictive reasoning. Users can explore questions like “What headwinds threaten upcoming sales?” or “How can retention be improved?” by leveraging multimodal data—structured and unstructured—across the organization. Features such as research mode enable forward-looking investigations, drawing from comprehensive knowledge sources.

Trust is paramount in generative AI adoption, where hallucinations or opaque reasoning erode confidence. Snowflake mitigates this through verified answers, full traceability to original sources, and transparent explanations. Responses include reformulated queries, step-by-step reasoning, and direct links to underlying SQL, allowing verification down to individual data points.

Enterprise readiness ensures all operations occur within Snowflake’s governed ecosystem. Dynamic discovery provides clear explanations of results, while integrations with marketplace data and enterprise tools unify insights. This holistic design transforms data utilization, placing organizational knowledge at users’ fingertips for instantaneous, reliable exploration.

Decile’s Journey: From Traditional Analytics to AI-Driven Insights

Decile operates as an e-commerce analytics platform, serving over 100 leading brands by aggregating data from sources like Shopify, Magento, marketing channels, and enrichment providers such as Acxiom. The platform creates dedicated Snowflake data warehouses per client, overlaid with application experiences for lifecycle reporting and customer segmentation.

Initially, Decile’s dashboards aimed to surpass native platform reporting by stitching disparate data for richer views. However, brand variability—ranging from retail integrations to diverse product analytics—complicated dashboard flexibility. This led to increased complexity for non-technical users, who grew reliant on customer success teams, effectively positioning Decile as an outsourced data function.

Recognizing this barrier, Decile sought to empower clients directly through an AI analyst. Early prototyping with various frameworks revealed significant hurdles: building vector stores for semantic understanding, ensuring SQL accuracy, providing visualizations, and establishing evaluation mechanisms. These challenges posed substantial investment risks for a startup.

Snowflake Intelligence emerged as an ideal solution, leveraging existing governed warehouses and dbt semantic models. Implementation involved extending dbt documentation with metadata (aliases, synonyms, sample values) to inform semantic views—YAML-defined structures describing dimensions, measures, relationships, and natural language descriptions.

Cortex Search services enhanced fuzzy matching for free-form queries, while verified queries predefined complex calculations (e.g., retention cohorts). A custom library automated provisioning of semantic views, search services, and agents via deployment pipelines.

Implementation Outcomes and Future Directions at Decile

Rapid deployment enabled pilot testing through Snowflake’s UI, gathering feedback to refine models. API access facilitated seamless integration into Decile’s application, branding the experience as Luma—a conversational AI analyst.

Users reported substantial time savings, with marketers and executives conducting analyses previously requiring extensive report stitching. Visible thinking steps—detailing semantic mappings and reasoning—built confidence, reducing perceived black-box risks. Support queries dropped 75% among adopters, as users self-served segments for activation.

Luma’s impact extends to operational efficiency: quicker market entry, reduced custom report demands, and empowered segmentation (e.g., identifying repeat purchasers for subscriptions).

Looking ahead, Decile plans per-brand instruction customization to capture nuances (e.g., subscription vendors, wholesale handling). Aspirations include user-contributed context for vertical-specific analyses and scheduled alerting for anomalies, emulating a proactive human analyst.

Implications for Enterprise AI Adoption

This collaboration exemplifies how Snowflake Intelligence lowers barriers to agentic AI in specialized domains. By providing turnkey frameworks—semantic views, APIs, and observability—platforms like Decile accelerate innovation without prohibitive development overhead.

Broader implications include democratized data access, reducing silos and delays while upholding trust through traceability. For e-commerce, this translates to agile responses to market dynamics, personalized strategies, and sustained growth.

Ultimately, such integrations signal a shift toward AI-augmented workflows, where tools complement human expertise, fostering cultures of data-driven agility and innovation.

Links:

• Lecture Video on YouTube
• Decile Company Website
• Luma AI by Decile
• Santiago Giraldo on LinkedIn

Lecturer

Timothy Spann is a Senior Solutions Engineer at Snowflake. He brings extensive experience in generative AI, large language models, Apache NiFi, Kafka, Pulsar, Flink, Spark, and related streaming and big-data technologies. Previously he held developer-advocate and field-engineering roles at Cloudera, StreamNative, Hortonworks, and other organizations. He maintains an active open-source presence and regularly publishes practical examples of NiFi processors.

Abstract

Apache NiFi provides a visual, highly configurable environment for building data-flow pipelines. Version 2.x introduces first-class support for Python processors, allowing developers to embed arbitrary Python logic—including rich libraries for machine learning, natural-language processing, and geospatial conversion—directly into streaming workflows. This article describes the architecture of Python processors, the packaging and deployment process, representative use cases ranging from image captioning to real-time transit-data conversion, and the operational advantages of running such processors inside a managed NiFi environment such as Snowflake Openflow.

NiFi Fundamentals and the Value of Python Integration

NiFi is a visual tool that lets users drag, drop, and connect processors to form directed data flows. It natively handles hundreds of sources and sinks, maintains detailed lineage and audit trails, and offers flexible error-handling and back-pressure mechanisms. Data are stored in content and attribute repositories that support interactive inspection and replay. Because NiFi already excels at integration, the addition of Python processors removes the need to re-implement sophisticated logic in Java or to off-load processing to external Spark or Flink clusters for many enrichment tasks.

A Python processor follows a simple contract. The developer supplies a class that declares dependencies, performs optional initialization, and implements a transform method. The method receives a flow-file (the unit of data moving through the pipeline) together with its attributes, may inspect or modify content, may add or alter attributes, and returns the flow-file for downstream routing. Packaging produces a NAR archive that is dropped into a NiFi extension directory or uploaded through a managed interface such as Openflow. Once loaded, the processor appears in the palette exactly like any native component and can be parameterized, scheduled, and monitored through the ordinary NiFi user interface.

Representative Processors and Demonstration Workflows

Concrete examples illustrate the range of possibilities. An RSS reader built on feedparser and pandas ingests government news feeds and emits CSV. An image-captioning processor loads a Hugging Face BLIP model, receives an image flow-file, writes a natural-language caption into an attribute, and passes the original image unchanged. Subsequent processors can apply ResNet-50 classification or NSFW detection without copying the binary content. Named-entity recognition with spaCy extracts organizations and persons from text; an OpenStreetMap geocoder converts postal addresses into latitude-longitude pairs; a GTFS-realtime converter transforms Protocol-Buffer transit feeds from the New York MTA into JSON.

In a live Openflow demonstration a GTFS processor is configured with a URL and a feed type (trip updates, vehicle positions, or alerts). Data flow through the processor, emerge as structured JSON, are split, attribute-extracted, merged, and finally loaded into Snowflake tables—all without leaving the NiFi canvas. Because processors can be started, stopped, and reconfigured while the flow remains active, developers obtain an interactive feedback loop that is difficult to replicate in batch-oriented environments.

Operational Considerations and Broader Implications

Python processors run as external processes outside the JVM; consequently they are subject to different resource constraints and are typically restricted to medium or large runtime sizes in managed offerings. Best practice therefore reserves them for tasks that genuinely benefit from the Python ecosystem—model inference, specialized parsing, or rapid prototyping—while leaving high-volume, CPU-bound work to native Java processors. Parameterization separates sensitive or environment-specific values from version-controlled flow definitions, facilitating promotion across development, test, and production clusters.

The combination of NiFi’s integration strengths with Python’s analytic libraries yields a pragmatic architecture for real-time enrichment pipelines. Unstructured data—images, archives, Protocol-Buffer streams—can be ingested, enriched with machine-learning metadata, and routed to downstream systems such as Kafka, Iceberg tables, or Slack channels. The same pattern supports prompt construction and calls to external large-language-model endpoints, positioning NiFi 2.x as a convenient orchestration layer for hybrid streaming and generative-AI workloads.

Links:

• Lecture video
• Timothy Spann GitHub speaker profile
• Snowflake Openflow Python processor guidelines

🙋‍♀️🙋‍♂️ Question

CREATE OR REPLACE TABLE sales (
    region STRING,
    amount NUMBER
);

INSERT INTO sales VALUES 
    ('North', 100),
    ('South', 200),
    ('North', NULL),
    ('East', 150);

SELECT SUM(amount) FROM sales WHERE region = 'West';

SELECT SUM(CASE WHEN amount IS NOT NULL THEN amount ELSE 0 END) AS total_sales 
FROM sales 
WHERE region = 'West'; 

SELECT COALESCE(SUM(amount), 0) AS total_sales 
FROM sales 
WHERE region = 'West';