Posts Tagged ‘PyDataGlobal2025’
[PyDataGlobal2025] Tools, Empathy, and the Craft of Building Delightful Data Experiences
Lecturer
Isabel Zimmerman is a Senior Software Engineer at Posit, PBC (formerly RStudio). She was the first full-time Python open-source hire at the company and began her tenure building MLOps packages before shifting focus to the Python experience inside interactive development environments. Her current work centers on Positron, a next-generation data-science IDE. Beyond computing she is an avid fantasy reader and bookbinder, interests that inform her view of tools as objects that can carry quiet power across generations of users.
Abstract
Every practitioner occupies a position on the continuum between tool user and tool builder. This keynote explores that continuum through the dual lenses of technical excellence and human empathy. Drawing on concrete examples from the Positron IDE and the broader open-source Python ecosystem, it articulates a set of “hard skills” (modularity, reproducibility, flexibility) and “soft skills” (knowing the user, discoverability, small improvements with large impact, and explaining one’s work). The argument is that tools become delightful only when both categories are deliberately cultivated, and that the barrier to becoming a builder has never been lower.
From Consumer to Creator: Reframing Everyday Practice
A tool is defined simply as anything that carries out a particular function. Under that definition most data scientists already build tools—whether a Git alias that corrects a habitual typo, a reusable function shared in Slack, a dashboard that informs business decisions, or a private utility that solves a personal measurement problem. The psychological barrier that prevents many practitioners from identifying as builders is therefore largely artificial. Framing the act of extraction and encapsulation as tool construction lowers that barrier and simultaneously improves personal productivity and future reproducibility.
The transition from pure consumer to occasional creator is further eased by contemporary language models. Functions that once required manual packaging can now be sketched in natural language and refined iteratively. The resulting artifacts need not be public; a private package that accelerates one’s own daily workflow is already a contribution to the wider ecosystem because it reduces friction for at least one user—oneself.
Hard Skills of Tool Design
Three technical properties form the backbone of robust tools. Modularity allows a system to grow with its users. By leaning on existing community infrastructure—FastAPI for REST endpoints, Code OSS for the editor substrate—builders can concentrate effort on the distinctive value they wish to add. The same modular surface also supplies clear extension points, encouraging specialized packages that solve narrow, high-value problems.
Reproducibility remains a foundational requirement of trustworthy science. Graphical exploration interfaces are powerful, yet they risk introducing non-reproducible click sequences. Positron’s data explorer illustrates one resolution: every filter and sort operation is internally represented so that a single button can emit executable code that recreates the identical view. The cycle of exploration is thereby closed inside a language rather than left as a sequence of manual steps.
Flexibility must be tempered by the Zen of Python’s preference for simplicity. Functions that accept an ever-expanding union of input types quickly become unmaintainable. Preferring a small number of well-defined entry points and composing them later yields systems that remain extensible without collapsing under their own complexity. Context windows supplied to language models follow the same principle: start with a carefully chosen default set of information and allow the user to add or remove context explicitly.
Soft Skills and the Human Side of Interfaces
Technical excellence alone does not produce tools that people love. Empathy for the intended user is equally decisive. Data work is characterized by iterative exploration of uncharted territory, whereas classical software engineering often constructs well-specified structures in known domains. An interface optimized solely for the latter will frustrate the former. Permanent, always-available consoles, column-aware completions, and language-server optimizations tuned to data-frame idioms are concrete expressions of that empathy.
Discoverability ensures that high-impact features do not remain secret passages. Action bars that surface “render on save,” one-click code-cell insertion, and help panes that render richly formatted docstrings bring frequently needed capabilities into immediate view. Small ergonomic improvements—running a Streamlit or Dash application with the correct launcher rather than a plain Python invocation—accumulate into large reductions in daily friction.
Finally, the act of explaining one’s work closes a vital feedback loop. Writing documentation, type annotations, or even lightweight notes in a project file forces clarity of thought. The same artifacts later serve both future collaborators and future selves. The principle “if your writing helps even one person it is worth doing, especially if that person is you” applies equally to private architectural notes and public getting-started guides.
Closing the Loop Between Building and Using
Tools improve through continuous cycles of use, observation of pain points, and iterative refinement. Feedback—whether GitHub issues, hallway conversations, or structured user testing—supplies the raw material for those cycles. Because every practitioner is simultaneously a consumer and a potential contributor, each unique perspective enriches the shared ecosystem. The mission is not the construction of a final, perfect package but the ongoing cultivation of experiences that feel beautiful, empowering, and precisely fitted to the work at hand.
Links:
[PyDataGlobal2025] Using Traditional AI and Large Language Models to Automate Complex and Critical Documents in Healthcare
Lecturer
Lily Xu is a Data Science Director in the corporate data-science and AI team at Vertex Pharmaceuticals, where she has worked for approximately seven years. She leads interdisciplinary groups of data scientists, data engineers, software engineers, and operations specialists focused on clinical-area solutions. She holds a doctorate in bioengineering from the Massachusetts Institute of Technology and an undergraduate degree from the University of California, Berkeley. Her earlier research produced publications on virtual microfluidics and the human microbiome; at Vertex she has driven projects spanning generative AI for clinical documentation, predictive patient modeling, large-scale claims analytics, protocol design, and centralized site intelligence.
Abstract
Informed consent forms constitute high-stakes, patient-facing, heavily regulated documents that must be tailored to jurisdictional requirements, local ethics boards, and plain-language standards. Their manual production across dozens of countries and hundreds of sites creates substantial operational bottlenecks in clinical-trial start-up. This article examines a production system developed at Vertex Pharmaceuticals that combines classical document-processing pipelines with large language models to auto-draft informed consent forms at scale. Emphasis is placed on architectural choices that minimize hallucination risk, rigorous measurement of end-to-end time savings, the centrality of change management, and the longer-term strategy of constructing a connected document network rather than isolated point solutions.
Clinical-Trial Operations Context and the Dual AI Portfolio
Clinical-trial operations span design, planning, execution, and monitoring phases, each generating or consuming large volumes of structured and unstructured documents. Failure to recruit patients, suboptimal site selection, or protracted regulatory review can each cost tens to hundreds of millions of dollars. Beginning in 2019 the Vertex data-strategy and solutions team—functioning as an internal SWAT unit—built trust through small, measurable pilots that combined public and private data into AI-ready assets. Over successive years the portfolio matured from ad-hoc analytics into standardized offerings for site identification, patient finding, enrollment forecasting, and, more recently, generative document automation.
The team deliberately distinguishes analytical AI (predictive modeling, Bayesian enrollment forecasts, rare-disease patient identification) from generative AI (first-draft document creation, knowledge-base chat, brand-copy generation). Business partners often approach the group believing a problem requires generative technology when structured data and classical machine learning would suffice; conversely, generative methods unlock previously intractable free-text tasks. Framing the two categories helps both data scientists and operational stakeholders select the appropriate tool. A foundational data layer aggregates site performance metrics, physician databases, claims, and census information; disease-specific analytic views and predictive models sit atop this foundation. Parallel generative pipelines extract structured content from lengthy protocols and feed downstream document generators, with embedded quality-control workflows so that extraction errors are corrected before they propagate into patient-facing material.
Architecture of the Informed-Consent-Form Auto-Drafting System
An informed consent form must convey risks, procedures, and rights in plain language while satisfying country-specific and sometimes site-specific regulatory requirements. A single multi-country trial may therefore require dozens of distinct variants. The solution developed at Vertex treats the clinical protocol as the primary source of truth, a blank regulatory template as the structural skeleton, and an approved language library as the repository of standardized phrasing.
Custom Python modules parse the protocol into logically coherent sections rather than arbitrary token chunks. Section-specific prompts and deterministic extraction routines pull the necessary facts. User-supplied answers to questions that cannot be parsed from the protocol are collected through a controlled interface. The resulting structured payload is inserted into the template; approved language snippets are retrieved via API from a purpose-built library that replaced earlier Excel spreadsheets and now maintains full audit trails and disease-area tagging.
The application is implemented in Flask and Dash, hosted on AWS behind single-sign-on, and calls a private Microsoft OpenAI endpoint for the generative steps. A monitoring dashboard continuously compares newly generated drafts against ground-truth forms produced by human experts, allowing the team to detect drift in accuracy over time. Because the generative component constitutes only a minority of the code base, the majority of engineering effort is devoted to robust parsing, template management, and workflow orchestration—skills that remain essential even as language models improve.
The design philosophy is “AI in the human loop” rather than “human in the AI loop.” Regulatory and patient-safety constraints demand that every draft undergo expert review; the system’s value lies in accelerating the initial drafting phase so that reviewers begin from a high-quality baseline rather than a blank page.
Measuring Impact, Change Management, and Scaling Strategy
Early controlled experiments compared pure manual drafting (one to three hours depending on trial complexity) with auto-draft generation (under ten minutes). Drafting-time reduction approached 90 percent. When subsequent editing and quality-control effort was included, net end-to-end time savings settled near 40 percent—still substantial given the volume of forms required across a growing portfolio. Because operational teams are chronically time-constrained, such measurements were performed on only two trials; the results nevertheless provided the quantitative foundation for continued investment.
Technology alone does not guarantee adoption. Change-management activities therefore received equal attention: standardization of templates and language libraries, transparent communication of model assumptions and known failure modes, and staged training that enabled business users to generate drafts independently. Treating free-text language assets with the same governance rigor applied to numerical data proved essential.
The longer-term vision is a connected document network rather than a collection of isolated point solutions. Clinical protocols and clinical study reports function as central hubs; mapping the full input–output relationships among start-up documents reveals opportunities for shared extraction components and cascading automation. The same platform is being extended to site budgets, case-report-form specifications, training materials, and other protocol-derived artifacts. Country-level templates are already linked so that a single protocol upload can spawn multiple jurisdiction-specific drafts simultaneously. Site-level customization remains outside the automated scope because the return on investment diminishes rapidly at that granularity; country-level guidance is instead provided to local teams.
Broader Lessons for Generative Applications in Regulated Environments
Several observations travel beyond the specific use case. First, impact measurement must be designed from the outset; without side-by-side timing studies and accuracy tracking, claims of productivity gain remain anecdotal. Second, the majority of engineering effort in production document systems continues to reside in classical software and data-engineering practices; large language models occupy a focused niche once reliable extraction and templating are in place. Third, alignment with business ownership is decisive: projects lacking motivated operational sponsors are deferred in favor of those with clear accountability and enthusiasm. Finally, the cumulative benefit of a systematically constructed document network can outweigh the initial development cost provided the organization persists past the early pilots.
Ambient listening, internal retrieval-augmented generation over institutional knowledge bases, and protocol optimization via real-world data are complementary initiatives already underway at Vertex and peer organizations. Collectively they illustrate a measured trajectory in which generative and analytical methods remove routine cognitive load while leaving critical reasoning and final accountability with domain experts.
Links:
[PyDataGlobal2025] What’s Next in AI for Data and Data Management
Lecturer
Lisa Amini is a Distinguished Engineer at IBM and Director of Data & AI Platforms Research, where she also leads IBM’s AI Horizons Network. Her career at IBM Research spans more than two decades and includes foundational work on stream processing systems that became the InfoSphere Streams product, leadership of the IBM Research laboratory in Ireland, and earlier roles directing knowledge and reasoning research. She has guided interdisciplinary efforts across cloud computing, artificial intelligence, and quantum computing, always with an emphasis on technologies that can be deployed at enterprise scale.
Abstract
Recent advances in large language models have catalyzed a wave of AI-assisted tools for data management and operations, ranging from code-generation assistants for data-flow pipelines to retrieval-augmented generation systems and increasingly autonomous data agents. This keynote examines the rapid evolution of generative and agentic capabilities, situates them within the broader data-management stack, and explores both near-term practical applications and longer-horizon research directions. Particular attention is given to the shift from human-operated systems augmented by copilots toward semi-autonomous stacks in which agents design, optimize, remediate, and continuously evaluate data products. The discussion balances technical opportunity with the enduring requirements of price-performance, open-source interoperability, and hybrid data architectures.
The Accelerating Capability Curve and the Emergence of Agency
The pace at which machine-learning benchmarks reach human-level performance has changed dramatically. Tasks that once required decades of incremental progress—handwriting recognition, for example—now reach parity within a few years or even months. Reading comprehension and predictive reasoning benchmarks follow similarly steep trajectories. While these evaluations remain narrow and do not constitute artificial general intelligence, they illustrate an unprecedented rate of improvement. Simultaneously, the cost per inference continues to fall even as model size and training compute grow, a trend driven by better systems design and algorithmic efficiency.
Within this landscape the progression from predictive models to generative models to conversational systems and finally to agents marks a qualitative shift. Agents do not merely answer questions; they dynamically control application flow, make decisions, take actions, and attempt self-correction. In the data domain this agency opens the possibility of systems that no longer wait for humans to formulate every query or repair every broken pipeline. Instead, agents can probe schema, resolve ambiguity, hypothesize data products, evaluate their own output, and iterate.
Transforming the Data Landscape and the Complementary Task Stack
Unstructured data has long existed, yet only recently has it assumed central importance. Machines can now reason over images, generate multimodal content, and extract structured signals from free text at scale. Classical database, warehouse, and lakehouse architectures, optimized primarily for structured tables, must therefore accommodate new access patterns. Retrieval-augmented generation pipelines replace static queries with dynamic retrieval-plus-generation cycles. User interaction moves from fixed application-generated SQL toward speculative, multi-step agent dialogues that probe metadata, formulate candidate queries, and refine them in light of intermediate results.
A useful conceptual inversion is to view the traditional storage–compute–query stack alongside a complementary human-task stack: infrastructure design, workload optimization, data discovery, enrichment, flow creation, remediation, governance, and insight generation. Each of these human activities constitutes fertile ground for agentic automation. Early systems already demonstrate learnable components inside query optimizers and routers; more ambitious research explores whether agents can search the design space of kernel-level software itself.
From Automation to Autonomy: Data Products and Continuous Evaluation
The practical goal is not merely to accelerate individual steps but to move entire workflows from human-operated to human-supervised. Consider the request to stand up a data stack and associated data products for a new application—robo-trading, for instance, that must combine public market data with sentiment signals and support periodic rebalancing. A multi-agent system can be tasked with discovering relevant sources, hypothesizing an ideal schema, populating that schema from heterogeneous tables and documents, extracting structured fields from natural-language text, and packaging the result as a governed data product.
Critical to autonomy is the ability to evaluate quality without constant human intervention. One effective strategy generates natural-language questions that a domain expert would plausibly ask of the intended data product, translates those questions into executable queries, and then monitors coverage metrics (tables and columns touched), topic coverage, query complexity, and latency. Agents iterate—adding sources, refining transformations, simplifying views—until the metrics stabilize within acceptable bounds or progress plateaus and human guidance is required. The same loop can later serve as continuous monitoring: questions that once succeeded can be re-executed to detect drift or regression.
Similar patterns apply to operational remediation. When a data-flow pipeline fails, agents can examine logs, generate natural-language root-cause hypotheses, propose script repairs, and, under appropriate guardrails, test those repairs in a sandbox before presenting them for approval. Across the spectrum of use, build, and optimize activities, the user’s role gradually shifts from operator to approver or observer.
Enduring Constraints and the Research Horizon
Price-performance remains non-negotiable; open-source components continue to enable rapid composition of storage formats, query engines, and table formats; hybrid architectures that span on-premises, cloud, and edge locations persist. Benchmarks, data contracts, open lineage standards, and carefully scoped open-weight models supply the interfaces and evaluation harnesses that allow agents to interoperate safely. Research prototypes already explore operator libraries that let developers request high-level transformations while large language models synthesize the concrete implementations behind the scenes.
The path forward is incremental. Fully autonomous data stacks will not appear overnight. Yet the combination of generative models, agent frameworks, and rigorous evaluation loops is already moving concrete workloads—data-product curation, flow repair, insight generation—along the continuum from assistance toward autonomy. The opportunity for data scientists and engineers is to shape the metrics, tools, and governance practices that will keep these systems both powerful and trustworthy.
Links:
[PyDataGlobal2025] Where Have All the Metrics Gone? Evaluating Generative Systems in a Multi-Dimensional Era of Error
Lecturer
Dr. Rebecca Bilbro is a data scientist and co-creator of the Yellowbrick library, an open-source diagnostic visualization toolkit that extends the scikit-learn and Matplotlib APIs. She has taught machine learning at Georgetown University and continues to work at the intersection of model interpretation, visual analytics, and the practical evaluation of contemporary AI systems. Her earlier work focused on making the model-selection process more transparent through visual diagnostics; the present discussion extends that concern into the generative era.
Abstract
Classical machine-learning metrics—F1 score, mean squared error, area under the ROC curve—once supplied a comforting scalar signal of progress and a shared vocabulary for model comparison. In the generative era those metrics have largely receded from daily practice. Large language models can fail simultaneously along semantic, stylistic, structural, behavioral, and temporal dimensions, rendering single-axis notions of accuracy ill-defined. This article enumerates recurring failure modes observed in production generative systems, argues for an experimental rather than assumptive framing of new projects, advocates systematic task decomposition as a prerequisite for meaningful measurement, and illustrates how lightweight, purpose-built metrics can restore visibility, accountability, and iterative improvement.
The Retreat of Scalar Certainty and the New Landscape of Error
Traditional supervised learning treated error as a one-dimensional discrepancy between a model’s prediction and a ground-truth label. Diagnostic visualizations such as residual plots for regression and confusion matrices for classification made that discrepancy tangible and actionable, guiding feature engineering, model selection, and hyperparameter search. The resulting workflow was comparatively linear: obtain data, explore, select and tune models, optimize a scalar metric, serialize the artifact, and await the next batch of data.
Generative models invert many of these assumptions. Practitioners increasingly consume foundation models rather than train them from scratch, and therefore inhabit the application side of the former training–deployment boundary. On that side, error is higher-dimensional and less crisply defined. An output may be factually incorrect, stylistically inappropriate, structurally malformed, behaviorally unbounded, or temporally outdated—frequently several of these at once. The feedback that arrives is no longer a number but a qualitative complaint: the system hallucinated, the tone is wrong, the answer is inconsistent, the result is too vague, or the model performed an action it should never have attempted. Even the word “accuracy” has become ambiguous; it may refer to mathematical correctness, to perceived alignment with user intent, or simply to the subjective impression that the output “feels right.”
A Working Taxonomy of Generative Failure Modes
Several distinct modes of failure recur with sufficient regularity to merit explicit names and separate measurement strategies. Domain failure occurs when an output employs correct jargon and plausible structure yet contains subtle inaccuracies detectable only by a genuine subject-matter expert. The classic illustration is an authoritative but incorrect set of instructions for a skilled manual task; only embodied expertise reveals the error. Form failure arises when content is otherwise acceptable but the required schema—JSON, HTML, a particular report template—is violated, breaking downstream automated systems. Mode collapse, familiar from the literature on generative adversarial networks, appears when synthetic data or requested stylistic variants exhibit insufficient diversity, collapsing into high-probability phrasings and structures. Consistency failure manifests as contradictory answers to essentially identical prompts issued on different occasions or with minor rephrasing. Boundary failure describes the model performing tasks outside its intended scope, such as a narrowly purposed customer-service agent solving advanced mathematical problems. Temporal failure reflects the model’s blindness to the passage of time, leading it to recommend deprecated APIs, outdated function signatures, or facts that have been superseded.
Collectively these modes demonstrate that a single scalar metric cannot capture the relevant notions of quality. Each mode points toward a different remediation strategy—retrieval grounding, schema enforcement, diversity sampling, consistency regularization, capability gating, or temporal knowledge injection—and therefore demands its own measurement instrument.
Experimental Framing, Task Decomposition, and the Rejection of the Mega-Prompt
Projects that begin with the declarative sentence “We are building an agent that can \ldots” implicitly treat the desired capability as already achieved and thereby skip the experimental design necessary to surface failure. Reframing the same ambition as “We want to test whether an agent can \ldots” converts the effort into a set of measurable hypotheses and observable failure modes. Because different applications are vulnerable to different subsets of the taxonomy, practitioners are advised to select two to four high-stakes modes at the outset rather than attempt exhaustive coverage. A system that emits structured records will prioritize form failure; a research assistant will prioritize hallucinated citations; a regulated healthcare application will prioritize boundary violations and leakage of protected information.
Once the relevant modes are chosen, the monolithic “mega-prompt” that attempts to solve an entire workflow inside a single generation becomes counterproductive. Any failure is global and unlocalizable; root-cause analysis is nearly impossible. Systematic task decomposition restores visibility. Retrieval, fact-checking, synthesis, formatting, and validation become separate stages, each with bounded inputs, explicit success criteria, and its own characteristic failure signature. The resulting pipeline resembles classical extract–transform–load systems or production machine-learning workflows and permits the same style of stepwise debugging and incremental improvement.
Constructing Intentional Measurement Strategies
Measurement itself must be reinvented for the generative setting. Binary, machine-checkable criteria—valid HTML, schema compliance enforced at generation time by libraries such as Pydantic or Outlines—can eliminate entire classes of form failure before they reach downstream consumers. Where domain expertise remains indispensable, simple Likert-scale ratings collected from subject-matter experts convert qualitative impressions into trackable numeric signals that can be monitored over successive iterations. Consistency can be quantified by generating multiple outputs from an identical prompt and computing average pairwise similarity in embedding space, yielding an interpretable score between zero and one. None of these metrics is universal or theoretically privileged; each is an intentional response to a previously identified failure mode and is valuable precisely because it is tailored.
The open-source ecosystem already supplies many of the required building blocks. Parameterized testing frameworks allow systematic variation of prompts and automatic checking of outputs. Structural validation libraries constrain generation. Tracing and logging harnesses provide the observability needed to locate failures inside a decomposed pipeline. The remaining work is largely cultural: the willingness to define operational notions of “good” rather than to inherit a default scalar from an earlier paradigm, and the humility to treat every new agent as an experiment whose failure modes must be anticipated, isolated, and measured.
Links:
[PyDataGlobal2025] The Lifecycle of a Jupyter Environment: From Exploratory Notebook to Production Pipeline
Lecturer
Dawn Wages is Director of Community and Developer Relations at Anaconda. She brings a background that spans business education, software development, and sustained open-source community work within the Python Software Foundation, NumFOCUS, and SciPy ecosystems. Her professional focus includes developer advocacy, packaging sustainability, and the practical maturation of data-science workflows from initial exploration to reliable production systems.
Abstract
Most machine-learning and data-science projects begin life as a Jupyter notebook—an interactive space for curiosity-driven exploration and rapid prototyping. The transition from that exploratory artifact to a reliable, scheduled, and maintainable pipeline introduces a series of engineering, organizational, and infrastructural challenges. This article traces the full lifecycle: the establishment of clear objectives and documentation practices, the modularization of notebook logic into reusable and testable components, the selection of appropriate tooling matched to concrete workflow needs, the maintenance of reproducible computational environments, and the deployment of resilient production systems. Emphasis is placed on domain-driven design principles, established software-engineering patterns, and the complementary roles of notebooks, scripts, configuration files, and managed cloud platforms.
Establishing Objectives, Documentation, and Shared Language
Projects that begin with solitary tinkering frequently carry forward unspoken assumptions that later prove costly to reverse. A brief but structured kickoff conversation that surfaces domain expertise, distinguishes desired outcomes from tangible outputs, and establishes a clear matrix of responsibilities (responsible, accountable, consulted, informed) can prevent weeks of misdirected effort. Documentation is treated not as an afterthought but as a primary project artifact; code follows conversation rather than the reverse. Incremental milestones are framed as opportunities for collective recognition rather than mere accountability checkpoints, fostering a collaborative rather than adversarial atmosphere.
Domain-driven design supplies a particularly useful vocabulary for this stage. Variable names, module boundaries, data contracts, and even file-system organization should reflect the language of the subject-matter experts rather than the transient notational convenience of the analyst. When nomenclature diverges from domain concepts, the mismatch itself becomes diagnostic of incomplete understanding and signals the need for further dialogue. Early attention to platform constraints and resource limits also surfaces at this stage, allowing teams to anticipate hardware, cost, and scalability considerations before architectural commitments harden.
Modularization, Architectural Patterns, and the Separation of Concerns
Once objectives stabilize, the notebook is systematically decomposed. Reusable fragments of logic are extracted into pure functions that possess explicit inputs and outputs; these functions then migrate into ordinary Python modules, shell scripts, or declarative configuration files. The notebook itself shrinks to a thin orchestration layer that imports and invokes the modular components. With clear boundaries in place, unit tests become feasible, and the chronic difficulty of knowing which cells must be executed in which order largely disappears.
Two illustrative patterns recur across successful transitions. A builder-style class for an ETL pipeline accumulates ordered steps—extraction, validation, cleaning, transformation, feature engineering—and executes them in sequence, providing a readable and extensible scaffold. Training and evaluation logic is likewise encapsulated in dedicated classes that accept data, produce fitted models, perform cross-validation, and return quantitative comparisons. Both patterns draw on established software-architecture literature and on mature libraries such as scikit-learn, allowing practitioners to leverage battle-tested abstractions rather than reinventing core functionality. The resulting structure supports maintainability, testability, and eventual scaling while preserving the interactive character of the original exploratory work.
Tool Selection, Environment Reproducibility, and Hardware Considerations
No single tooling stack is universally optimal; the decisive criterion is fit to the concrete workflow rather than current popularity. Papermill enables parameterized execution of notebooks, supporting batch reporting, systematic variation of data sets, and lightweight A/B testing without abandoning the notebook paradigm. MLflow supplies experiment tracking, model versioning, and a lightweight registry, reducing the risk that promising configurations are lost. Managed platforms such as Snowflake, Amazon SageMaker, or Azure reduce the operational burden of infrastructure provisioning while introducing cost-visibility dashboards that help prevent unexpected expenditure.
Environment reproducibility remains a persistent and under-appreciated difficulty. The Python packaging ecosystem continues to evolve; initiatives such as wheel-next seek to improve the handling of system-level libraries that pip alone cannot reliably manage. Project-local environment managers keep dependencies co-located with source code and thereby improve portability, while global environments remain useful for shared tooling. GPU-accelerated libraries such as RAPIDS can accelerate familiar pandas-style workflows without requiring code changes, provided the underlying hardware is available—either on local machines or through cloud providers that expose appropriate accelerators. Binary dependencies and conflicts among system libraries continue to demand careful attention, especially when multiple packages link against incompatible versions of the same underlying C or C++ library.
Deployment Practices, Resilience, and Closing the Feedback Loop
Production systems require automated testing, staged rollbacks, health checks, and continuous monitoring. Idempotent pipeline steps, retry logic protected by rate limiting or load shedding, and feature flags reduce the blast radius of individual failures. Logging, metrics, and alerting—standard offerings of major cloud providers—close the observational feedback loop. Pipeline design must simultaneously consider task complexity, hardware constraints, collaborative experimentation needs, developer-tooling preferences, and the requirements of downstream applications. A concise reference checklist covering these dimensions proves valuable at the start of each new project.
Interactive visualization layers—PyScript for in-browser Python execution, Voilà, Panel, and the HoloViz ecosystem—extend the lifecycle beyond batch pipelines into stakeholder-facing dashboards. In this way the original notebook, once a private exploratory artifact, becomes the seed of a living, shareable system that supports both scheduled production runs and ad-hoc investigation.
Links:
[PyDataGlobal2025] The SAT Math Gap: Gender Difference or Selection Bias? A Bayesian Investigation
Lecturer
Allen Downey is the author of several widely used books on Bayesian statistics and computational methods, including Think Bayes. He collaborates with PyMC Labs, a consulting group that maintains the PyMC package for Bayesian modeling in Python, and has worked with the American Institute for Boys and Men on questions related to educational outcomes and selection effects. His work emphasizes practical, computational approaches to statistical inference that remain accessible to practitioners familiar with Python and common probability distributions.
Abstract
Male test-takers have scored approximately thirty points higher than female test-takers on the mathematics section of the SAT for roughly five decades on a scale ranging from 200 to 800. Some commentators interpret the persistent gap as evidence of a genuine difference in mathematical ability; others attribute it primarily to differential selection into the test-taking population. This article presents a complete Bayesian hierarchical model that recovers latent distributions of efficacy and estimates selection probabilities as a function of that efficacy. The model accounts for score truncation at the extremes, year-specific scoring artifacts arising from the discrete combinatorics of the examination, and group differences. Validation proceeds through parameter-recovery experiments on synthetic data and posterior predictive checks on the observed histograms. Results indicate that selection bias can plausibly explain the majority of the observed gap, offering a quantitative framework for disentangling latent traits from observed outcomes.
Selection Bias as a Fundamental Challenge and the Poincaré Illustration
Selection bias ranks among the most intractable difficulties in observational research. By the time data have been collected under an unknown or only partially known sampling process, correction is frequently impossible. In favorable cases, however, the shape of the observed distribution retains a detectable signature of the underlying selection mechanism. The classic, if almost certainly apocryphal, story of the mathematician Henri Poincaré and the baker supplies an intuitive illustration. Poincaré allegedly weighed a loaf of bread every day for a year and found a mean of 950 grams rather than the legally required 1 000 grams. After a warning from the authorities the following year’s mean rose to exactly 1 000 grams, yet the shape of the distribution was no longer Gaussian. Instead it resembled the distribution that would arise if the baker systematically selected the heaviest of several loaves whenever Poincaré appeared. A Bayesian model that treats the number of loaves examined as a latent parameter can recover both the true mean weight of the bakery’s output and the intensity of the selection process from a single year of daily observations.
The same principle is applied to SAT mathematics scores. Roughly half of college-bound seniors in the United States take the examination; those who anticipate low performance are less likely to sit for it. If the propensity to take the test rises with underlying mathematical efficacy—a composite of aptitude, educational history, and test-taking acumen—the observed score distributions for the two gender groups will be differentially truncated. An apparent mean difference can therefore emerge even if the latent distributions of efficacy are nearly identical. The question is whether the observed histograms contain sufficient information to recover both the latent distributions and the selection functions.
Data Characteristics and Generative Model Construction
College Board reports supply the full frequency distributions of scores for male and female test-takers for the years 2011 through 2015. Scores range from 200 to 800 and are scaled so that the population mean typically lies near 500 and the standard deviation near 100. In a sample of several hundred thousand observations one would expect an approximately Gaussian shape, yet the empirical histograms exhibit two distinctive features. First, excess mass appears at the extreme scores of 200 and 800 because all latent efficacy values lying outside the observable range are folded into those endpoint bins. Second, pronounced year-specific spikes arise from the discrete combinatorics of a fifty-four-question examination that awards +1 for a correct answer, 0 for an omitted answer, and –0.25 for an incorrect answer. Certain raw-score totals are impossible; others are more or less probable depending on the relative difficulty of the items in a given year. These spikes are highly consistent between male and female test-takers within a single year yet vary markedly from year to year, providing an additional source of identifying variation.
The generative model therefore comprises three principal components. A latent Gaussian distribution of efficacy is characterized by mean (\mu) and standard deviation (\sigma). A logistic selection function whose slope (\beta) governs the probability of taking the test as a function of efficacy captures the differential participation. A zero-mean, detrended Gaussian random walk absorbs the year-specific scoring spikes without contaminating estimates of location or selection intensity. Truncation is handled by extending the support of the latent distribution beyond the observed range and folding the tail probabilities into the endpoint bins. Careful constraints are required to avoid non-identifiability: the random walk is forced to have mean zero and is detrended so that it does not compete with either the intercept of the efficacy distribution or the slope of the selection function.
Hierarchical Estimation, Validation, and Substantive Interpretation
A hierarchical specification pools information across the five years while allowing group-specific selection slopes. Location and scale parameters are constrained to be nearly constant over time; the scoring artifacts are constrained to be identical across gender within each year; the selection slopes remain unconstrained. Under this specification the estimated selection gradient for male test-takers is substantially steeper than that for female test-takers. The implied probability of taking the test rises from roughly 25 percent at the low end of the efficacy distribution to 75 percent at the high end for males, while the corresponding range for females is far narrower. The posterior standard deviation of efficacy is also higher for males, consistent with a broader pattern of greater male variability observed in many cognitive and educational measures.
The posterior means of latent efficacy are 508 for males and 502 for females—a difference of only six points. Given that the observed score gap is approximately thirty points, selection bias can account for as much as four-fifths of the disparity. Parameter-recovery experiments on synthetic data generated from known values confirm that the model can retrieve (\mu), (\sigma), and (\beta) with acceptable accuracy. Posterior predictive checks on the real histograms show that the generative process reproduces both the overall shape and the fine-scale spikes of the observed data.
The result remains sensitive to the modeling assumption that deviations from normality are produced by selection rather than by intrinsic non-Gaussianity of the male efficacy distribution. Nevertheless, the analysis demonstrates that a substantial selection effect is statistically plausible and supplies a quantitative language for discussing the relative contributions of latent differences and participation patterns. The broader methodological lesson is that carefully specified generative models can sometimes extract selection parameters from the very distributions that selection has distorted. When sample sizes are large and the data-generating process can be articulated with reasonable fidelity, Bayesian inference offers a principled route from observed histograms to latent traits and selection intensities, moving the conversation beyond simple affirmation or denial of group differences.
Links:
[PyDataGlobal2025] Enhancing Apache NiFi 2.x with Python Processors
Lecturer
Timothy Spann is a Senior Solutions Engineer at Snowflake. He brings extensive experience in generative AI, large language models, Apache NiFi, Kafka, Pulsar, Flink, Spark, and related streaming and big-data technologies. Previously he held developer-advocate and field-engineering roles at Cloudera, StreamNative, Hortonworks, and other organizations. He maintains an active open-source presence and regularly publishes practical examples of NiFi processors.
Abstract
Apache NiFi provides a visual, highly configurable environment for building data-flow pipelines. Version 2.x introduces first-class support for Python processors, allowing developers to embed arbitrary Python logic—including rich libraries for machine learning, natural-language processing, and geospatial conversion—directly into streaming workflows. This article describes the architecture of Python processors, the packaging and deployment process, representative use cases ranging from image captioning to real-time transit-data conversion, and the operational advantages of running such processors inside a managed NiFi environment such as Snowflake Openflow.
NiFi Fundamentals and the Value of Python Integration
NiFi is a visual tool that lets users drag, drop, and connect processors to form directed data flows. It natively handles hundreds of sources and sinks, maintains detailed lineage and audit trails, and offers flexible error-handling and back-pressure mechanisms. Data are stored in content and attribute repositories that support interactive inspection and replay. Because NiFi already excels at integration, the addition of Python processors removes the need to re-implement sophisticated logic in Java or to off-load processing to external Spark or Flink clusters for many enrichment tasks.
A Python processor follows a simple contract. The developer supplies a class that declares dependencies, performs optional initialization, and implements a transform method. The method receives a flow-file (the unit of data moving through the pipeline) together with its attributes, may inspect or modify content, may add or alter attributes, and returns the flow-file for downstream routing. Packaging produces a NAR archive that is dropped into a NiFi extension directory or uploaded through a managed interface such as Openflow. Once loaded, the processor appears in the palette exactly like any native component and can be parameterized, scheduled, and monitored through the ordinary NiFi user interface.
Representative Processors and Demonstration Workflows
Concrete examples illustrate the range of possibilities. An RSS reader built on feedparser and pandas ingests government news feeds and emits CSV. An image-captioning processor loads a Hugging Face BLIP model, receives an image flow-file, writes a natural-language caption into an attribute, and passes the original image unchanged. Subsequent processors can apply ResNet-50 classification or NSFW detection without copying the binary content. Named-entity recognition with spaCy extracts organizations and persons from text; an OpenStreetMap geocoder converts postal addresses into latitude-longitude pairs; a GTFS-realtime converter transforms Protocol-Buffer transit feeds from the New York MTA into JSON.
In a live Openflow demonstration a GTFS processor is configured with a URL and a feed type (trip updates, vehicle positions, or alerts). Data flow through the processor, emerge as structured JSON, are split, attribute-extracted, merged, and finally loaded into Snowflake tables—all without leaving the NiFi canvas. Because processors can be started, stopped, and reconfigured while the flow remains active, developers obtain an interactive feedback loop that is difficult to replicate in batch-oriented environments.
Operational Considerations and Broader Implications
Python processors run as external processes outside the JVM; consequently they are subject to different resource constraints and are typically restricted to medium or large runtime sizes in managed offerings. Best practice therefore reserves them for tasks that genuinely benefit from the Python ecosystem—model inference, specialized parsing, or rapid prototyping—while leaving high-volume, CPU-bound work to native Java processors. Parameterization separates sensitive or environment-specific values from version-controlled flow definitions, facilitating promotion across development, test, and production clusters.
The combination of NiFi’s integration strengths with Python’s analytic libraries yields a pragmatic architecture for real-time enrichment pipelines. Unstructured data—images, archives, Protocol-Buffer streams—can be ingested, enriched with machine-learning metadata, and routed to downstream systems such as Kafka, Iceberg tables, or Slack channels. The same pattern supports prompt construction and calls to external large-language-model endpoints, positioning NiFi 2.x as a convenient orchestration layer for hybrid streaming and generative-AI workloads.
Links:
[PyDataGlobal2025] Lessons in Decision Making from the Monty Hall Problem
Lecturer
Eyal Kazin is an ex-cosmologist turned data scientist with two decades of experience solving complex quantitative problems. Based in London, he applies statistical and causal inference, machine learning, and optimization methods to produce practical, communicable insights for organizational decision makers. He regularly publishes accessible articles on applied statistics and machine learning on Medium.
Abstract
The Monty Hall problem, popularized by a television game show and later by a magazine column, continues to serve as a rich source of insight into probabilistic reasoning. This article reconstructs the classic three-door formulation, contrasts frequentist and Bayesian analyses, quantifies the informational value of the host’s actions through the notion of surprisal, and extracts a series of decision-making lessons relevant to data science practice. Qualitative and quantitative criteria for recognizing genuine real-world analogues are examined, together with cautionary observations about over-simplification and the limits of analogy.
The Classic Problem and Competing Intuitions
A contestant faces three doors; a prize lies behind one and undesirable objects behind the others. After the contestant selects a door, the host—who knows the prize location—opens a different door that does not conceal the prize and offers the contestant the chance to switch to the remaining closed door. The question is whether switching improves the probability of winning.
A frequentist intuition notes that only two doors remain and therefore assigns equal probability to each, concluding that switching is irrelevant. A Bayesian analysis begins with a uniform prior of one-third on each door. After the contestant’s initial choice, the complementary probability mass of two-thirds resides on the other two doors collectively. When the host reveals one of those doors to be empty, the entire two-thirds mass concentrates on the single remaining unchosen door. Switching therefore doubles the probability of success from one-third to two-thirds.
The same logic becomes intuitively transparent when the number of doors is increased to one hundred or one million. The contestant’s initial selection retains probability (1/n); after the host opens all but one of the remaining doors, the complementary mass ((n-1)/n) resides on the single unopened alternative. The dramatic disparity renders the advantage of switching obvious. Information-theoretic surprisal formalizes the difference: the host’s action conveys progressively more information as (n) grows, widening the gap between the surprisal of the originally chosen door and that of the residual alternative.
Decision-Making Lessons for Data Practice
Several transferable lessons emerge. First, probability assessment is frequently counter-intuitive; Kahneman’s distinction between fast, intuitive System 1 thinking and slower, deliberative System 2 thinking maps directly onto the frequentist versus Bayesian treatments of the problem. Second, ambiguity in problem statements must be resolved explicitly; the host’s policy of never opening the prize door is rarely stated yet is essential to the correct solution. Third, new information should update beliefs rather than be discarded. Fourth, well-designed visualizations—such as exhaustive enumeration of the six equally likely game configurations—can convert an abstract argument into an immediately graspable demonstration. Fifth, multiple solution paths exist; Marilyn vos Savant’s public defense of the Bayesian answer, despite initial resistance from many professional mathematicians, illustrates both the value of alternative representations and the necessity of intellectual humility.
Simulations, while powerful for complex systems, are unnecessary when an exact analytic solution is available; the Monty Hall problem can be solved with pen and paper. Useful analogies should be sought when a problem appears intractable, yet the trade-off between simplification and oversimplification must be kept in view. Occam’s razor, Menken’s warning against excessive simplicity, and Box’s maxim that all models are wrong but some are useful together supply a balanced heuristic.
Real-World Analogues and the Limits of Analogy
Genuine applications must reproduce both the qualitative dependence structure and the quantitative conditional probabilities of the original problem. The host’s choice depends on both the contestant’s selection and the prize location; the two latter variables are independent of each other. Candidate analogies such as short-listing job applicants or adaptive tutoring succeed only when the “host” possesses complete knowledge of the target and the learner has essentially no prior information. Superficial similarities—three cups in a magic trick, for example—satisfy the formal conditions yet remain of limited practical interest. Causal graphs make the required independence and dependence relations explicit and thereby guard against misleading parallels.
Links:
[PyDataGlobal2025] Reviving Survival Analysis: Timeless Concepts for Modern Prediction Problems
Lecturer
Malte Tichy works in data analytics and artificial intelligence at Zean Zeusa. His professional focus includes statistical modeling of time-to-event phenomena. He has contributed scholarly work on the Kaplan–Meier estimator and related topics in survival analysis, reflecting a sustained interest in both classical methods and their contemporary applications.
Abstract
Survival analysis addresses the fundamental question of when an event of interest will occur. Although the field originated centuries ago in demography and has long been central to medicine and reliability engineering, it remains under-represented in many data-science communities. This article reviews the core statistical formulation of the survival problem, the role of the Weibull distribution and competing risks, the distinctive challenge of censoring, the Kaplan–Meier estimator, likelihood-based and Bayesian fitting procedures, and the integration of modern machine-learning techniques that enable individualized predictions. Counter-intuitive phenomena that arise under censoring are examined, underscoring why survival analysis constitutes a prediction task in its own right rather than a simple blend of regression and classification.
Historical Roots and the Statistical Survival Problem
The systematic study of lifetimes can be traced to seventeenth-century English bills of mortality. In 1665, during a severe plague week, records showed thousands of deaths against a handful of births; such data permitted crude, age-independent forecasts of remaining life. By 1693 Edmund Halley had constructed the first age-dependent life tables, motivated by the practical need of ecclesiastical institutions to set sustainable pensions for clergy and widows. These early efforts already contained the essential ingredients of modern survival analysis: a population of units, observation of event times, and the requirement to plan for future failures or departures.
In contemporary language the problem is stated as follows. A unit—whether a mechanical component, a patient, an employee, a package in transit, or a radioactive nucleus—is observed from a defined starting time until it experiences a terminal event or is removed from observation. The analyst seeks the expected remaining lifetime and, more generally, the entire distribution of event times. Because organizations must provision spare parts, schedule maintenance, or forecast attrition, both reactive planning and proactive improvement of the underlying process become possible once reliable lifetime models are available.
The empirical foundation is the collection of observed failure ages (t_1, t_2, \dots). For each unit the individual cumulative distribution function is a step that jumps from zero to one at the observed failure time. Summing these steps yields the empirical cumulative failure probability of the population. From this curve one can read, for example, that 14 percent of items have failed by five years and 64 percent by ten years; the curve need not converge to unity if a positive fraction of the population never experiences the event.
Parametric Models, Competing Risks, and the Bathtub Curve
The simplest parametric assumption is that the instantaneous failure probability is constant. This memoryless property produces the exponential distribution, whose cumulative form is (1 – e^{-\tau/\theta}). Many real processes—radioactive decay, inter-arrival times in a homogeneous Poisson process, waiting times in a randomly assigned call center—approximate this ideal. When the constant-hazard assumption is untenable, a shape parameter (\beta) is introduced, yielding the Weibull family. Values of (\beta < 1) generate early-life or infant-mortality behavior; values of (\beta > 1) generate aging or wear-out behavior.
Human mortality data from the United States in 2023 illustrate that a single Weibull component is insufficient: mortality is elevated in infancy, reaches a minimum in childhood, and then rises steadily with age. Competing risks are therefore combined by multiplying the individual survival functions and taking the complement. The resulting hazard function frequently exhibits the classic bathtub shape—high at the extremes and low in the middle—which appears across domains ranging from software reliability and electronic burn-in to the structural integrity of buildings and aircraft.
Censoring is the distinguishing practical difficulty. Units enter observation at staggered times and many remain event-free at the close of the study. The Kaplan–Meier estimator, introduced in 1958, constructs a non-parametric cumulative distribution by redistributing the probability mass of censored observations onto later observed failures. Implementations are readily available in the Python package lifelines and the R package survival. When parametric models are fitted, the likelihood contribution of a censored observation is the survival function evaluated at the censoring time rather than the density at a failure time. Bayesian analyses make the dual sources of uncertainty—sample size and degree of censoring—explicit; posterior uncertainty bands widen dramatically under heavy censoring even when the number of units is moderate.
Machine Learning and Individualized Forecasts
Classical methods yield population-level curves. Contemporary practice demands individualized trajectories that incorporate covariates such as operating conditions, maintenance history, or patient characteristics. Two complementary routes exist. Survival-specific algorithms—survival trees and random survival forests—embed the censoring mechanism directly inside the learning procedure. Alternatively, any off-the-shelf regressor may be trained on a suitably transformed target and the resulting predictions mapped back into the survival domain. Either approach permits forecasts that vary continuously with age and with a rich set of covariates, far surpassing the single number offered by seventeenth-century life tables.
Surprising phenomena appear once censoring is taken seriously. Under certain model assumptions the arrival of an early failure can lower the predicted cumulative failure probability at a later horizon, because it revises the inferred shape parameter toward a less age-dependent regime. Such counter-intuitive updates reinforce the claim that survival analysis is not merely regression or classification under another name; it is a distinct prediction problem whose proper treatment requires explicit management of incomplete information.