Posts Tagged ‘SethLarson’
[PyConUS2025] Phantom Dependencies: Revealing the Hidden Software Inside Python Packages
Lecturer
Seth Michael Larson is Security Developer-in-Residence at the Python Software Foundation, a role funded by the OpenSSF Alpha-Omega project. A PSF Fellow, he focuses on supply-chain security for the Python Package Index and the broader ecosystem. His public writing and conference presentations address practical techniques for making Python software measurable and auditable. Personal site: https://sethmlarson.dev.
Abstract
Modern Python packages routinely embed non-Python code—C, C++, Rust, JavaScript, and system libraries—yet conventional manifests such as requirements.txt and the output of pip freeze record only the top-level Python distributions. These “phantom dependencies” remain invisible to most vulnerability scanners and compliance tools. Larson explains the legitimate engineering reasons for their presence, demonstrates how Software Bills of Materials (SBOMs) can surface them, and describes PEP 770, a recently accepted standard that reserves space inside wheels for machine-readable SBOM documents. Practical guidance is offered for generating high-quality manifests and for verifying that scanners actually detect known vulnerabilities across multiple language ecosystems.
The Nature and Origins of Phantom Dependencies
A phantom dependency is any software component that is present and potentially executed yet absent from the project’s declared manifest. The term, popularized by Endor Labs, captures a structural mismatch between the rich, multi-language reality of contemporary packages and the Python-centric metadata that packaging tools historically recorded.
Legitimate motives for embedding foreign code are numerous. Package managers such as pip vendor entire dependency trees in order to bootstrap themselves. Scientific and imaging libraries ship pre-compiled extension modules so that end users need not possess a compiler. The manylinux and musllinux wheel standards further encourage the bundling of shared libraries (via tools such as auditwheel) to guarantee binary compatibility across Linux distributions. Because Python’s packaging metadata cannot express non-Python components, those libraries become invisible to pip freeze, to simple Software Composition Analysis tools, and to many vulnerability scanners.
A concrete illustration is the Pillow imaging library. After installation, a virtual environment contains not only the expected pure-Python and extension modules but also a Pillow.libs directory holding libwebp, libjpeg, and other native libraries. Neither pip freeze nor a naïve file-system inventory performed by Syft reports these libraries, leaving a scanner unable to determine whether a known vulnerability in libwebp is present.
Software Bills of Materials and PEP 770
An SBOM is an ecosystem-agnostic inventory of software components, typically expressed in CycloneDX or SPDX format. Because an SBOM can describe Python packages, native shared objects, Rust crates, and JavaScript modules in a single document, it supplies the missing labels that scanners require. Larson’s PEP 770, accepted in April 2025, formalizes a conventional location—.dist-info/sboms/—inside a wheel where build tools may deposit such documents. Auditwheel, for example, can now emit a CycloneDX SBOM enumerating every library it has bundled; once the wheel is installed, the SBOM appears in the environment and is immediately consumable by Syft, Grype, or similar tools.
Adoption requires almost no change from package maintainers: upgrading the build tool that already performs bundling is usually sufficient. As more build backends and installers begin writing SBOMs, the prevalence of unlabeled phantom dependencies is expected to decline.
Practical Scanning Discipline
Larson advocates a two-stage workflow. First generate a high-quality SBOM from the target environment or container image, then inspect that SBOM manually. Confirm that expected packages appear, that package URLs or CPE identifiers are present, and that no obvious native libraries have been omitted. Only after the manifest is trustworthy should a vulnerability scanner be run against it. Open-source scanners such as Grype and Trivy support multiple ecosystems; pip-audit remains useful solely for pure-Python dependency graphs and should be combined with a multi-language scanner when native code is present.
To validate that vulnerability data actually exist for each ecosystem, a controlled experiment is recommended: edit the SBOM to force an outdated version of a component known to be vulnerable, re-scan, and verify that the expected advisory appears. Absence of the advisory indicates either missing upstream data or an incomplete SBOM—information that is far more valuable than a silent false negative.
Phantom dependencies are not malevolent; they are the inevitable consequence of Python’s role as a glue language and of the practical engineering trade-offs required for portable binary distributions. With the arrival of PEP 770 and the growing availability of SBOM-aware tooling, those dependencies can be made visible, measurable, and therefore manageable. The remaining work is largely one of tooling adoption and of cultivating the habit of treating every deployed environment as a multi-language artifact whose complete inventory must be known before any claim of security can be made.
Links:
[PyConUS2025] Python Software Foundation Welcome and Ecosystem Updates
Lecturer
Deb Nicholson is the Executive Director of the Python Software Foundation. She joined the organization in April 2022 after prior roles at the Open Source Initiative (Interim General Manager), Software Freedom Conservancy (Director of Community Operations), and the Open Invention Network. A founding organizer of the Seattle GNU/Linux Conference, she has received the O’Reilly Open Source Award and the Award for the Advancement of Free Software. Nicholson resides in Cambridge, Massachusetts. Her professional profile is available on LinkedIn at https://www.linkedin.com/in/denicholson and on X under the handle @baconandcoconut.
Abstract
This multi-part welcome session presents the current state of the Python Software Foundation, its infrastructure challenges, security initiatives, and community programs, interwoven with short addresses from major sponsors. Deb Nicholson surveys growth metrics for PyPI, the grants program, staffing, and fiscal sponsorships while reflecting on the social strengths and external pressures facing the community. Accompanying remarks from representatives of AWS, Alpha Omega, Meta, and Google highlight collaborative investments in security, tooling, and language evolution.
Security Initiatives and Infrastructure Stewardship
The session opened with brief sponsor remarks. An NVIDIA representative invited attendees to discussions on faster Python and smaller downloads. Hannah Aubrey of AWS Open Source then framed the company’s commitment to long-term security of critical open-source projects, noting that AWS depends on Python’s reliability. Michael Windsor of Alpha Omega described the organization’s four-pronged approach—staffing dedicated security roles, improving package managers, conducting audits, and funding experiments—initiated in response to supply-chain crises such as Log4Shell. He credited Seth Larson and Mike Fiedler for leadership within the Python ecosystem.
Seth Larson, Security Developer in Residence at the PSF, outlined the contemporary threat landscape: vulnerability exploitation, package poisoning, and social-engineering attacks on maintainers. His mandate is to establish secure-by-default experiences for millions of users. He displayed a visualization of social relationships among top PyPI packages, underscoring that security work is distributed across a dense contributor network rather than concentrated in a single office. Larson invited participation in security-focused talks, open spaces, and a meet-the-experts session at the AWS booth later that day.
Ian Barber of Meta, a visionary sponsor, reported that Python is now the most-used language inside the company, with more than three thousand developers. Meta’s investments include Pyfly, an IDE plugin and type-check engine released in alpha for instant autocomplete and navigation on large codebases; free-threaded Python aimed at true multi-threading without the GIL, particularly beneficial for AI workloads; and continued development of Triton and PyTorch, the latter now supporting NVIDIA’s Blackwell architecture and flex attention. Barber directed attendees to the typing summit and the Meta booth.
Deb Nicholson then assumed the podium. She restated the PSF mission: to promote, protect, and advance the Python language and to foster a diverse international community. The Foundation serves as the nonprofit home for both the language and its community, providing infrastructure, security, and legal support while channeling grants and fiscal sponsorships. Python’s ascent to the most-popular language on GitHub was noted with measured pride, accompanied by gratitude to Fastly for a five-year commitment that underpins CDN capacity.
PyPI growth figures illustrated the scale of demand: an 84 percent rise in download counts and 48 percent rise in bandwidth in 2024, following 57 percent and 45 percent increases in the two preceding years. The newly launched PyPI Organizations feature, intended to support collaborative teams and to generate modest administrative revenue for further improvements, attracted 8 500 initial requests—many duplicates or low-quality. Maria Asha cleared the resulting backlog, and the feature is now fully operational under the stewardship of infrastructure lead E.
The grants program has undergone substantial redesign under Community Communications Manager Marie Nordon. Application processes are shorter, more transparent, and more equitable; community updates are regular. Demand has expanded in every dimension—events, amounts, and geographic reach—while revenue has not kept pace, prompting an open invitation for new funding ideas and corporate partnerships.
Community Resilience, Staffing, and Forward Outlook
Nicholson turned to the social character of the Python community, describing its historic openness to “weird kids” and unconventional projects—ranging from analysis of pre-Columbian tooth enamel to experimental tooling. This receptivity, she argued, keeps the language fresh and expands its application domains. Yet she acknowledged the broader context of geopolitical and economic uncertainty that has deterred hundreds of potential international attendees and complicated nonprofit operations. Layoffs, regulatory flux, and funding volatility place additional strain on a small organization whose largest single budget item remains PyCon US itself.
Despite these pressures, the human infrastructure of the PSF remains robust. Nicholson introduced the approximately thirteen staff members who support more than eight million users worldwide. Olivia Sauls directs the conference; Seth Larson and Mike Fiedler manage security reporting and infrastructure; E oversees core systems with assistance from Jacob Coffee; Marie Nordon handles grants, D&I, fellows, and communications; Jamie has transitioned from contractor to full-time events staff; accountants Phyllis and Laura manage fiscal sponsorships, payroll, and compliance; and Lauren has been promoted to Deputy Executive Director with expanded responsibilities for partnerships and strategic planning. Three CPython Developer-in-Residence positions—Lucas, Peter, and Siri—support core development, with Wukush also present for the language summit.
The Board of Directors and a suite of volunteer working groups (code of conduct, diversity and inclusion, education and outreach, fellows, grants, infrastructure, jobs board, packaging, trademarks) extend capacity further. The Foundation acts as fiscal sponsor for twenty projects, most prominently the global PyLadies network. Thousands of additional volunteers worldwide organize local conferences, meetups, and educational programs that sustain Python’s growth.
Sponsor advocacy received special thanks: the individuals inside corporations who repeatedly champion larger contributions and greater conference attendance. Nicholson closed by urging the community to “stay weird and keep welcoming all the nice weirdos,” then introduced Lisa of Google. Lisa, attending her first PyCon, praised the energy and the dedicated newcomers’ session. She reaffirmed Google’s reliance on Python across scripting, DevOps, and machine learning, and expressed particular interest in free-threaded Python, typing-standard evolution, and performance work. Multiple Google colleagues were present for the typing summit, a talk on programming for oneself, and a supply-chain security open space.
Collectively the addresses portray a Foundation that has scaled its technical and social infrastructure in step with extraordinary adoption while remaining attentive to inclusion, security, and long-term sustainability. The interplay of staff, volunteers, sponsors, and core developers illustrated in the session constitutes the operational backbone that enables the language’s continued vitality.