[PyConUS2025] Phantom Dependencies: Revealing the Hidden Software Inside Python Packages
Lecturer
Seth Michael Larson is Security Developer-in-Residence at the Python Software Foundation, a role funded by the OpenSSF Alpha-Omega project. A PSF Fellow, he focuses on supply-chain security for the Python Package Index and the broader ecosystem. His public writing and conference presentations address practical techniques for making Python software measurable and auditable. Personal site: https://sethmlarson.dev.
Abstract
Modern Python packages routinely embed non-Python code—C, C++, Rust, JavaScript, and system libraries—yet conventional manifests such as requirements.txt and the output of pip freeze record only the top-level Python distributions. These “phantom dependencies” remain invisible to most vulnerability scanners and compliance tools. Larson explains the legitimate engineering reasons for their presence, demonstrates how Software Bills of Materials (SBOMs) can surface them, and describes PEP 770, a recently accepted standard that reserves space inside wheels for machine-readable SBOM documents. Practical guidance is offered for generating high-quality manifests and for verifying that scanners actually detect known vulnerabilities across multiple language ecosystems.
The Nature and Origins of Phantom Dependencies
A phantom dependency is any software component that is present and potentially executed yet absent from the project’s declared manifest. The term, popularized by Endor Labs, captures a structural mismatch between the rich, multi-language reality of contemporary packages and the Python-centric metadata that packaging tools historically recorded.
Legitimate motives for embedding foreign code are numerous. Package managers such as pip vendor entire dependency trees in order to bootstrap themselves. Scientific and imaging libraries ship pre-compiled extension modules so that end users need not possess a compiler. The manylinux and musllinux wheel standards further encourage the bundling of shared libraries (via tools such as auditwheel) to guarantee binary compatibility across Linux distributions. Because Python’s packaging metadata cannot express non-Python components, those libraries become invisible to pip freeze, to simple Software Composition Analysis tools, and to many vulnerability scanners.
A concrete illustration is the Pillow imaging library. After installation, a virtual environment contains not only the expected pure-Python and extension modules but also a Pillow.libs directory holding libwebp, libjpeg, and other native libraries. Neither pip freeze nor a naïve file-system inventory performed by Syft reports these libraries, leaving a scanner unable to determine whether a known vulnerability in libwebp is present.
Software Bills of Materials and PEP 770
An SBOM is an ecosystem-agnostic inventory of software components, typically expressed in CycloneDX or SPDX format. Because an SBOM can describe Python packages, native shared objects, Rust crates, and JavaScript modules in a single document, it supplies the missing labels that scanners require. Larson’s PEP 770, accepted in April 2025, formalizes a conventional location—.dist-info/sboms/—inside a wheel where build tools may deposit such documents. Auditwheel, for example, can now emit a CycloneDX SBOM enumerating every library it has bundled; once the wheel is installed, the SBOM appears in the environment and is immediately consumable by Syft, Grype, or similar tools.
Adoption requires almost no change from package maintainers: upgrading the build tool that already performs bundling is usually sufficient. As more build backends and installers begin writing SBOMs, the prevalence of unlabeled phantom dependencies is expected to decline.
Practical Scanning Discipline
Larson advocates a two-stage workflow. First generate a high-quality SBOM from the target environment or container image, then inspect that SBOM manually. Confirm that expected packages appear, that package URLs or CPE identifiers are present, and that no obvious native libraries have been omitted. Only after the manifest is trustworthy should a vulnerability scanner be run against it. Open-source scanners such as Grype and Trivy support multiple ecosystems; pip-audit remains useful solely for pure-Python dependency graphs and should be combined with a multi-language scanner when native code is present.
To validate that vulnerability data actually exist for each ecosystem, a controlled experiment is recommended: edit the SBOM to force an outdated version of a component known to be vulnerable, re-scan, and verify that the expected advisory appears. Absence of the advisory indicates either missing upstream data or an incomplete SBOM—information that is far more valuable than a silent false negative.
Phantom dependencies are not malevolent; they are the inevitable consequence of Python’s role as a glue language and of the practical engineering trade-offs required for portable binary distributions. With the arrival of PEP 770 and the growing availability of SBOM-aware tooling, those dependencies can be made visible, measurable, and therefore manageable. The remaining work is largely one of tooling adoption and of cultivating the habit of treating every deployed environment as a multi-language artifact whose complete inventory must be known before any claim of security can be made.