Posts Tagged ‘SBOM’
[PyConUS2025] Phantom Dependencies: Revealing the Hidden Software Inside Python Packages
Lecturer
Seth Michael Larson is Security Developer-in-Residence at the Python Software Foundation, a role funded by the OpenSSF Alpha-Omega project. A PSF Fellow, he focuses on supply-chain security for the Python Package Index and the broader ecosystem. His public writing and conference presentations address practical techniques for making Python software measurable and auditable. Personal site: https://sethmlarson.dev.
Abstract
Modern Python packages routinely embed non-Python code—C, C++, Rust, JavaScript, and system libraries—yet conventional manifests such as requirements.txt and the output of pip freeze record only the top-level Python distributions. These “phantom dependencies” remain invisible to most vulnerability scanners and compliance tools. Larson explains the legitimate engineering reasons for their presence, demonstrates how Software Bills of Materials (SBOMs) can surface them, and describes PEP 770, a recently accepted standard that reserves space inside wheels for machine-readable SBOM documents. Practical guidance is offered for generating high-quality manifests and for verifying that scanners actually detect known vulnerabilities across multiple language ecosystems.
The Nature and Origins of Phantom Dependencies
A phantom dependency is any software component that is present and potentially executed yet absent from the project’s declared manifest. The term, popularized by Endor Labs, captures a structural mismatch between the rich, multi-language reality of contemporary packages and the Python-centric metadata that packaging tools historically recorded.
Legitimate motives for embedding foreign code are numerous. Package managers such as pip vendor entire dependency trees in order to bootstrap themselves. Scientific and imaging libraries ship pre-compiled extension modules so that end users need not possess a compiler. The manylinux and musllinux wheel standards further encourage the bundling of shared libraries (via tools such as auditwheel) to guarantee binary compatibility across Linux distributions. Because Python’s packaging metadata cannot express non-Python components, those libraries become invisible to pip freeze, to simple Software Composition Analysis tools, and to many vulnerability scanners.
A concrete illustration is the Pillow imaging library. After installation, a virtual environment contains not only the expected pure-Python and extension modules but also a Pillow.libs directory holding libwebp, libjpeg, and other native libraries. Neither pip freeze nor a naïve file-system inventory performed by Syft reports these libraries, leaving a scanner unable to determine whether a known vulnerability in libwebp is present.
Software Bills of Materials and PEP 770
An SBOM is an ecosystem-agnostic inventory of software components, typically expressed in CycloneDX or SPDX format. Because an SBOM can describe Python packages, native shared objects, Rust crates, and JavaScript modules in a single document, it supplies the missing labels that scanners require. Larson’s PEP 770, accepted in April 2025, formalizes a conventional location—.dist-info/sboms/—inside a wheel where build tools may deposit such documents. Auditwheel, for example, can now emit a CycloneDX SBOM enumerating every library it has bundled; once the wheel is installed, the SBOM appears in the environment and is immediately consumable by Syft, Grype, or similar tools.
Adoption requires almost no change from package maintainers: upgrading the build tool that already performs bundling is usually sufficient. As more build backends and installers begin writing SBOMs, the prevalence of unlabeled phantom dependencies is expected to decline.
Practical Scanning Discipline
Larson advocates a two-stage workflow. First generate a high-quality SBOM from the target environment or container image, then inspect that SBOM manually. Confirm that expected packages appear, that package URLs or CPE identifiers are present, and that no obvious native libraries have been omitted. Only after the manifest is trustworthy should a vulnerability scanner be run against it. Open-source scanners such as Grype and Trivy support multiple ecosystems; pip-audit remains useful solely for pure-Python dependency graphs and should be combined with a multi-language scanner when native code is present.
To validate that vulnerability data actually exist for each ecosystem, a controlled experiment is recommended: edit the SBOM to force an outdated version of a component known to be vulnerable, re-scan, and verify that the expected advisory appears. Absence of the advisory indicates either missing upstream data or an incomplete SBOM—information that is far more valuable than a silent false negative.
Phantom dependencies are not malevolent; they are the inevitable consequence of Python’s role as a glue language and of the practical engineering trade-offs required for portable binary distributions. With the arrival of PEP 770 and the growing availability of SBOM-aware tooling, those dependencies can be made visible, measurable, and therefore manageable. The remaining work is largely one of tooling adoption and of cultivating the habit of treating every deployed environment as a multi-language artifact whose complete inventory must be known before any claim of security can be made.
Links:
[DevoxxBE2023] Securing the Supply Chain for Your Java Applications by Thomas Vitale
At Devoxx Belgium 2023, Thomas Vitale, a software engineer and architect at Systematic, delivered an authoritative session on securing the software supply chain for Java applications. As the author of Cloud Native Spring in Action and a passionate advocate for cloud-native technologies, Thomas provided a comprehensive exploration of securing every stage of the software lifecycle, from source code to deployment. Drawing on the SLSA framework and CNCF research, he demonstrated practical techniques for ensuring integrity, authenticity, and resilience using open-source tools like Gradle, Sigstore, and Kyverno. Through a blend of theoretical insights and live demonstrations, Thomas illuminated the critical importance of supply chain security in today’s threat landscape.
Safeguarding Source Code with Git Signatures
Thomas began by defining the software supply chain as the end-to-end process of delivering software, encompassing code, dependencies, tools, practices, and people. He emphasized the risks at each stage, starting with source code. Using Git as an example, Thomas highlighted its audit trail capabilities but cautioned that commit authorship can be manipulated. In a live demo, he showed how he could impersonate a colleague by altering Git’s username and email, underscoring the need for signed commits. By enforcing signed commits with GPG or SSH keys—or preferably a keyless approach via GitHub’s single sign-on—developers can ensure commit authenticity, establishing a verifiable provenance trail critical for supply chain security.
Managing Dependencies with Software Bills of Materials (SBOMs)
Moving to dependencies, Thomas stressed the importance of knowing exactly what libraries are included in a project, especially given vulnerabilities like Log4j. He introduced Software Bills of Materials (SBOMs) as a standardized inventory of software components, akin to a list of ingredients. Using the CycloneDX plugin for Gradle, Thomas demonstrated generating an SBOM during the build process, which provides precise dependency details, including versions, licenses, and hashes for integrity verification. This approach, integrated into Maven or Gradle, ensures accuracy over post-build scanning tools like Snyk, enabling developers to identify vulnerabilities, check license compliance, and verify component integrity before production.
Thomas further showcased Dependency-Track, an OWASP project, to analyze SBOMs and flag vulnerabilities, such as a critical issue in SnakeYAML. He introduced the Vulnerability Exploitability Exchange (VEX) standard, which complements SBOMs by documenting whether vulnerabilities affect an application. In his demo, Thomas marked a SnakeYAML vulnerability as a false positive due to Spring Boot’s safe deserialization, demonstrating how VEX communicates security decisions to stakeholders, reducing unnecessary alerts and ensuring compliance with emerging regulations.
Building Secure Artifacts with Reproducible Builds
The build phase, Thomas explained, is another critical juncture for security. Using Spring Boot as an example, he outlined three packaging methods: JAR files, native executables, and container images. He critiqued Dockerfiles for introducing non-determinism and maintenance overhead, advocating for Cloud Native Buildpacks as a reproducible, secure alternative. In a demo, Thomas built a container image with Buildpacks, highlighting its fixed creation timestamp (January 1, 1980) to ensure identical outputs for unchanged inputs, enhancing security by eliminating variability. This reproducibility, coupled with SBOM generation during the build, ensures artifacts are both secure and traceable.
Signing and Verifying Artifacts with SLSA
To ensure artifact integrity, Thomas introduced the SLSA framework, which provides guidelines for securing software artifacts across the supply chain. He demonstrated signing container images with Sigstore’s Cosign tool, using a keyless approach to avoid managing private keys. This process, integrated into a GitHub Actions pipeline, ensures that artifacts are authentically linked to their creator. Thomas further showcased SLSA’s provenance generation, which documents the artifact’s origin, including the Git commit hash and build steps. By achieving SLSA Level 3, his pipeline provided non-falsifiable provenance, ensuring traceability from source code to deployment.
Securing Deployments with Policy Enforcement
The final stage, deployment, requires validating artifacts to ensure they meet security standards. Thomas demonstrated using Cosign and the SLSA Verifier to validate signatures and provenance, ensuring only trusted artifacts are deployed. On Kubernetes, he introduced Kyverno, a policy engine that enforces signature and provenance checks, automatically rejecting non-compliant deployments. This approach ensures that production environments remain secure, aligning with the principle of validating metadata to prevent unauthorized or tampered artifacts from running.
Conclusion: A Holistic Approach to Supply Chain Security
Thomas’s session at Devoxx Belgium 2023 provided a robust framework for securing Java application supply chains. By addressing source code integrity, dependency management, build reproducibility, artifact signing, and deployment validation, he offered a comprehensive strategy to mitigate risks. His practical demonstrations, grounded in open-source tools and standards like SLSA and VEX, empowered developers to adopt these practices without overwhelming complexity. Thomas’s emphasis on asking “why” at each step encouraged attendees to tailor security measures to their context, ensuring both compliance and resilience in an increasingly regulated landscape.