Posts Tagged ‘Helm’
[KCDUK2024] Charting the Course: The History and Evolution of Kubernetes Security | KCDUK2024
Early Days of Kubernetes Security
At KCDUK2024, Rory McCune, a Senior Security Advocate at Datadog, delivered a captivating session titled “Charting the Course: The History and Evolution of Kubernetes Security.” Reflecting on a decade of Kubernetes, Rory traced the platform’s security evolution, from its vulnerable beginnings to modern advancements. As a former penetration tester, Rory’s insights were grounded in real-world experiences, offering a nuanced perspective on Kubernetes’ security landscape.
In 2016, Kubernetes clusters were fraught with vulnerabilities. The Kubelet, a critical node API, lacked authentication, enabling unauthenticated command execution if network access was gained. Rory recounted discovering the “Kubelet exploit,” which allowed attackers to execute commands on every node without credentials. Similarly, the read-only Kubelet port exposed pod configurations, providing attackers with valuable reconnaissance. These early flaws underscored the nascent state of Kubernetes security, requiring manual interventions like SSH tunneling to secure clusters.
Authentication and Authorization Challenges
Rory highlighted the evolution of authentication and authorization in Kubernetes. The insecure API server port, available until version 1.20, granted cluster admin access without authentication, a flaw exploited by Rory’s team in numerous pentests. Irrevocable credentials, such as client certificates and service account tokens, posed persistent risks, as they could not be revoked without deleting critical components. These issues, exemplified by CVE-18982, persisted for years, complicating cluster security.
Modern Kubernetes has mitigated these risks through token request APIs and cloud provider authentication, enabling revocable credentials. However, Rory cautioned that client certificates and long-lived tokens remain potential vulnerabilities in some distributions, requiring careful configuration. The introduction of RBAC (Role-Based Access Control) replaced the permissive “always allow” mode, though some distributions still default to the latter, necessitating vigilance.
Pod Security and Helm Vulnerabilities
Pod security presented another challenge in early Kubernetes. Rory described the “most pointless Kubernetes pod,” a manifest granting root access to nodes, exploitable by anyone with pod creation rights. Pod Security Policies (PSPs), introduced to restrict such actions, were cumbersome and deprecated in 1.25. Modern alternatives like Pod Security Admission and Validating Admission Policies offer flexible, fine-grained controls, though none are enabled by default, requiring proactive configuration.
Rory also addressed Helm 2’s Tiller service, which ran unauthenticated within clusters, granting cluster admin access to anyone reaching it. A 2024 incident involving an SAP AI service demonstrated Tiller’s lingering presence, underscoring the need for authentication configuration. These historical vulnerabilities highlight the importance of securing auxiliary components in the Kubernetes ecosystem.
Unpatchable CVEs and Networking Complexities
Rory concluded with four unpatchable Kubernetes CVEs, each illustrating networking complexities. CVE-2020-8554 allows traffic hijacking in multi-tenant clusters via misconfigured service objects, mitigated by admission controllers. CVE-2020-8555 and CVE-2021-25741 exploit server-side request forgery (SSRF), leveraging the API server’s proxy nature to access restricted networks. CVE-2021-25662 enables namespace bypass via shared load balancers, a risk in multi-tenant environments.
These CVEs underscore Kubernetes’ intricate networking model, where the API server’s proxy capabilities and IP tables rules create potential vulnerabilities. Rory emphasized the need for tailored configurations, such as connectivity services for managed Kubernetes, to mitigate these risks. His narrative highlighted the platform’s security progress while acknowledging ongoing challenges, particularly in multi-tenant scenarios.
Links:
[KCDUK2024] An Odyssey with ArgoCD: From Git to Helm | KCDUK2024
Introduction to GitOps and ArgoCD
In the ever-evolving landscape of Kubernetes management, GitOps has emerged as a cornerstone for streamlined application deployment. At KCDUK2024, Farah Adbib and Antonio Alferez, both esteemed professionals from The Workshop, delivered an insightful session titled “An Odyssey with ArgoCD: From Git to Helm.” Their talk elucidated the journey of implementing GitOps using ArgoCD within their organization, navigating through initial challenges and innovative solutions. Farah, a DevOps Solution Architect, and Antonio, a Platform Engineer, shared their expertise on leveraging Git repositories and Helm charts to manage Kubernetes clusters efficiently, offering a narrative rich with practical insights.
The session began with an overview of their ecosystem, managing 55 Kubernetes clusters across public cloud and on-premises infrastructure, supporting 1,600 nodes. These clusters cater to 350 software engineers across various business units, each responsible for deploying their applications. The need for simplicity and security in deployment processes was paramount, given the isolation requirements for some clusters. Farah and Antonio’s narrative underscored the importance of aligning technological solutions with organizational needs, setting the stage for their exploration of ArgoCD’s capabilities.
Initial Approach: Git as the Source of Truth
Initially, Farah and Antonio adopted Git repositories as the primary source for ArgoCD, a logical choice given GitOps’ emphasis on declarative configuration. They opted for a decentralized approach, deploying an ArgoCD instance per cluster to meet stringent security and isolation requirements. Each application’s Helm chart was separated from the application code, stored in distinct Git repositories to avoid replication across data centers. This separation was driven by the differing lifecycles of application code and Helm charts, aiming to streamline management.
However, this approach revealed several pain points. The necessity for Git mirrors introduced additional infrastructure complexity, requiring maintenance and coordination between development and operations teams. The use of Git branches as target revisions led to confusion, as development and operational branches coexisted, complicating version control. Moreover, rendering Helm charts within ArgoCD itself delayed feedback loops, causing failures late in the pipeline. Farah and Antonio’s candid reflection on these challenges highlighted the need for a more robust solution, prompting a strategic pivot.
Evolving to Helm Rendered Manifests
Recognizing the limitations of their initial setup, Farah and Antonio transitioned to a Helm-rendered manifest pattern. This innovative approach involved pre-rendering Helm manifests in the CI/CD pipeline and storing them in an artifact registry, such as Nexus, rather than Git mirrors. By pointing ArgoCD to these pre-rendered manifests, they eliminated the need for ArgoCD to handle rendering, significantly reducing complexity and accelerating feedback loops. This shift also unified application code and Helm charts into a single repository, simplifying versioning and pipeline management.
A key enhancement was the introduction of an external Helm values repository per cluster, allowing operational changes like scaling without triggering a full pipeline rebuild. This decoupling of configuration from code enhanced flexibility and developer experience. Antonio emphasized the adoption of semantic versioning (SemVer) for both Docker images and Helm charts, incorporating commit hashes for traceability. This meticulous versioning strategy ensured clarity on deployed versions across 55 clusters, leveraging Git notes for additional auditing information.
Lessons Learned and Developer Experience
The transition to Helm-rendered manifests yielded significant improvements. By moving rendering logic to the CI/CD pipeline, Farah and Antonio achieved a “shift-left” approach, enabling earlier failure detection and faster iterations. The elimination of Git mirrors reduced infrastructure overhead, while unified repositories streamlined development workflows. The external Helm values repository facilitated rapid operational adjustments, enhancing agility.
Farah and Antonio underscored the importance of tailoring solutions to the organizational ecosystem. Their journey highlighted the pitfalls of adhering to industry defaults without considering specific requirements. They emphasized that developer experience is paramount, advocating for solutions that empower engineers while minimizing management overhead. Their narrative serves as a testament to the value of iterative improvement, encouraging practitioners to reassess and redesign when initial solutions fall short.