Posts Tagged ‘Datadog’
[KCDUK2024] Charting the Course: The History and Evolution of Kubernetes Security | KCDUK2024
Early Days of Kubernetes Security
At KCDUK2024, Rory McCune, a Senior Security Advocate at Datadog, delivered a captivating session titled “Charting the Course: The History and Evolution of Kubernetes Security.” Reflecting on a decade of Kubernetes, Rory traced the platform’s security evolution, from its vulnerable beginnings to modern advancements. As a former penetration tester, Rory’s insights were grounded in real-world experiences, offering a nuanced perspective on Kubernetes’ security landscape.
In 2016, Kubernetes clusters were fraught with vulnerabilities. The Kubelet, a critical node API, lacked authentication, enabling unauthenticated command execution if network access was gained. Rory recounted discovering the “Kubelet exploit,” which allowed attackers to execute commands on every node without credentials. Similarly, the read-only Kubelet port exposed pod configurations, providing attackers with valuable reconnaissance. These early flaws underscored the nascent state of Kubernetes security, requiring manual interventions like SSH tunneling to secure clusters.
Authentication and Authorization Challenges
Rory highlighted the evolution of authentication and authorization in Kubernetes. The insecure API server port, available until version 1.20, granted cluster admin access without authentication, a flaw exploited by Rory’s team in numerous pentests. Irrevocable credentials, such as client certificates and service account tokens, posed persistent risks, as they could not be revoked without deleting critical components. These issues, exemplified by CVE-18982, persisted for years, complicating cluster security.
Modern Kubernetes has mitigated these risks through token request APIs and cloud provider authentication, enabling revocable credentials. However, Rory cautioned that client certificates and long-lived tokens remain potential vulnerabilities in some distributions, requiring careful configuration. The introduction of RBAC (Role-Based Access Control) replaced the permissive “always allow” mode, though some distributions still default to the latter, necessitating vigilance.
Pod Security and Helm Vulnerabilities
Pod security presented another challenge in early Kubernetes. Rory described the “most pointless Kubernetes pod,” a manifest granting root access to nodes, exploitable by anyone with pod creation rights. Pod Security Policies (PSPs), introduced to restrict such actions, were cumbersome and deprecated in 1.25. Modern alternatives like Pod Security Admission and Validating Admission Policies offer flexible, fine-grained controls, though none are enabled by default, requiring proactive configuration.
Rory also addressed Helm 2’s Tiller service, which ran unauthenticated within clusters, granting cluster admin access to anyone reaching it. A 2024 incident involving an SAP AI service demonstrated Tiller’s lingering presence, underscoring the need for authentication configuration. These historical vulnerabilities highlight the importance of securing auxiliary components in the Kubernetes ecosystem.
Unpatchable CVEs and Networking Complexities
Rory concluded with four unpatchable Kubernetes CVEs, each illustrating networking complexities. CVE-2020-8554 allows traffic hijacking in multi-tenant clusters via misconfigured service objects, mitigated by admission controllers. CVE-2020-8555 and CVE-2021-25741 exploit server-side request forgery (SSRF), leveraging the API server’s proxy nature to access restricted networks. CVE-2021-25662 enables namespace bypass via shared load balancers, a risk in multi-tenant environments.
These CVEs underscore Kubernetes’ intricate networking model, where the API server’s proxy capabilities and IP tables rules create potential vulnerabilities. Rory emphasized the need for tailored configurations, such as connectivity services for managed Kubernetes, to mitigate these risks. His narrative highlighted the platform’s security progress while acknowledging ongoing challenges, particularly in multi-tenant scenarios.
Links:
[NodeCongress2021] Demystifying Memory Leaks in JavaScript – Ruben Bridgewater
Unraveling the enigma of escalating heap usage transforms from arcane ritual to methodical pursuit under Ruben Bridgewater’s guidance. As principal software architect at Datadog and Node.js Technical Steering Committee member, Ruben demystifies leaks—unfreed allocations snowballing to OOM crashes or inflated bills—via V8’s innards and profiling arsenal.
Ruben invokes Wikipedia: leaks arise from mismanaged RAM, no longer needed yet unreclaimed, yielding upward trajectories on usage graphs versus steady baselines. JavaScript’s GC—mark-sweep for majors, scavenge for minors—orchestrates reclamation, yet closures, globals, or detached DOM snare objects in retention webs.
Profiling the Culprits
Chrome DevTools reigns: timelines chart allocations, heap snapshots freeze states for delta diffs—2.4MB spikes spotlight string hordes in func contexts. Ruben demos: inspect reveals var string chains, tracing to errant accumulators.
Clinic.js automates: clinic doctor flags leaks via flame graphs; heap-profiler pinpoints retainers. Production? APMs like Datadog monitor baselines, alerting deviations—avoid snapshots’ pauses therein.
Browser parity extends tooling: inspect Memory tab mirrors Node’s inspector.
Remediation Roadmaps
Ruben’s playbook: surveil via APMs, snapshot judiciously (controlled environs), diff deltas for deltas, excise roots—globals to WeakMaps, arrays to Sets. Data choices matter—primitives over objects; restarts as Hail Marys.
Ken Thompson’s quip—ditching code boosts productivity—caps Ruben’s ode to parsimony. Memory’s dual toll—fiscal, performative—demands preemption, yielding snappier, thriftier apps.