Posts Tagged ‘KCDUK2024’
[KCDUK2024] Kubernetes Privilege Escalation Tactics: Unveiling Vulnerabilities and Fortifying Defenses
Iain Smart and Andrew Martin presented a compelling exploration of Kubernetes privilege escalation tactics, offering a deep dive into how both trusted and unprivileged users can exploit vulnerabilities within the system. Their discussion, a highlight of KCDUK2024, provided invaluable insights for SREs, security teams, and pentesters aiming to enhance cluster security.
The core of their presentation focused on the critical need to understand potential attack vectors and implement robust defense mechanisms. They articulated that while penetration testing Kubernetes should inherently be challenging, certain oversights can inadvertently simplify the process for malicious actors. The speakers emphasized the multifaceted nature of threats, ranging from rogue SREs and disaffected platform developers to external hostile internet citizens.
Smart and Martin meticulously guided attendees through various methods to escalate privileges, achieve persistence, and potentially wreak havoc across a cluster, all while attempting to obscure any traces of activity. Their expertise illuminated the intricate interplay of Kubernetes components and how unusual interactions or component abuse can be leveraged for unauthorized access.
Understanding Kubernetes Vulnerabilities and Exploitation
The presenters underscored the importance of comprehending the array of Kubernetes vulnerabilities that security professionals must be aware of. They elaborated on specific techniques that adversaries might employ, detailing how seemingly minor misconfigurations or overlooked edge cases can become critical points of entry. The discussion extended to identifying different adversary levels, stressing that tailoring defenses according to the threat model is paramount for effective security.
Smart and Martin provided practical insights into the most cost-effective and efficient strategies for fortifying Kubernetes clusters. They advocated for a proactive approach that encompasses preventative controls, such as static analysis on deployed artifacts and meticulous enumeration of Role-Based Access Control (RBAC). The emphasis was not solely on prevention but also on robust detective controls, including monitoring external traffic through split-horizon DNS to identify suspicious outbound communications.
Mitigating Risks and Ensuring Robust Security
A key takeaway from Smart and Martin’s presentation was the critical role of remediative controls. These controls are designed to detect ongoing attacks and initiate automated responses, such as node draining and shutdown procedures, to prevent data exfiltration. Despite implementing a comprehensive suite of preventative, detective, and remediative measures, they acknowledged that complete isolation and absolute security are unattainable ideals. The ever-evolving threat landscape, exemplified by nation-state attacks and sophisticated persistent threats, necessitates continuous vigilance and adaptation.
The speakers concluded by reiterating that detection is as crucial as prevention in the complex puzzle of cloud-native security. They highlighted that while various security tools and practices are available, the dynamic nature of threats requires an ongoing commitment to learning, monitoring, and adapting. Their insights provided a valuable roadmap for organizations striving to secure their Kubernetes environments against an increasingly sophisticated array of threats.
Links:
[KCDUK2024] The Operator Antipattern | Gerald Schmidt
At KCDUK2024, Gerald Schmidt, data platform engineering lead at DS Smith, delivered a candid reflection on the Kubernetes operator pattern, questioning its efficacy in modern platform engineering. Drawing from his experience at Babylon Health, Go City, and Thoughtworks, Gerald shared a journey from enthusiasm to disillusionment, highlighting how operators, despite their promise, often introduce complexity and fragility that outweigh their benefits.
The Allure and Pitfalls of Operators
Gerald opened with his initial excitement for operators, which promised to extend Kubernetes’ API for managing complex stateful applications. He referenced Brandon Phillips’ vision of operators embedding domain expertise to automate tasks, such as managing databases or message queues. However, Gerald’s experience revealed that operators rarely delivered on this promise. Instead, they introduced tight coupling between custom resource definitions (CRDs) and controllers, creating operational challenges.
A pivotal anecdote from his time at a health startup illustrated this. Gerald’s team built operators for data flows, only to face disaster when CRDs were inadvertently deleted, requiring a weekend-long recovery. This incident led to their team losing deployment privileges, underscoring the risks of operators in environments with shrinking teams and heightened scrutiny. Gerald argued that operators, while powerful, often fail to match the robustness of managed services like AWS RDS, which offer superior durability and availability.
Persistent Volumes Over Stateful Applications
Gerald challenged the notion that Kubernetes struggles with stateful applications, suggesting the real issue lies with persistent volumes. Solutions like Thanos and WarpStream, which leverage object storage, address this effectively without relying on operators. He highlighted the maturity of the object storage ecosystem, with providers like AWS S3, Google Cloud Storage, and Wasabi offering cost-effective, reliable alternatives. The Container Object Storage Interface (COSI), though still emerging, promises further simplification, reducing the need for complex operator-based solutions.
He contrasted this with the operator pattern’s limitations, particularly around CRDs. Unlike controllers, which integrate seamlessly with Kubernetes’ control loop, CRDs require administrative privileges and complicate versioning and upgrades. Gerald cited the AWS Controllers for Kubernetes (ACK), still in v1alpha1, as an example of versioning challenges, where upgrades introduce new failure modes and complexity.
Toward Simpler, Developer-Friendly Solutions
Reflecting on developer experience, Gerald advocated for alternatives like config maps over CRDs. He praised Grafana’s approach, which uses config maps for dashboards, avoiding the need for administrative access and simplifying deployment. Similarly, older Prometheus setups allowed developers to control scraping with simple annotations, granting autonomy without the overhead of CRDs. Gerald argued that controllers alone, paired with config maps, provide most of the operator’s benefits without the downsides.
He concluded with a decision framework: only pursue the operator pattern if the use case justifies a domain-specific language. For operational tasks with low blast radius, like Kyverno’s policy enforcement, operators excel. However, for development teams, Gerald recommended avoiding custom operators, as they often lead to wasted effort and fragility, as seen in his team’s abandoned six-month project.
Links:
[KCDUK2024] CVEs and Kubernetes: A Love Story? | Marcus Tenorio
In a lively lightning talk at KCDUK2024, Marcus Tenorio, an engineering manager with a background in incident response, brought a fresh perspective on the relationship between Kubernetes and Common Vulnerabilities and Exposures (CVEs). With a nod to the conference’s community spirit, Marcus framed security challenges as opportunities for growth, likening the evolution of Kubernetes security to a love story where vulnerabilities drive collaboration and improvement.
The Evolution of Kubernetes Security
Marcus began by exploring the CVE landscape, drawing from the official CVE feed, MITRE, and NVD databases. He noted that while Kubernetes, launched in 2014, has seen a rise in reported CVEs, this reflects increased scrutiny rather than declining security. Early vulnerabilities, often identified by community members like a Google engineer on GitHub, showcased the power of open-source collaboration. Marcus highlighted that critical CVEs in Kubernetes are relatively rare, contrasting with infamous incidents like Log4j, suggesting a stable core.
He analyzed a sample of 55 CVEs, revealing that the growth in reported vulnerabilities corresponds to Kubernetes’ maturity. As the platform evolves, the community actively identifies and resolves issues, strengthening its security posture. Marcus emphasized that this process mirrors a relationship where challenges foster growth, with each CVE contributing to a more robust ecosystem.
Community-Driven Security
The heart of Marcus’s talk was the role of community in Kubernetes security. He shared an anecdote from an e-commerce platform where a team’s proactive vulnerability hunting led to safer systems, not because vulnerabilities were abundant, but because they were addressed collaboratively. This approach, rooted in policies and shared learning, transforms potential threats into opportunities for improvement.
Marcus encouraged attendees to embrace this “love” for security by fostering open communication and leveraging data to understand vulnerabilities. Tools like Datadog, despite occasional AI hallucinations, help teams analyze and respond to CVEs effectively. By viewing security as a collective journey, Marcus underscored how Kubernetes’ community-driven model drives resilience, aligning with KCDUK2024’s ethos of collaboration.
[KCDUK2024] Platform Orchestrators: The Missing Middle of Internal Developer Platforms | Daniel Bryant
At KCDUK2024, Daniel Bryant, a seasoned platform engineering advocate, delivered a compelling case for platform orchestrators as the critical “missing middle” in internal developer platforms (IDPs). Drawing from his extensive experience with Kubernetes, Mesos, and tools like Backstage and Crossplane, Daniel explored how orchestrators bridge the gap between developer-facing portals and infrastructure layers, enabling scalable, efficient, and user-centric platforms. His talk offered a blueprint for organizations to balance speed, safety, and scale in their platform engineering efforts.
The Evolution of Platform Engineering
Daniel began by contrasting three approaches to platform building: top-down, app-centric portals; bottom-up, infrastructure-focused solutions; and a middle-out, platform-engineering-focused model. Top-down approaches, like Backstage, excel at providing quick wins with developer portals but struggle with day-two operations like upgrades and maintenance. Bottom-up approaches, such as Terraform or Crossplane, offer robust automation but often overwhelm developers with infrastructure complexity. The middle-out approach, which Daniel champions, treats platforms as products, prioritizing user needs and process automation.
He referenced Gartner’s platform engineering model, which identifies three layers: application choreography, platform orchestration, and infrastructure composition. The platform orchestration layer, often overlooked, manages the platform’s lifecycle and APIs, ensuring seamless integration between developer workflows and infrastructure. Daniel’s experience with tools like Crossplane and CNOE (Cloud Native Operational Excellence) highlighted how orchestrators codify business processes, reducing coordination overhead and enabling scalability.
Addressing Developer Pain Points
Modern software engineering faces challenges like slow delivery, high-risk deployments, and tech sprawl. Daniel cited statistics showing that 50% of organizations deploy code less than once a month, and 42% of developers fear production failures. Platform orchestrators address these by offering “everything as a service,” from databases to domain-specific services like fraud detection in finance. By automating manual processes, such as security sign-offs, orchestrators enhance safety and efficiency, allowing developers to focus on coding.
Daniel emphasized the importance of progressive disclosure—presenting simple interfaces initially while enabling advanced functionality as needed. He recounted a past experience where a 500-line YAML configuration overwhelmed developers, underscoring the need for intuitive abstractions. Tools like Open Application Model (OAM) and Score, donated to the CNCF, provide developer-friendly APIs, while orchestrators like Kritik and Cusion Stack manage complex workflows, ensuring platforms remain adaptable to changing business needs.
Building Platforms as Products
The heart of Daniel’s message was treating platforms as products, designed with user needs at the forefront. He advocated for clear domain boundaries, inspired by principles like SOLID and CUPID, to ensure platforms are composable and maintainable. Tools like Kritik, where Daniel contributes, use Kubernetes CRDs to define platform APIs, allowing workflows to be containerized and reusable. This approach enables teams to manage platform components at scale, from rolling out security fixes to integrating auditing processes.
Drawing from Team Topologies, Daniel stressed collaboration between platform and development teams to align on goals like adoption rates and onboarding times. He warned against the “build it and they will come” mentality, urging platform engineers to engage users early and measure success through leading indicators like onboarding efficiency and lagging indicators like incident reduction. By treating platforms as products, organizations can achieve the speed, safety, and scale needed to thrive in cloud-native environments.
Links:
[KCDUK2024] The Joy of DevEx: Tightening Developer Feedback Loops in Kubernetes | Matthew Revell-Gordon
At KCDUK2024, Matthew Revell-Gordon, a senior platform engineering consultant at OpenCredo, shared an engaging exploration of how to enhance the developer experience (DevEx) by tightening feedback loops in Kubernetes. His talk addressed the challenges of local testing for cloud-native applications, offering practical solutions to align local Kubernetes environments with cloud deployments. By leveraging tools like Colima, KIND, and MetalLB, Matthew demonstrated how to create a robust local setup that mirrors production, fostering faster development cycles and a more joyful DevEx.
Challenges of Local Testing in Cloud-Native Environments
Cloud-native technologies like Kubernetes are designed to thrive in distributed, scalable cloud environments, which poses a significant hurdle for local testing. Matthew highlighted that while unit tests and linting are straightforward on a laptop, integration testing and infrastructure configuration validation are far more complex. Laptops lack the scale and fidelity of cloud environments, leading developers to rely heavily on cloud-based testing. This approach, however, introduces lengthy feedback loops, as CI/CD pipelines can take time to queue, bootstrap, and execute—often failing due to trivial errors like typos that could have been caught earlier.
Matthew recounted a real-world scenario where developers used Docker Compose for local testing, only to encounter issues when deploying to Kubernetes due to untested Ingress controller changes. This mismatch delayed releases and frustrated teams, underscoring the need for a local environment that closely emulates production. He emphasized that while cloud testing is inevitable, optimizing local setups can significantly reduce cycle times, catching errors before they reach costly cloud pipelines.
Crafting a Production-Like Local Kubernetes Setup
To address these challenges, Matthew proposed a solution centered on Colima, KIND, and MetalLB to create a local Kubernetes cluster that mirrors cloud deployments. Colima, a lightweight alternative to Docker Desktop, provides a configurable Linux-based virtual machine, allowing developers to fine-tune their environment. KIND (Kubernetes IN Docker) enables multi-node, high-availability clusters with precise control over Kubernetes versions, ensuring consistency with production. MetalLB replaces cloud-specific load balancer controllers, provisioning external IP addresses for local services.
In a live demo, Matthew provisioned a multi-node Kubernetes cluster using KIND, deploying a simple application with node affinity and a load balancer service accessible directly from his Mac. He addressed networking challenges by configuring routes and IP tables, ensuring seamless connectivity between the host, Colima VM, and KIND network. This setup, while not identical to a cloud environment, offers a close approximation, enabling developers to test complex configurations like node affinity and rollouts locally. Matthew also recommended automating these setups with scripts, making them accessible to developers who may not be infrastructure experts, thus enhancing team efficiency.
Enhancing DevEx Through Automation and Community
The true joy of DevEx, Matthew argued, lies in empowering developers to focus on coding rather than wrestling with infrastructure. By packaging local Kubernetes setups into one-click scripts, platform engineers can democratize access to production-like environments, reducing friction and boosting productivity. He cautioned against anti-patterns like creating local-only manifests or altering production setups for testing, as these diverge from the goal of fidelity. Instead, he advocated an iterative approach, starting with the most pressing pain points and gradually refining the local environment.
Matthew’s passion for community resonated throughout his talk. He encouraged platform engineers to share their tooling and scripts, fostering collaboration and easing the burden on developers. By tightening feedback loops, these solutions not only save time and costs but also cultivate a sense of joy in the development process, aligning with the broader ethos of KCDUK2024’s focus on community-driven innovation.
Links:
[KCDUK2024] From Greenfield Dreams to Brownfield Realities: Navigating Platform Development
Cat Morris, a product manager at Snyk, delivered a thought-provoking session at KCDUK2024, exploring the challenges of building platforms in brownfield environments. Drawing from her extensive experience in legacy modernization, Cat challenged the audience to rethink traditional approaches to platform development. Her talk, rich with insights from psychology and practical strategies, offered a roadmap for transforming existing systems into robust, user-adopted platforms, emphasizing the importance of trust, ownership, and user-centric design.
The Brownfield Reality
Cat introduced the concept of brownfield development, borrowed from civil engineering, where new projects are built amidst existing, often hazardous, structures. In software, brownfield describes developing new systems alongside legacy applications, a reality for most platform teams. Unlike greenfield projects, which start from scratch, brownfield environments inherit constraints and complexities. Cat’s experience at BT, government agencies, and Thoughtworks underscored that platforms rarely begin with a clean slate; they emerge from existing engineering teams’ workflows, making adoption a significant challenge.
Her story of a neo-bank transitioning from a crypto wallet to a regulated entity illustrated this. Initially, her team built a greenfield platform to address compliance needs, only to find that existing teams preferred their familiar, albeit imperfect, processes. This resistance led Cat to a key realization: platforms are inherently brownfield and emergent, evolving from common pain points rather than imposed solutions. Understanding this shift is critical for building platforms that resonate with users.
Psychological Barriers to Adoption
Cat leveraged psychological principles to explain user resistance. Prospect theory, from Daniel Kahneman and Amos Tversky, suggests people prefer certain outcomes over risky, potentially higher-reward alternatives. In platform development, this translates to teams sticking with familiar processes despite a new platform’s benefits. Cat advocated focusing on a “job to be done” framework, ensuring platforms enable users to complete specific tasks end-to-end, reducing perceived risk and encouraging adoption.
The endowment effect further complicates adoption, as users value their existing systems—however flawed—because they “own” them. Cat shared how pairing with a team at the neo-bank, integrating their needs into the platform, fostered a sense of ownership. This collaboration turned skeptics into advocates, driving near-100% adoption within six months. By involving users in design, platform teams can transfer this sense of ownership, making the new system feel like a natural extension of their workflow.
Building Trust for Long-Term Success
Trust, Cat emphasized, is the linchpin of platform adoption. Drawing on trust repair research, she highlighted how trust, once broken, is difficult to restore fully. A platform outage or failure can erode user confidence, requiring significant effort to rebuild. To avoid this, Cat recommended transparent roadmaps, high-quality monitoring to preempt issues, and consistent experiences through tools like fleet management. Her anecdote about her cat, Opal, illustrated this vividly: failing to reward Opal’s tricks broke trust, requiring weeks to restore, a lesson applicable to platform users.
Cat’s strategies—focusing on user outcomes, fostering ownership, and prioritizing trust—offer a blueprint for navigating brownfield challenges. By building platforms alongside users, incorporating familiar elements, and pacing changes to match user workflows, teams can overcome resistance and drive adoption. Her call to action urged platform engineers to embrace these principles, ensuring platforms are not just functional but trusted and valued by their users.
Links:
[KCDUK2024] Comprehensible Kubernetes: Empowering Scientists with Scalable and Secure Platforms for HPC and AI
At KCDUK2024, Scott Coulton and Tyler, representing StackHPC, delivered an insightful presentation on making Kubernetes accessible to scientists and researchers with minimal technical backgrounds. Their talk focused on crafting cloud-native platforms that prioritize usability, security, and scalability, enabling researchers to focus on their work rather than grappling with complex infrastructure. By leveraging open-source tools like ClusterAPI, Helm, Zenith, and Keycloak, StackHPC has bridged the gap between high-performance computing (HPC) and cloud-native technologies, transforming how scientific research is conducted.
Bridging HPC and Cloud-Native for Research
StackHPC, a Bristol-based consultancy specializing in HPC and cloud solutions, was founded on traditional HPC expertise, such as Slurm clusters and high-performance networking. Scott and Tyler outlined their mission to translate this expertise into the cloud-native realm, primarily for universities and research institutions. Their approach centers on three pillars: reconfigurable infrastructure, performance optimization, and self-service applications. By deploying private OpenStack clouds and Kubernetes clusters, they enable researchers to access tailored environments without needing deep technical knowledge.
The diversity of scientific use cases presents unique challenges. Researchers may require Slurm clusters for batch processing, JupyterHub for data analysis, or GPU-enabled environments for machine learning. Scott highlighted a common thread: scientists are not platform engineers and seek intuitive, reliable platforms. StackHPC’s solution, the LOKI stack (Linux, OpenStack, Kubernetes Infrastructure), integrates open-source tools to provide a flexible, self-service platform that meets these needs while maintaining security and scalability.
The LOKI Stack: A Scalable Solution
Tyler delved into the technical underpinnings of StackHPC’s LOKI stack, which combines ClusterAPI, Zenith, and Azimuth to deliver seamless infrastructure management. ClusterAPI, a declarative API, simplifies Kubernetes cluster lifecycle management, supporting auto-healing and auto-scaling across providers like OpenStack. Tyler explained how Helm charts streamline cluster provisioning, ensuring consistency and ease of deployment. This approach allows operators to manage infrastructure efficiently, freeing researchers from administrative burdens.
Zenith, an innovative application proxy, enables secure external access to applications without public IPs, using SSH tunneling and OIDC authentication. Azimuth, a self-service web portal, offers pre-configured appliances like JupyterHub and Slurm clusters, customizable via Helm charts. Keycloak integration ensures secure access management, allowing platform administrators to create user accounts without granting cloud access. This architecture empowers researchers to deploy and manage platforms independently, aligning complexity with their expertise.
Case Studies: Real-World Impact
Scott presented two case studies illustrating the LOKI stack’s versatility. The first involved deploying JupyterHub for training courses on a private OpenStack cloud. Using Azimuth, administrators created isolated Keycloak realms to manage attendee accounts, ensuring secure access via Zenith’s proxying. This setup allowed course tutors to focus on teaching, with attendees accessing familiar JupyterHub environments without needing cloud credentials. The solution’s simplicity and security made it ideal for educational settings.
The second case study addressed a university’s request for a privacy-preserving large language model (LLM) service. StackHPC developed a Helm chart to deploy open-source LLMs, integrated with Azimuth for ad-hoc testing and ArgoCD for production-grade management. The service, designed to be GDPR-compliant, provided an anonymous interface for staff and students to experiment with LLMs. Monitoring via Prometheus and Grafana ensured reliability, demonstrating how StackHPC’s stack adapts to emerging technologies while maintaining robustness.
Empowering Researchers Through Simplicity
The core takeaway from Scott and Tyler’s talk was that scientists prioritize research over infrastructure management. By offering a platform that balances infrastructure-as-a-service flexibility with platform-as-a-service simplicity, StackHPC empowers researchers to work efficiently. Their commitment to open-source, evidenced by plans to donate Zenith and Azimuth to the CNCF sandbox, underscores their dedication to community-driven innovation. The LOKI stack’s ability to abstract complexity while preserving functionality positions it as a transformative tool for scientific computing.
Links:
[KCDUK2024] From Free Kicks to Git Commits: Steve Wade’s Journey at KCDUK2024
At KCDUK2024, Steve Wade, a cloud native consultant and trainer at Jetstack, captivated the audience with a narrative of transformation, tracing his remarkable journey from professional football to a thriving career in technology. His talk, a blend of personal reflection and technical insight, illuminated the parallels between orchestrating a football team and building self-service platforms on Kubernetes. Steve’s story is one of resilience, adaptability, and the power of transferable skills, offering a compelling blueprint for navigating career pivots and leveraging past experiences to excel in the tech landscape.
From the Pitch to the Keyboard: A Career Pivot
Steve’s journey began at the tender age of four, scouted for his football potential, a moment that ignited a lifelong passion. His early career was defined by dreams of playing in iconic stadiums like Wembley or in a Champions League final. However, a devastating knee injury at 18—shattering his ACL and ligaments—halted his aspirations, plunging him into an identity crisis. Unable to walk for 18 months, Steve faced a profound setback, spiraling into depression and grappling with the question, “Who am I if not a footballer?” This period of introspection, however, became a catalyst for reinvention.
Inspired by his father’s work in IT, Steve’s curiosity was piqued. From his bed, he began exploring technology, diving into coding and the intricacies of programming languages. This marked the beginning of a significant pivot, transitioning from the physical demands of the football field to the intellectual challenges of the digital realm. His relentless drive, honed on the pitch, translated into a disciplined approach to learning, setting the foundation for his eventual expertise in cloud native technologies.
Lessons from Football: Teamwork, Empathy, and Resilience
Steve eloquently drew parallels between the skills cultivated in football and those essential in technology. On the pitch, teamwork was paramount—11 players working cohesively toward a single goal, with intricate passes leading to success. Similarly, in tech, delivering a product requires collaboration across diverse teams, where no one “scores” alone. Steve emphasized empathy as a cornerstone, recounting how supporting teammates in football translated to understanding developers’ needs in tech. Knowing when a colleague needs support, he argued, transforms good teams into great ones, a principle that resonates in both domains.
Resilience, another lesson from his sporting days, proved invaluable. Just as a footballer bounces back from a crushing defeat, Steve learned to recover from failed deployments or security breaches. Strategic planning, akin to reading an opponent’s tactics, mirrored the importance of roadmaps and architectural design in tech. Leadership, too, played a critical role—captains rallying teams on the field were akin to tech leads guiding projects to success. These transferable skills, Steve noted, were his “secret weapon” in navigating the tech industry’s challenges.
Kubernetes as the Ultimate Orchestrator
The discovery of Kubernetes marked a turning point in Steve’s career. He likened it to orchestrating a world-class football squad, where containers are players, roles are defined by role-based access control, and tactics mirror efficient orchestration. Kubernetes, to Steve, was more than a tool—it was a framework for empowering developers through self-service platforms. By providing guardrails, these platforms enable rapid decision-making without compromising stability, much like a coach guiding a team within a strategic framework.
Steve’s work at Jetstack focuses on building such platforms, drawing on the agility and collaboration learned from football. He highlighted the importance of empowering developers to move quickly within structured environments, avoiding the manual processes that bog down innovation. His role as a trainer, having empowered 8,000 individuals worldwide, reflects his commitment to fostering the next generation of cloud native engineers, a mission rooted in the collaborative spirit of his football days.
Overcoming Setbacks as Opportunities
Central to Steve’s narrative was the idea that setbacks are opportunities in disguise. His injury, though catastrophic, forced a reevaluation of his path, leading to a fulfilling career in tech. He encouraged the audience to view challenges as catalysts for growth, urging them to embrace change and maintain a mindset of continuous learning. In the fast-evolving cloud native landscape, where new projects emerge daily, Steve advised focusing on what truly matters rather than attempting to master the entire ecosystem.
He also emphasized the human element in technology. Collaboration, trust, and communication are as vital in tech as they were on the field. Steve’s creation of the Cloud Native Club, a community to support aspiring engineers, underscores his belief that success is amplified by community. By sharing his journey, he invited others to see their challenges as stepping stones, reinforcing that the journey is as significant as the destination.
Links:
[KCDUK2024] Building the Future, Together | KCDUK2024
From Neopets to Google: A Tech Journey
Cheryl Hung, a Senior Director at Arm, delivered an inspiring keynote at KCDUK2024 titled “Building the Future, Together.” Reflecting on her journey as a cloud native pioneer, Cheryl shared personal anecdotes and community insights, weaving a narrative of resilience and collaboration. Born to a non-technical family of Hong Kong immigrants, Cheryl’s entry into tech was sparked by fascination with Google’s innovative culture and early experiments with Neopets’ HTML coding.
At Cambridge, Cheryl pursued a computer science degree, driven by her ambition to join Google. Despite a failed interview at 19, she persevered, securing a role as a software engineer working on Google Maps’ C++ backend. However, burnout and a lack of fulfillment led her to quit after five years, questioning her place in tech. Cheryl’s rediscovery of passion through community meetups, particularly around Docker and containers, marked a turning point, aligning with her prior exposure to Google’s Borg infrastructure.
Embracing Developer Advocacy
Defying warnings about career risks, Cheryl embraced developer advocacy, joining a startup as employee number 12. There, she honed her public speaking skills and founded the Cloud Native London Meetup, building strong community ties. Her role as a CNCF ambassador, secured through a simple photo submission, amplified her influence, leading to her appointment as Director of Ecosystem at CNCF. Cheryl’s tenure involved extensive travel, delivering keynotes and engaging with cloud native “celebrities” like Kelsey Hightower and Liz Rice, though the 2020 pandemic disrupted this dynamic lifestyle.
Cheryl’s interactions revealed the human side of tech luminaries, from Dan Kohn’s supportive mentorship to Kelsey’s off-stage humor. These experiences underscored the importance of authentic connections in driving innovation. Her narrative highlighted the CNCF’s growth from a nascent organization to a powerhouse managing 190 projects, reflecting the community’s role in shaping cloud native technologies.
Navigating Career Transitions
Pregnancy prompted Cheryl to reassess her role at CNCF, leading to a move to Apple, followed by a swift transition to Arm as Senior Director of Ecosystem. Despite personal challenges, including back-to-back maternity leaves, Cheryl’s commitment to community remained steadfast. Her narrative emphasized the power of relationships in career navigation, as opportunities arose through trusted connections rather than formal applications.
Cheryl’s candid reflection on overcoming burnout and imposter syndrome resonated deeply, illustrating the resilience required in tech. Her move to Arm, driven by an irresistible role, underscored her belief in seizing opportunities that align with personal and professional growth. This journey of adaptation and reinvention highlighted the importance of flexibility in a rapidly evolving industry.
Building a Collaborative Future
Cheryl concluded with a call to action, urging attendees to foster community through connection. Acknowledging her struggle with facial blindness, she emphasized the courage required to engage with strangers, advocating for inclusive spaces where diverse ideas thrive. Her vision for the future centers on collective effort, encouraging practitioners to build together, leveraging events like KCDUK2024 to forge lasting relationships.
Cheryl’s narrative serves as a beacon for aspiring technologists, demonstrating that the future of cloud native lies in community-driven innovation. By sharing her vulnerabilities and triumphs, she inspired attendees to embrace collaboration, ensuring the ecosystem continues to evolve through shared knowledge and diverse perspectives.
Links:
[KCDUK2024] Charting the Course: The History and Evolution of Kubernetes Security | KCDUK2024
Early Days of Kubernetes Security
At KCDUK2024, Rory McCune, a Senior Security Advocate at Datadog, delivered a captivating session titled “Charting the Course: The History and Evolution of Kubernetes Security.” Reflecting on a decade of Kubernetes, Rory traced the platform’s security evolution, from its vulnerable beginnings to modern advancements. As a former penetration tester, Rory’s insights were grounded in real-world experiences, offering a nuanced perspective on Kubernetes’ security landscape.
In 2016, Kubernetes clusters were fraught with vulnerabilities. The Kubelet, a critical node API, lacked authentication, enabling unauthenticated command execution if network access was gained. Rory recounted discovering the “Kubelet exploit,” which allowed attackers to execute commands on every node without credentials. Similarly, the read-only Kubelet port exposed pod configurations, providing attackers with valuable reconnaissance. These early flaws underscored the nascent state of Kubernetes security, requiring manual interventions like SSH tunneling to secure clusters.
Authentication and Authorization Challenges
Rory highlighted the evolution of authentication and authorization in Kubernetes. The insecure API server port, available until version 1.20, granted cluster admin access without authentication, a flaw exploited by Rory’s team in numerous pentests. Irrevocable credentials, such as client certificates and service account tokens, posed persistent risks, as they could not be revoked without deleting critical components. These issues, exemplified by CVE-18982, persisted for years, complicating cluster security.
Modern Kubernetes has mitigated these risks through token request APIs and cloud provider authentication, enabling revocable credentials. However, Rory cautioned that client certificates and long-lived tokens remain potential vulnerabilities in some distributions, requiring careful configuration. The introduction of RBAC (Role-Based Access Control) replaced the permissive “always allow” mode, though some distributions still default to the latter, necessitating vigilance.
Pod Security and Helm Vulnerabilities
Pod security presented another challenge in early Kubernetes. Rory described the “most pointless Kubernetes pod,” a manifest granting root access to nodes, exploitable by anyone with pod creation rights. Pod Security Policies (PSPs), introduced to restrict such actions, were cumbersome and deprecated in 1.25. Modern alternatives like Pod Security Admission and Validating Admission Policies offer flexible, fine-grained controls, though none are enabled by default, requiring proactive configuration.
Rory also addressed Helm 2’s Tiller service, which ran unauthenticated within clusters, granting cluster admin access to anyone reaching it. A 2024 incident involving an SAP AI service demonstrated Tiller’s lingering presence, underscoring the need for authentication configuration. These historical vulnerabilities highlight the importance of securing auxiliary components in the Kubernetes ecosystem.
Unpatchable CVEs and Networking Complexities
Rory concluded with four unpatchable Kubernetes CVEs, each illustrating networking complexities. CVE-2020-8554 allows traffic hijacking in multi-tenant clusters via misconfigured service objects, mitigated by admission controllers. CVE-2020-8555 and CVE-2021-25741 exploit server-side request forgery (SSRF), leveraging the API server’s proxy nature to access restricted networks. CVE-2021-25662 enables namespace bypass via shared load balancers, a risk in multi-tenant environments.
These CVEs underscore Kubernetes’ intricate networking model, where the API server’s proxy capabilities and IP tables rules create potential vulnerabilities. Rory emphasized the need for tailored configurations, such as connectivity services for managed Kubernetes, to mitigate these risks. His narrative highlighted the platform’s security progress while acknowledging ongoing challenges, particularly in multi-tenant scenarios.