Posts Tagged ‘Kubernetes’
[KCDUK2024] Building the Future, Together | KCDUK2024
From Neopets to Google: A Tech Journey
Cheryl Hung, a Senior Director at Arm, delivered an inspiring keynote at KCDUK2024 titled “Building the Future, Together.” Reflecting on her journey as a cloud native pioneer, Cheryl shared personal anecdotes and community insights, weaving a narrative of resilience and collaboration. Born to a non-technical family of Hong Kong immigrants, Cheryl’s entry into tech was sparked by fascination with Google’s innovative culture and early experiments with Neopets’ HTML coding.
At Cambridge, Cheryl pursued a computer science degree, driven by her ambition to join Google. Despite a failed interview at 19, she persevered, securing a role as a software engineer working on Google Maps’ C++ backend. However, burnout and a lack of fulfillment led her to quit after five years, questioning her place in tech. Cheryl’s rediscovery of passion through community meetups, particularly around Docker and containers, marked a turning point, aligning with her prior exposure to Google’s Borg infrastructure.
Embracing Developer Advocacy
Defying warnings about career risks, Cheryl embraced developer advocacy, joining a startup as employee number 12. There, she honed her public speaking skills and founded the Cloud Native London Meetup, building strong community ties. Her role as a CNCF ambassador, secured through a simple photo submission, amplified her influence, leading to her appointment as Director of Ecosystem at CNCF. Cheryl’s tenure involved extensive travel, delivering keynotes and engaging with cloud native “celebrities” like Kelsey Hightower and Liz Rice, though the 2020 pandemic disrupted this dynamic lifestyle.
Cheryl’s interactions revealed the human side of tech luminaries, from Dan Kohn’s supportive mentorship to Kelsey’s off-stage humor. These experiences underscored the importance of authentic connections in driving innovation. Her narrative highlighted the CNCF’s growth from a nascent organization to a powerhouse managing 190 projects, reflecting the community’s role in shaping cloud native technologies.
Navigating Career Transitions
Pregnancy prompted Cheryl to reassess her role at CNCF, leading to a move to Apple, followed by a swift transition to Arm as Senior Director of Ecosystem. Despite personal challenges, including back-to-back maternity leaves, Cheryl’s commitment to community remained steadfast. Her narrative emphasized the power of relationships in career navigation, as opportunities arose through trusted connections rather than formal applications.
Cheryl’s candid reflection on overcoming burnout and imposter syndrome resonated deeply, illustrating the resilience required in tech. Her move to Arm, driven by an irresistible role, underscored her belief in seizing opportunities that align with personal and professional growth. This journey of adaptation and reinvention highlighted the importance of flexibility in a rapidly evolving industry.
Building a Collaborative Future
Cheryl concluded with a call to action, urging attendees to foster community through connection. Acknowledging her struggle with facial blindness, she emphasized the courage required to engage with strangers, advocating for inclusive spaces where diverse ideas thrive. Her vision for the future centers on collective effort, encouraging practitioners to build together, leveraging events like KCDUK2024 to forge lasting relationships.
Cheryl’s narrative serves as a beacon for aspiring technologists, demonstrating that the future of cloud native lies in community-driven innovation. By sharing her vulnerabilities and triumphs, she inspired attendees to embrace collaboration, ensuring the ecosystem continues to evolve through shared knowledge and diverse perspectives.
Links:
[DevoxxGR2025] Optimized Kubernetes Scaling with Karpenter
Alex König, an AWS expert, delivered a 39-minute talk at Devoxx Greece 2025, exploring how Karpenter enhances Kubernetes cluster autoscaling for speed, cost-efficiency, and availability.
Karpenter’s Dynamic Autoscaling
König introduced Karpenter as an open-source, Kubernetes-native autoscaling solution, contrasting it with the traditional Cluster Autoscaler. Unlike the latter, which relies on uniform node groups (e.g., nodes with four CPUs and 16GB RAM), Karpenter uses the EC2 Fleet API to dynamically provision nodes tailored to workload needs. For instance, if a pod requires one CPU, Karpenter allocates a node with minimal excess capacity, avoiding resource waste. This right-sizing, combined with groupless scaling, enables faster and more cost-effective scaling, especially in dynamic environments.
Ensuring Availability with Constraints
König addressed availability challenges reported by users, emphasizing Kubernetes-native scheduling constraints to mitigate disruptions. Topology spread constraints distribute pods across availability zones, reducing the risk of downtime if a node fails. Pod disruption budgets, affinity/anti-affinity rules, and priority classes further ensure critical workloads are scheduled appropriately. For stateful workloads using EBS, König recommended setting the volume binding mode to “wait for first consumer” to avoid pod-volume mismatches across zones, preventing crashes and ensuring reliability.
Integrating with KEDA for Application Scaling
For advanced scaling, König highlighted combining Karpenter with KEDA for event-driven, application-specific scaling. KEDA scales pods based on metrics like Kafka topic sizes or SQS queues, beyond CPU/memory. Karpenter then provisions nodes for pending pods, enabling seamless scaling for workloads like flash sales. König outlined a four-step migration from Cluster Autoscaler to Karpenter, emphasizing its simplicity and open-source documentation.
Links
[KCDUK2024] Charting the Course: The History and Evolution of Kubernetes Security | KCDUK2024
Early Days of Kubernetes Security
At KCDUK2024, Rory McCune, a Senior Security Advocate at Datadog, delivered a captivating session titled “Charting the Course: The History and Evolution of Kubernetes Security.” Reflecting on a decade of Kubernetes, Rory traced the platform’s security evolution, from its vulnerable beginnings to modern advancements. As a former penetration tester, Rory’s insights were grounded in real-world experiences, offering a nuanced perspective on Kubernetes’ security landscape.
In 2016, Kubernetes clusters were fraught with vulnerabilities. The Kubelet, a critical node API, lacked authentication, enabling unauthenticated command execution if network access was gained. Rory recounted discovering the “Kubelet exploit,” which allowed attackers to execute commands on every node without credentials. Similarly, the read-only Kubelet port exposed pod configurations, providing attackers with valuable reconnaissance. These early flaws underscored the nascent state of Kubernetes security, requiring manual interventions like SSH tunneling to secure clusters.
Authentication and Authorization Challenges
Rory highlighted the evolution of authentication and authorization in Kubernetes. The insecure API server port, available until version 1.20, granted cluster admin access without authentication, a flaw exploited by Rory’s team in numerous pentests. Irrevocable credentials, such as client certificates and service account tokens, posed persistent risks, as they could not be revoked without deleting critical components. These issues, exemplified by CVE-18982, persisted for years, complicating cluster security.
Modern Kubernetes has mitigated these risks through token request APIs and cloud provider authentication, enabling revocable credentials. However, Rory cautioned that client certificates and long-lived tokens remain potential vulnerabilities in some distributions, requiring careful configuration. The introduction of RBAC (Role-Based Access Control) replaced the permissive “always allow” mode, though some distributions still default to the latter, necessitating vigilance.
Pod Security and Helm Vulnerabilities
Pod security presented another challenge in early Kubernetes. Rory described the “most pointless Kubernetes pod,” a manifest granting root access to nodes, exploitable by anyone with pod creation rights. Pod Security Policies (PSPs), introduced to restrict such actions, were cumbersome and deprecated in 1.25. Modern alternatives like Pod Security Admission and Validating Admission Policies offer flexible, fine-grained controls, though none are enabled by default, requiring proactive configuration.
Rory also addressed Helm 2’s Tiller service, which ran unauthenticated within clusters, granting cluster admin access to anyone reaching it. A 2024 incident involving an SAP AI service demonstrated Tiller’s lingering presence, underscoring the need for authentication configuration. These historical vulnerabilities highlight the importance of securing auxiliary components in the Kubernetes ecosystem.
Unpatchable CVEs and Networking Complexities
Rory concluded with four unpatchable Kubernetes CVEs, each illustrating networking complexities. CVE-2020-8554 allows traffic hijacking in multi-tenant clusters via misconfigured service objects, mitigated by admission controllers. CVE-2020-8555 and CVE-2021-25741 exploit server-side request forgery (SSRF), leveraging the API server’s proxy nature to access restricted networks. CVE-2021-25662 enables namespace bypass via shared load balancers, a risk in multi-tenant environments.
These CVEs underscore Kubernetes’ intricate networking model, where the API server’s proxy capabilities and IP tables rules create potential vulnerabilities. Rory emphasized the need for tailored configurations, such as connectivity services for managed Kubernetes, to mitigate these risks. His narrative highlighted the platform’s security progress while acknowledging ongoing challenges, particularly in multi-tenant scenarios.
Links:
[DevoxxGR2024] The Art of Debugging Inside K8s Environment at Devoxx Greece 2024 by Andrii Soldatenko
At Devoxx Greece 2024, Andrii Soldatenko, a seasoned software engineer and tech evangelist at Dynatrace, delivered an engaging presentation on mastering the art of debugging within Kubernetes (K8s) environments. With a blend of humor, practical insights, and real-world strategies, Andrii illuminated the complexities of troubleshooting cloud-native applications. Drawing from his extensive experience, he provided actionable techniques to enhance debugging efficiency, making the session a valuable resource for developers navigating the intricacies of Kubernetes. His talk emphasized proactive design, robust tooling, and a systematic approach to resolving issues in distributed systems.
The Challenges of Debugging in Kubernetes
Andrii began by acknowledging the inherent difficulties of debugging in modern cloud-native environments. Unlike traditional development, where a local debugger suffices, Kubernetes introduces layers of complexity with containers, pods, and distributed architectures. He humorously outlined his “eight stages of debugging,” from denial (“this can’t happen”) to self-realization (“I wrote this code”), resonating with developers who face similar emotional journeys. These stages underscore the psychological and technical hurdles of troubleshooting in K8s, where issues often stem from accidental complexities like misconfigured resources or network policies.
The dynamic nature of Kubernetes, with its orchestration of pods, nodes, and services, demands a shift in debugging mindset. Andrii emphasized that while writing YAML manifests for K8s is straightforward, ensuring they function as intended is not. He highlighted the absence of comprehensive debugging guides, noting that most literature focuses on deployment rather than troubleshooting. This gap inspired his talk, which aimed to equip developers with practical strategies to diagnose and resolve issues effectively.
Strategies for Effective Debugging
To tackle Kubernetes debugging, Andrii proposed a structured approach, starting with a high-level mind map for assessing pod states. For instance, a pod in a “Pending” state might indicate resource shortages or port conflicts, while a “Crashing” pod could signal health probe failures. He focused on scenarios where pods are running but behaving unexpectedly, a common yet challenging issue. Andrii advocated revisiting init containers, which perform setup tasks like data migrations. By temporarily replacing their commands with a sleep directive, developers can use kubectl exec to inspect the container’s state, checking volumes, permissions, or network access.
For containers lacking debugging tools, Andrii introduced ephemeral containers, a Kubernetes feature since version 1.8 designed for interactive troubleshooting. By launching an ephemeral container with tools like netcat or a debugger, developers can inspect a pod’s state without altering its primary container. He shared a practical example of debugging a Go application by sharing process namespaces, allowing access to the application’s processes. This approach enables setting breakpoints and navigating code, even in minimal, distroless containers.
Leveraging Tools for Enhanced Debugging
Andrii showcased several tools to streamline Kubernetes debugging. He recommended building custom debug containers tailored to specific needs, such as including sqlite, python, or network utilities, and shared his own debug container on GitHub. For network-related issues, he highlighted a pre-existing container with tools like tcpdump, which simplifies packet inspection without requiring manual installations. Andrii also praised Stern, a CLI tool for tailing logs across multiple pods in a replica set, making it easier to trace requests and identify exceptions.
For developers using Visual Studio Code, Andrii demonstrated remote debugging by configuring a launch.json file to connect to a Kubernetes pod. By exposing a debug port and using tools like Telepresence, developers can intercept cluster traffic and test changes locally, bypassing slow CI/CD cycles. He also highlighted K9s, a terminal-based UI for Kubernetes, with a custom plugin for initiating debug sessions via kubectl debug. These tools collectively enhance efficiency, allowing developers to focus on problem-solving rather than manual configuration.
Best Practices for Proactive Debugging
Andrii concluded with actionable best practices to prevent and address debugging challenges. He stressed embedding version information, like Git commit SHAs, into container images to synchronize codebases during remote debugging. Scaling down traffic to a single pod ensures consistent debugging sessions, avoiding request distribution across replicas. He also advocated for a blameless culture, where developers use debuggers to slow down and analyze issues methodically rather than rushing to fix symptoms.
By sharing his GitHub repository and additional resources, Andrii encouraged attendees to experiment with these techniques. His talk was a compelling call to action for developers to embrace robust debugging practices, ensuring resilience and reliability in Kubernetes environments. Through practical demonstrations and a lighthearted approach, he demystified the complexities of cloud-native debugging, empowering developers to tackle issues with confidence.
Links:
[KCDUK2024] Sustainability Chronicles: Innovate Through Green Technology With Kepler and KEDA | KCDUK2024
The Imperative of Cloud Sustainability
At KCDUK2024, Katie Gamanji, a Senior Field Engineer at Apple and a prominent figure in the Cloud Native Computing Foundation (CNCF), delivered a compelling session titled “Sustainability Chronicles: Innovate Through Green Technology With Kepler and KEDA.” Katie’s presentation addressed the critical need for environmental consciousness in the tech sector, particularly within the cloud native ecosystem. With a decade of Kubernetes driving industry transformation, Katie emphasized the urgency of integrating sustainability into technological decision-making, given the tech sector’s 1.4% contribution to global greenhouse gas emissions.
Katie outlined the global context, referencing the Paris Agreement (COP21) and the United Nations’ Sustainable Development Goals (SDG13), which call for proactive climate action. She highlighted that adopting renewable energy could reduce tech emissions by 80%, a potential that major cloud providers like GCP, AWS, and Azure are pursuing through net-zero targets. Katie’s narrative framed sustainability as an integral part of operational efficiency, introducing the concept of GreenOps, which aligns resource optimization with reduced environmental impact.
Measuring Emissions with Kepler
Katie introduced Kepler, a CNCF sandbox project developed by Red Hat and IBM, as a pivotal tool for measuring application emissions in Kubernetes. Kepler leverages eBPF to collect energy consumption data, converting it into CO2 emissions using hardcoded emission factors for coal, petroleum, and natural gas. Deployed on a local Kind cluster, Kepler’s Grafana dashboards provide granular insights into emissions per container, day, and namespace. This capability enables organizations to identify high-emission workloads and implement targeted optimizations.
Katie encouraged community engagement with Kepler, noting its sandbox status and potential for growth. By providing feedback or contributing features, practitioners can enhance Kepler’s utility, fostering a collaborative approach to sustainability. Her demonstration underscored the importance of observability in addressing emissions, setting the stage for proactive workload management.
Scaling with KEDA and Carbon-Aware Operator
To move beyond measurement, Katie introduced KEDA (Kubernetes Event-Driven Autoscaler) and its Carbon-Aware Operator, which optimize application scaling based on carbon intensity. KEDA, a graduated CNCF project, scales workloads in response to external events, such as carbon intensity data from grid providers like WattTime or Electricity Map. The Carbon-Aware Operator adjusts replica counts inversely to carbon intensity, maximizing replicas when renewable energy is abundant and minimizing them during high-emission periods.
Katie’s local deployment showcased this dynamic scaling, visualized through Grafana, where low carbon intensity correlated with higher replica counts. This approach exemplifies proactive sustainability, aligning computational demands with environmental impact. Katie’s advocacy for KEDA highlighted its evolution from a niche solution to an industry standard, encouraging practitioners to explore its potential for green innovation.
Community-Driven Sustainability
Beyond technology, Katie emphasized the role of community in driving sustainability. The Kubernetes community’s inclusive ethos fosters innovation through open governance and diverse contributions. She highlighted the CNCF’s Technical Advisory Group (TAG) for Environmental Sustainability, which promotes green practices across projects. Initiatives like the Green Reviews Working Group and sustainability white papers offer avenues for engagement, encouraging practitioners to contribute beyond code.
Katie’s call to action urged attendees to diversify technical boards and create welcoming spaces for new ideas. Her narrative wove together technological innovation and community collaboration, positioning sustainability as a shared responsibility. By integrating tools like Kepler and KEDA with community efforts, the cloud native ecosystem can lead in green technology, shaping a sustainable future.
Links:
[KCD UK 2024] Deep Dive into Kubernetes Runtime Security
Saeid Bostandoust, founder of CubeDemi.io, delivered an in-depth presentation at KCDUK2024 on Kubernetes runtime security, focusing on tools and techniques to secure containers during execution. As a CNCF project contributor, Saeid explored Linux security features like Linux Capabilities, SELinux, AppArmor, Seccomp-BPF, and KubeArmor, providing a comprehensive overview of how these can be applied in Kubernetes to mitigate zero-day attacks and enforce policies. His talk emphasized practical implementation, observability, and policy enforcement, aligning with KCDUK2024’s focus on securing cloud-native environments.
Understanding Runtime Security
Saeid defined runtime security as protecting applications during execution, contrasting it with pre-runtime measures like static code analysis. Runtime security focuses on mitigating zero-day attacks and malicious behavior through real-time intrusion detection, process isolation, policy enforcement, and monitoring. Linux offers over 30 security mechanisms, including SELinux, AppArmor, Linux Capabilities, Seccomp-BPF, and namespaces, alongside kernel drivers to counter threats like Meltdown and Spectre. Saeid focused on well-known features, explaining their roles and Kubernetes integration.
Linux Capabilities: Historically, processes were either privileged (with full root permissions) or unprivileged, leading to vulnerabilities like privilege escalation via commands like ping. Linux Capabilities, introduced to granularly assign permissions, allow processes to perform specific actions (e.g., opening raw sockets for ping) without full root privileges. In Kubernetes, capabilities can be configured in pod manifests to drop unnecessary permissions, enhancing security even for root-run containers.
Seccomp-BPF: Seccomp (Secure Computing) restricts system calls a process can make. Originally limited to basic calls (read, write, exit), Seccomp-BPF extends this with customizable profiles. In Kubernetes, a Seccomp-BPF profile can be defined in a JSON file and applied via a pod’s security context, terminating processes that attempt unauthorized system calls. Saeid demonstrated a restrictive profile that limits a container to basic operations, preventing it from running if additional system calls are needed.
LSM Modules (AppArmor, SELinux, BPF-LSM): Linux Security Modules (LSM) provide hooks to intercept operations, such as file access or network communication. AppArmor uses path-based profiles, while SELinux employs label-based policies. BPF-LSM, a newer option, allows dynamic policy injection via eBPF code, offering flexibility without requiring application restarts. Saeid noted that BPF-LSM, available in kernel versions 5.7 and later, supports stacking with other LSMs, enhancing Kubernetes security.
KubeArmor: Simplifying Policy Enforcement
KubeArmor, a CNCF project, simplifies runtime security by allowing users to define policies via Kubernetes Custom Resource Definitions (CRDs). These policies are translated into AppArmor, SELinux, or BPF-LSM profiles, depending on the worker node’s LSM. KubeArmor addresses the challenge of syncing profiles across cluster nodes, automating deployment and updates. It uses eBPF for observability, monitoring system calls and generating telemetry for tools like Prometheus and Elasticsearch. Saeid showcased KubeArmor’s architecture, including a daemon set with an init container for compiling eBPF code and a relay server for aggregating logs and alerts.
An example policy demonstrated KubeArmor denying access to sensitive files (e.g., /etc/passwd, /etc/shadow) and commands (e.g., apt, apt-get), with logs showing enforcement details. Unlike manual AppArmor or SELinux profiles, which are complex and hard to scale, KubeArmor’s declarative approach and default deny policies simplify securing containers, preventing access to dangerous assets like /proc mounts.
Practical Implementation and Community Engagement
Saeid provided practical examples, such as configuring a pod to drop all capabilities except those needed, applying a Seccomp-BPF profile to restrict system calls, and using KubeArmor to enforce file access policies. He highlighted KubeArmor’s integration with tools like OPA Gatekeeper to block unauthorized commands (e.g., kubectl exec) when BPF-LSM is unavailable. For further learning, Saeid offered a 50% discount on CubeDemi.io’s container security workshop, encouraging KCDUK2024 attendees to deepen their Kubernetes security expertise.
Links:
Hashtags: #Kubernetes #RuntimeSecurity #KubeArmor #CNCF #LinuxSecurity #SaeidBostandoust #KCDUK2024
[KCDUK2024] An Odyssey with ArgoCD: From Git to Helm | KCDUK2024
Introduction to GitOps and ArgoCD
In the ever-evolving landscape of Kubernetes management, GitOps has emerged as a cornerstone for streamlined application deployment. At KCDUK2024, Farah Adbib and Antonio Alferez, both esteemed professionals from The Workshop, delivered an insightful session titled “An Odyssey with ArgoCD: From Git to Helm.” Their talk elucidated the journey of implementing GitOps using ArgoCD within their organization, navigating through initial challenges and innovative solutions. Farah, a DevOps Solution Architect, and Antonio, a Platform Engineer, shared their expertise on leveraging Git repositories and Helm charts to manage Kubernetes clusters efficiently, offering a narrative rich with practical insights.
The session began with an overview of their ecosystem, managing 55 Kubernetes clusters across public cloud and on-premises infrastructure, supporting 1,600 nodes. These clusters cater to 350 software engineers across various business units, each responsible for deploying their applications. The need for simplicity and security in deployment processes was paramount, given the isolation requirements for some clusters. Farah and Antonio’s narrative underscored the importance of aligning technological solutions with organizational needs, setting the stage for their exploration of ArgoCD’s capabilities.
Initial Approach: Git as the Source of Truth
Initially, Farah and Antonio adopted Git repositories as the primary source for ArgoCD, a logical choice given GitOps’ emphasis on declarative configuration. They opted for a decentralized approach, deploying an ArgoCD instance per cluster to meet stringent security and isolation requirements. Each application’s Helm chart was separated from the application code, stored in distinct Git repositories to avoid replication across data centers. This separation was driven by the differing lifecycles of application code and Helm charts, aiming to streamline management.
However, this approach revealed several pain points. The necessity for Git mirrors introduced additional infrastructure complexity, requiring maintenance and coordination between development and operations teams. The use of Git branches as target revisions led to confusion, as development and operational branches coexisted, complicating version control. Moreover, rendering Helm charts within ArgoCD itself delayed feedback loops, causing failures late in the pipeline. Farah and Antonio’s candid reflection on these challenges highlighted the need for a more robust solution, prompting a strategic pivot.
Evolving to Helm Rendered Manifests
Recognizing the limitations of their initial setup, Farah and Antonio transitioned to a Helm-rendered manifest pattern. This innovative approach involved pre-rendering Helm manifests in the CI/CD pipeline and storing them in an artifact registry, such as Nexus, rather than Git mirrors. By pointing ArgoCD to these pre-rendered manifests, they eliminated the need for ArgoCD to handle rendering, significantly reducing complexity and accelerating feedback loops. This shift also unified application code and Helm charts into a single repository, simplifying versioning and pipeline management.
A key enhancement was the introduction of an external Helm values repository per cluster, allowing operational changes like scaling without triggering a full pipeline rebuild. This decoupling of configuration from code enhanced flexibility and developer experience. Antonio emphasized the adoption of semantic versioning (SemVer) for both Docker images and Helm charts, incorporating commit hashes for traceability. This meticulous versioning strategy ensured clarity on deployed versions across 55 clusters, leveraging Git notes for additional auditing information.
Lessons Learned and Developer Experience
The transition to Helm-rendered manifests yielded significant improvements. By moving rendering logic to the CI/CD pipeline, Farah and Antonio achieved a “shift-left” approach, enabling earlier failure detection and faster iterations. The elimination of Git mirrors reduced infrastructure overhead, while unified repositories streamlined development workflows. The external Helm values repository facilitated rapid operational adjustments, enhancing agility.
Farah and Antonio underscored the importance of tailoring solutions to the organizational ecosystem. Their journey highlighted the pitfalls of adhering to industry defaults without considering specific requirements. They emphasized that developer experience is paramount, advocating for solutions that empower engineers while minimizing management overhead. Their narrative serves as a testament to the value of iterative improvement, encouraging practitioners to reassess and redesign when initial solutions fall short.
Links:
[DevoxxGR2024] Devoxx Greece 2024 Sustainability Chronicles: Innovate Through Green Technology With Kepler and KEDA
At Devoxx Greece 2024, Katie Gamanji, a senior field engineer at Apple and a technical oversight committee member for the Cloud Native Computing Foundation (CNCF), delivered a compelling presentation on advancing environmental sustainability within the cloud-native ecosystem. With Kubernetes celebrating its tenth anniversary, Katie emphasized the urgent need for technologists to integrate green practices into their infrastructure strategies. Her talk explored how tools like Kepler and KEDA’s carbon-aware operator enable practitioners to measure and mitigate carbon emissions, while fostering a vibrant, inclusive community to drive these efforts forward. Drawing from her extensive experience and leadership in the CNCF, Katie provided a roadmap for aligning technological innovation with climate responsibility.
The Imperative of Cloud Sustainability
Katie began by underscoring the critical role of sustainability in the tech sector, particularly given the industry’s contribution to global greenhouse gas emissions. She highlighted that the tech sector accounts for 1.4% of global emissions, a figure that could soar to 10% within a decade without intervention. However, by leveraging renewable energy, emissions could be reduced by up to 80%. International agreements like COP21 and the United Nations’ Sustainable Development Goals (SDGs) have spurred national regulations, compelling organizations to assess and report their carbon footprints. Major cloud providers, such as Google Cloud Platform (GCP), have set ambitious net-zero targets, with GCP already operating on renewable energy since 2022. Yet, Katie stressed that sustainability cannot be outsourced solely to cloud providers; organizations must embed these principles internally.
The emergence of “GreenOps,” inspired by FinOps, encapsulates the processes, tools, and cultural shifts needed to achieve digital sustainability. By optimizing infrastructure—through strategies like using spot instances or serverless architectures—organizations can reduce both costs and emissions. Katie introduced a four-phase strategy proposed by the FinOps Foundation’s Environmental Sustainability Working Group: awareness, discovery, roadmap, and execution. This framework encourages organizations to educate stakeholders, benchmark emissions, implement automated tools, and iteratively pursue ambitious sustainability goals.
Measuring Emissions with Kepler
To address emissions within Kubernetes clusters, Katie introduced Kepler, a CNCF sandbox project developed by Red Hat and IBM. Kepler, a Kubernetes Efficient Power Level Exporter, utilizes eBPF to probe system statistics and export power consumption metrics to Prometheus for visualization in tools like Grafana. Deployed as a daemon set, Kepler collects node- and container-level metrics, focusing on power usage and resource utilization. By tracing CPU performance counters and Linux kernel trace points, it calculates energy consumption in joules, converting this to kilowatt-hours and multiplying by region-specific emission factors for gases like coal, petroleum, and natural gas.
Katie demonstrated Kepler’s practical application using a Grafana dashboard, which displayed emissions per gas and allowed granular analysis by container, day, or namespace. This visibility enables organizations to identify high-emission components, such as during traffic spikes, and optimize accordingly. As a sandbox project, Kepler is gaining momentum, and Katie encouraged attendees to explore it, provide feedback, or contribute to its development, reinforcing its potential to establish a baseline for carbon accounting in cloud-native environments.
Scaling Sustainably with KEDA’s Carbon-Aware Operator
Complementing Kepler’s observational capabilities, Katie introduced KEDA (Kubernetes Event-Driven Autoscaler), a graduated CNCF project, and its carbon-aware operator. KEDA, created by Microsoft and Red Hat, scales applications based on external events, offering a rich catalog of triggers. The carbon-aware operator optimizes emissions by scaling applications according to carbon intensity—grams of CO2 equivalent emitted per kilowatt-hour consumed. In scenarios where infrastructure is powered by renewable sources like solar or wind, carbon intensity approaches zero, allowing for maximum application replicas. Conversely, high carbon intensity, such as from coal-based energy, prompts scaling down to minimize emissions.
Katie illustrated this with a custom resource definition (CRD) that configures scaling behavior based on carbon intensity forecasts from providers like WattTime or Electricity Maps. In her demo, a Grafana dashboard showed an application scaling from 15 replicas at a carbon intensity of 530 to a single replica at 580, dynamically responding to grid data. This proactive approach ensures sustainability is embedded in scheduling decisions, aligning resource usage with environmental impact.
Nurturing a Sustainable Community
Beyond technology, Katie emphasized the pivotal role of the Kubernetes community in driving sustainability. Operating on principles of inclusivity, open governance, and transparency, the community fosters innovation through technical advisory groups (TAGs) focused on domains like observability, security, and environmental sustainability. The TAG Environmental Sustainability, established just over a year ago, aims to benchmark emissions across graduated CNCF projects, raising awareness and encouraging greener practices.
To sustain this momentum, Katie highlighted the need for education and upskilling. Resources like the Kubernetes and Cloud Native Associate (KCNA) certification and her own Cloud Native Fundamentals course on Udacity lower entry barriers for newcomers. By diversifying technical and governing boards, the community can continue to evolve, ensuring it scales alongside technological advancements. Katie’s vision is a cloud-native ecosystem where innovation and sustainability coexist, supported by a nurturing, inclusive community.
Conclusion
Katie Gamanji’s presentation at Devoxx Greece 2024 was a clarion call for technologists to prioritize environmental sustainability. By leveraging tools like Kepler and KEDA’s carbon-aware operator, practitioners can measure and mitigate emissions within Kubernetes clusters, aligning infrastructure with climate goals. Equally important is the community’s role in fostering education, inclusivity, and collaboration to sustain these efforts. Katie’s insights, grounded in her leadership at Apple and the CNCF, offer a blueprint for innovating through green technology while building a resilient, forward-thinking ecosystem.
Links:
[DevoxxBE2023] Securing the Supply Chain for Your Java Applications by Thomas Vitale
At Devoxx Belgium 2023, Thomas Vitale, a software engineer and architect at Systematic, delivered an authoritative session on securing the software supply chain for Java applications. As the author of Cloud Native Spring in Action and a passionate advocate for cloud-native technologies, Thomas provided a comprehensive exploration of securing every stage of the software lifecycle, from source code to deployment. Drawing on the SLSA framework and CNCF research, he demonstrated practical techniques for ensuring integrity, authenticity, and resilience using open-source tools like Gradle, Sigstore, and Kyverno. Through a blend of theoretical insights and live demonstrations, Thomas illuminated the critical importance of supply chain security in today’s threat landscape.
Safeguarding Source Code with Git Signatures
Thomas began by defining the software supply chain as the end-to-end process of delivering software, encompassing code, dependencies, tools, practices, and people. He emphasized the risks at each stage, starting with source code. Using Git as an example, Thomas highlighted its audit trail capabilities but cautioned that commit authorship can be manipulated. In a live demo, he showed how he could impersonate a colleague by altering Git’s username and email, underscoring the need for signed commits. By enforcing signed commits with GPG or SSH keys—or preferably a keyless approach via GitHub’s single sign-on—developers can ensure commit authenticity, establishing a verifiable provenance trail critical for supply chain security.
Managing Dependencies with Software Bills of Materials (SBOMs)
Moving to dependencies, Thomas stressed the importance of knowing exactly what libraries are included in a project, especially given vulnerabilities like Log4j. He introduced Software Bills of Materials (SBOMs) as a standardized inventory of software components, akin to a list of ingredients. Using the CycloneDX plugin for Gradle, Thomas demonstrated generating an SBOM during the build process, which provides precise dependency details, including versions, licenses, and hashes for integrity verification. This approach, integrated into Maven or Gradle, ensures accuracy over post-build scanning tools like Snyk, enabling developers to identify vulnerabilities, check license compliance, and verify component integrity before production.
Thomas further showcased Dependency-Track, an OWASP project, to analyze SBOMs and flag vulnerabilities, such as a critical issue in SnakeYAML. He introduced the Vulnerability Exploitability Exchange (VEX) standard, which complements SBOMs by documenting whether vulnerabilities affect an application. In his demo, Thomas marked a SnakeYAML vulnerability as a false positive due to Spring Boot’s safe deserialization, demonstrating how VEX communicates security decisions to stakeholders, reducing unnecessary alerts and ensuring compliance with emerging regulations.
Building Secure Artifacts with Reproducible Builds
The build phase, Thomas explained, is another critical juncture for security. Using Spring Boot as an example, he outlined three packaging methods: JAR files, native executables, and container images. He critiqued Dockerfiles for introducing non-determinism and maintenance overhead, advocating for Cloud Native Buildpacks as a reproducible, secure alternative. In a demo, Thomas built a container image with Buildpacks, highlighting its fixed creation timestamp (January 1, 1980) to ensure identical outputs for unchanged inputs, enhancing security by eliminating variability. This reproducibility, coupled with SBOM generation during the build, ensures artifacts are both secure and traceable.
Signing and Verifying Artifacts with SLSA
To ensure artifact integrity, Thomas introduced the SLSA framework, which provides guidelines for securing software artifacts across the supply chain. He demonstrated signing container images with Sigstore’s Cosign tool, using a keyless approach to avoid managing private keys. This process, integrated into a GitHub Actions pipeline, ensures that artifacts are authentically linked to their creator. Thomas further showcased SLSA’s provenance generation, which documents the artifact’s origin, including the Git commit hash and build steps. By achieving SLSA Level 3, his pipeline provided non-falsifiable provenance, ensuring traceability from source code to deployment.
Securing Deployments with Policy Enforcement
The final stage, deployment, requires validating artifacts to ensure they meet security standards. Thomas demonstrated using Cosign and the SLSA Verifier to validate signatures and provenance, ensuring only trusted artifacts are deployed. On Kubernetes, he introduced Kyverno, a policy engine that enforces signature and provenance checks, automatically rejecting non-compliant deployments. This approach ensures that production environments remain secure, aligning with the principle of validating metadata to prevent unauthorized or tampered artifacts from running.
Conclusion: A Holistic Approach to Supply Chain Security
Thomas’s session at Devoxx Belgium 2023 provided a robust framework for securing Java application supply chains. By addressing source code integrity, dependency management, build reproducibility, artifact signing, and deployment validation, he offered a comprehensive strategy to mitigate risks. His practical demonstrations, grounded in open-source tools and standards like SLSA and VEX, empowered developers to adopt these practices without overwhelming complexity. Thomas’s emphasis on asking “why” at each step encouraged attendees to tailor security measures to their context, ensuring both compliance and resilience in an increasingly regulated landscape.
Links:
[SpringIO2022] How to foster a Culture of Resilience
Benjamin Wilms, founder of Steadybit, delivered a compelling session at Spring I/O 2022, exploring how to build a culture of resilience through chaos engineering. Drawing from his experience and the evolution of chaos engineering since his 2019 Spring I/O talk, Benjamin emphasized proactive strategies to enhance system reliability. His presentation combined practical demonstrations with a framework for integrating resilience into development workflows, advocating for collaboration and automation.
Understanding Resilience and Chaos Engineering
Benjamin began by defining resilience as the outcome of well-architected, automated, and thoroughly tested systems capable of recovering from faults while delivering customer value. Unlike traditional stability, resilience involves handling partial outages with fallbacks or alternatives, ensuring service continuity. He introduced chaos engineering as a method to test this resilience by intentionally injecting faults—latency, exceptions, or service outages—to build confidence in system capabilities.
Chaos engineering involves defining a steady state (e.g., successful Netflix play button clicks), forming hypotheses (e.g., surviving a payment service outage), and running experiments to verify outcomes. Benjamin highlighted its evolution from a niche practice at Netflix to a growing community discipline, but noted its time-intensive nature often deters teams. He stressed that resilience extends beyond systems to organizational responsiveness, such as detecting incidents in seconds rather than minutes.
Pitfalls of Ad-Hoc Chaos Engineering
To illustrate common mistakes, Benjamin demonstrated a flawed approach using a Kubernetes-based microservice system with a gateway and three backend services. Running a random “delete pod” attack on the hotel service caused errors in the gateway’s product list aggregation, visible in a demo UI. However, the experiment yielded little insight, as it only confirmed the attack’s impact without actionable learnings. He critiqued such ad-hoc attacks—using tools like Pumbaa—for disrupting workflows and requiring expertise in CI/CD integration, diverting focus from core development.
This approach fails to generate knowledge or improve systems, often becoming a “rabbit hole” of additional work. Benjamin argued that starting with tools or attacks, rather than clear objectives, undermines the value of chaos engineering, leaving teams with vague results and no clear path to enhancement.
Building a Culture of Resilience
Benjamin proposed a structured approach to foster resilience, starting with the “why”: understanding motivations like surviving AWS zone outages or ensuring checkout services handle payment downtimes. The “what” involves defining specific capabilities, such as maintaining 95% request success during pod failures or implementing retry patterns. He advocated encoding these capabilities as policies—code-based checks integrated into the development pipeline.
In a demo, Benjamin showed how to define a policy for the gateway service, specifying pod redundancy and steady-state checks via a product list endpoint. The policy, stored in the codebase, runs in a CI/CD pipeline (e.g., GitHub Actions) on a staging environment, verifying resilience after each commit. This automation ensures continuous validation without manual intervention, embedding resilience into daily workflows. Policies include pre-built experiments from communities (e.g., Zalando) or static weak spot checks, like missing Kubernetes readiness probes, making resilience accessible to all developers.
Organizational Strategies and Community Impact
Benjamin addressed organizational adoption, suggesting a central component to schedule experiments and avoid overlapping tests in shared environments. For consulting scenarios, he recommended analyzing past incidents to demonstrate resilience gaps, such as running experiments to recreate outages. He shared a case where a client’s system collapsed during a rolling update under load, underscoring the need for combined testing scenarios.
He encouraged starting with static linters to identify configuration risks and replaying past incidents to prevent recurrence. By integrating resilience checks into pipelines, teams can focus on feature delivery while maintaining reliability. Benjamin’s vision of a resilience culture—where proactive testing is instinctive—resonates with developers seeking to balance velocity and stability.