Posts Tagged ‘CloudNative’
[DevoxxGR2024] The Art of Debugging Inside K8s Environment at Devoxx Greece 2024 by Andrii Soldatenko
At Devoxx Greece 2024, Andrii Soldatenko, a seasoned software engineer and tech evangelist at Dynatrace, delivered an engaging presentation on mastering the art of debugging within Kubernetes (K8s) environments. With a blend of humor, practical insights, and real-world strategies, Andrii illuminated the complexities of troubleshooting cloud-native applications. Drawing from his extensive experience, he provided actionable techniques to enhance debugging efficiency, making the session a valuable resource for developers navigating the intricacies of Kubernetes. His talk emphasized proactive design, robust tooling, and a systematic approach to resolving issues in distributed systems.
The Challenges of Debugging in Kubernetes
Andrii began by acknowledging the inherent difficulties of debugging in modern cloud-native environments. Unlike traditional development, where a local debugger suffices, Kubernetes introduces layers of complexity with containers, pods, and distributed architectures. He humorously outlined his “eight stages of debugging,” from denial (“this can’t happen”) to self-realization (“I wrote this code”), resonating with developers who face similar emotional journeys. These stages underscore the psychological and technical hurdles of troubleshooting in K8s, where issues often stem from accidental complexities like misconfigured resources or network policies.
The dynamic nature of Kubernetes, with its orchestration of pods, nodes, and services, demands a shift in debugging mindset. Andrii emphasized that while writing YAML manifests for K8s is straightforward, ensuring they function as intended is not. He highlighted the absence of comprehensive debugging guides, noting that most literature focuses on deployment rather than troubleshooting. This gap inspired his talk, which aimed to equip developers with practical strategies to diagnose and resolve issues effectively.
Strategies for Effective Debugging
To tackle Kubernetes debugging, Andrii proposed a structured approach, starting with a high-level mind map for assessing pod states. For instance, a pod in a “Pending” state might indicate resource shortages or port conflicts, while a “Crashing” pod could signal health probe failures. He focused on scenarios where pods are running but behaving unexpectedly, a common yet challenging issue. Andrii advocated revisiting init containers, which perform setup tasks like data migrations. By temporarily replacing their commands with a sleep directive, developers can use kubectl exec to inspect the container’s state, checking volumes, permissions, or network access.
For containers lacking debugging tools, Andrii introduced ephemeral containers, a Kubernetes feature since version 1.8 designed for interactive troubleshooting. By launching an ephemeral container with tools like netcat or a debugger, developers can inspect a pod’s state without altering its primary container. He shared a practical example of debugging a Go application by sharing process namespaces, allowing access to the application’s processes. This approach enables setting breakpoints and navigating code, even in minimal, distroless containers.
Leveraging Tools for Enhanced Debugging
Andrii showcased several tools to streamline Kubernetes debugging. He recommended building custom debug containers tailored to specific needs, such as including sqlite, python, or network utilities, and shared his own debug container on GitHub. For network-related issues, he highlighted a pre-existing container with tools like tcpdump, which simplifies packet inspection without requiring manual installations. Andrii also praised Stern, a CLI tool for tailing logs across multiple pods in a replica set, making it easier to trace requests and identify exceptions.
For developers using Visual Studio Code, Andrii demonstrated remote debugging by configuring a launch.json file to connect to a Kubernetes pod. By exposing a debug port and using tools like Telepresence, developers can intercept cluster traffic and test changes locally, bypassing slow CI/CD cycles. He also highlighted K9s, a terminal-based UI for Kubernetes, with a custom plugin for initiating debug sessions via kubectl debug. These tools collectively enhance efficiency, allowing developers to focus on problem-solving rather than manual configuration.
Best Practices for Proactive Debugging
Andrii concluded with actionable best practices to prevent and address debugging challenges. He stressed embedding version information, like Git commit SHAs, into container images to synchronize codebases during remote debugging. Scaling down traffic to a single pod ensures consistent debugging sessions, avoiding request distribution across replicas. He also advocated for a blameless culture, where developers use debuggers to slow down and analyze issues methodically rather than rushing to fix symptoms.
By sharing his GitHub repository and additional resources, Andrii encouraged attendees to experiment with these techniques. His talk was a compelling call to action for developers to embrace robust debugging practices, ensuring resilience and reliability in Kubernetes environments. Through practical demonstrations and a lighthearted approach, he demystified the complexities of cloud-native debugging, empowering developers to tackle issues with confidence.
Links:
[DevoxxGR2024] Devoxx Greece 2024 Sustainability Chronicles: Innovate Through Green Technology With Kepler and KEDA
At Devoxx Greece 2024, Katie Gamanji, a senior field engineer at Apple and a technical oversight committee member for the Cloud Native Computing Foundation (CNCF), delivered a compelling presentation on advancing environmental sustainability within the cloud-native ecosystem. With Kubernetes celebrating its tenth anniversary, Katie emphasized the urgent need for technologists to integrate green practices into their infrastructure strategies. Her talk explored how tools like Kepler and KEDA’s carbon-aware operator enable practitioners to measure and mitigate carbon emissions, while fostering a vibrant, inclusive community to drive these efforts forward. Drawing from her extensive experience and leadership in the CNCF, Katie provided a roadmap for aligning technological innovation with climate responsibility.
The Imperative of Cloud Sustainability
Katie began by underscoring the critical role of sustainability in the tech sector, particularly given the industry’s contribution to global greenhouse gas emissions. She highlighted that the tech sector accounts for 1.4% of global emissions, a figure that could soar to 10% within a decade without intervention. However, by leveraging renewable energy, emissions could be reduced by up to 80%. International agreements like COP21 and the United Nations’ Sustainable Development Goals (SDGs) have spurred national regulations, compelling organizations to assess and report their carbon footprints. Major cloud providers, such as Google Cloud Platform (GCP), have set ambitious net-zero targets, with GCP already operating on renewable energy since 2022. Yet, Katie stressed that sustainability cannot be outsourced solely to cloud providers; organizations must embed these principles internally.
The emergence of “GreenOps,” inspired by FinOps, encapsulates the processes, tools, and cultural shifts needed to achieve digital sustainability. By optimizing infrastructure—through strategies like using spot instances or serverless architectures—organizations can reduce both costs and emissions. Katie introduced a four-phase strategy proposed by the FinOps Foundation’s Environmental Sustainability Working Group: awareness, discovery, roadmap, and execution. This framework encourages organizations to educate stakeholders, benchmark emissions, implement automated tools, and iteratively pursue ambitious sustainability goals.
Measuring Emissions with Kepler
To address emissions within Kubernetes clusters, Katie introduced Kepler, a CNCF sandbox project developed by Red Hat and IBM. Kepler, a Kubernetes Efficient Power Level Exporter, utilizes eBPF to probe system statistics and export power consumption metrics to Prometheus for visualization in tools like Grafana. Deployed as a daemon set, Kepler collects node- and container-level metrics, focusing on power usage and resource utilization. By tracing CPU performance counters and Linux kernel trace points, it calculates energy consumption in joules, converting this to kilowatt-hours and multiplying by region-specific emission factors for gases like coal, petroleum, and natural gas.
Katie demonstrated Kepler’s practical application using a Grafana dashboard, which displayed emissions per gas and allowed granular analysis by container, day, or namespace. This visibility enables organizations to identify high-emission components, such as during traffic spikes, and optimize accordingly. As a sandbox project, Kepler is gaining momentum, and Katie encouraged attendees to explore it, provide feedback, or contribute to its development, reinforcing its potential to establish a baseline for carbon accounting in cloud-native environments.
Scaling Sustainably with KEDA’s Carbon-Aware Operator
Complementing Kepler’s observational capabilities, Katie introduced KEDA (Kubernetes Event-Driven Autoscaler), a graduated CNCF project, and its carbon-aware operator. KEDA, created by Microsoft and Red Hat, scales applications based on external events, offering a rich catalog of triggers. The carbon-aware operator optimizes emissions by scaling applications according to carbon intensity—grams of CO2 equivalent emitted per kilowatt-hour consumed. In scenarios where infrastructure is powered by renewable sources like solar or wind, carbon intensity approaches zero, allowing for maximum application replicas. Conversely, high carbon intensity, such as from coal-based energy, prompts scaling down to minimize emissions.
Katie illustrated this with a custom resource definition (CRD) that configures scaling behavior based on carbon intensity forecasts from providers like WattTime or Electricity Maps. In her demo, a Grafana dashboard showed an application scaling from 15 replicas at a carbon intensity of 530 to a single replica at 580, dynamically responding to grid data. This proactive approach ensures sustainability is embedded in scheduling decisions, aligning resource usage with environmental impact.
Nurturing a Sustainable Community
Beyond technology, Katie emphasized the pivotal role of the Kubernetes community in driving sustainability. Operating on principles of inclusivity, open governance, and transparency, the community fosters innovation through technical advisory groups (TAGs) focused on domains like observability, security, and environmental sustainability. The TAG Environmental Sustainability, established just over a year ago, aims to benchmark emissions across graduated CNCF projects, raising awareness and encouraging greener practices.
To sustain this momentum, Katie highlighted the need for education and upskilling. Resources like the Kubernetes and Cloud Native Associate (KCNA) certification and her own Cloud Native Fundamentals course on Udacity lower entry barriers for newcomers. By diversifying technical and governing boards, the community can continue to evolve, ensuring it scales alongside technological advancements. Katie’s vision is a cloud-native ecosystem where innovation and sustainability coexist, supported by a nurturing, inclusive community.
Conclusion
Katie Gamanji’s presentation at Devoxx Greece 2024 was a clarion call for technologists to prioritize environmental sustainability. By leveraging tools like Kepler and KEDA’s carbon-aware operator, practitioners can measure and mitigate emissions within Kubernetes clusters, aligning infrastructure with climate goals. Equally important is the community’s role in fostering education, inclusivity, and collaboration to sustain these efforts. Katie’s insights, grounded in her leadership at Apple and the CNCF, offer a blueprint for innovating through green technology while building a resilient, forward-thinking ecosystem.
Links:
[DevoxxBE2023] Securing the Supply Chain for Your Java Applications by Thomas Vitale
At Devoxx Belgium 2023, Thomas Vitale, a software engineer and architect at Systematic, delivered an authoritative session on securing the software supply chain for Java applications. As the author of Cloud Native Spring in Action and a passionate advocate for cloud-native technologies, Thomas provided a comprehensive exploration of securing every stage of the software lifecycle, from source code to deployment. Drawing on the SLSA framework and CNCF research, he demonstrated practical techniques for ensuring integrity, authenticity, and resilience using open-source tools like Gradle, Sigstore, and Kyverno. Through a blend of theoretical insights and live demonstrations, Thomas illuminated the critical importance of supply chain security in today’s threat landscape.
Safeguarding Source Code with Git Signatures
Thomas began by defining the software supply chain as the end-to-end process of delivering software, encompassing code, dependencies, tools, practices, and people. He emphasized the risks at each stage, starting with source code. Using Git as an example, Thomas highlighted its audit trail capabilities but cautioned that commit authorship can be manipulated. In a live demo, he showed how he could impersonate a colleague by altering Git’s username and email, underscoring the need for signed commits. By enforcing signed commits with GPG or SSH keys—or preferably a keyless approach via GitHub’s single sign-on—developers can ensure commit authenticity, establishing a verifiable provenance trail critical for supply chain security.
Managing Dependencies with Software Bills of Materials (SBOMs)
Moving to dependencies, Thomas stressed the importance of knowing exactly what libraries are included in a project, especially given vulnerabilities like Log4j. He introduced Software Bills of Materials (SBOMs) as a standardized inventory of software components, akin to a list of ingredients. Using the CycloneDX plugin for Gradle, Thomas demonstrated generating an SBOM during the build process, which provides precise dependency details, including versions, licenses, and hashes for integrity verification. This approach, integrated into Maven or Gradle, ensures accuracy over post-build scanning tools like Snyk, enabling developers to identify vulnerabilities, check license compliance, and verify component integrity before production.
Thomas further showcased Dependency-Track, an OWASP project, to analyze SBOMs and flag vulnerabilities, such as a critical issue in SnakeYAML. He introduced the Vulnerability Exploitability Exchange (VEX) standard, which complements SBOMs by documenting whether vulnerabilities affect an application. In his demo, Thomas marked a SnakeYAML vulnerability as a false positive due to Spring Boot’s safe deserialization, demonstrating how VEX communicates security decisions to stakeholders, reducing unnecessary alerts and ensuring compliance with emerging regulations.
Building Secure Artifacts with Reproducible Builds
The build phase, Thomas explained, is another critical juncture for security. Using Spring Boot as an example, he outlined three packaging methods: JAR files, native executables, and container images. He critiqued Dockerfiles for introducing non-determinism and maintenance overhead, advocating for Cloud Native Buildpacks as a reproducible, secure alternative. In a demo, Thomas built a container image with Buildpacks, highlighting its fixed creation timestamp (January 1, 1980) to ensure identical outputs for unchanged inputs, enhancing security by eliminating variability. This reproducibility, coupled with SBOM generation during the build, ensures artifacts are both secure and traceable.
Signing and Verifying Artifacts with SLSA
To ensure artifact integrity, Thomas introduced the SLSA framework, which provides guidelines for securing software artifacts across the supply chain. He demonstrated signing container images with Sigstore’s Cosign tool, using a keyless approach to avoid managing private keys. This process, integrated into a GitHub Actions pipeline, ensures that artifacts are authentically linked to their creator. Thomas further showcased SLSA’s provenance generation, which documents the artifact’s origin, including the Git commit hash and build steps. By achieving SLSA Level 3, his pipeline provided non-falsifiable provenance, ensuring traceability from source code to deployment.
Securing Deployments with Policy Enforcement
The final stage, deployment, requires validating artifacts to ensure they meet security standards. Thomas demonstrated using Cosign and the SLSA Verifier to validate signatures and provenance, ensuring only trusted artifacts are deployed. On Kubernetes, he introduced Kyverno, a policy engine that enforces signature and provenance checks, automatically rejecting non-compliant deployments. This approach ensures that production environments remain secure, aligning with the principle of validating metadata to prevent unauthorized or tampered artifacts from running.
Conclusion: A Holistic Approach to Supply Chain Security
Thomas’s session at Devoxx Belgium 2023 provided a robust framework for securing Java application supply chains. By addressing source code integrity, dependency management, build reproducibility, artifact signing, and deployment validation, he offered a comprehensive strategy to mitigate risks. His practical demonstrations, grounded in open-source tools and standards like SLSA and VEX, empowered developers to adopt these practices without overwhelming complexity. Thomas’s emphasis on asking “why” at each step encouraged attendees to tailor security measures to their context, ensuring both compliance and resilience in an increasingly regulated landscape.
Links:
[DevoxxPL2022] How We Migrate Customers and Internal Teams to Kubernetes • Piotr Bochyński
At Devoxx Poland 2022, Piotr Bochyński, a seasoned cloud native expert at SAP, shared a compelling narrative on transitioning customers and internal teams from a Cloud Foundry-based platform to Kubernetes. His presentation illuminated the strategic imperatives, technical challenges, and practical solutions that defined SAP’s journey toward a multi-cloud Kubernetes ecosystem. By leveraging open-source projects like Kyma and Gardener, Piotr’s team addressed the limitations of their legacy platform, fostering developer productivity and operational scalability. His insights offer valuable lessons for organizations contemplating a similar migration.
Understanding Platform as a Service
Piotr began by contextualizing Platform as a Service (PaaS), a model that abstracts infrastructure complexities, allowing developers to focus on application development. Unlike Infrastructure as a Service (IaaS), which provides raw virtual machines, PaaS delivers managed runtimes, middleware, and automation, accelerating time-to-market. However, this convenience comes with trade-offs, such as reduced control and potential vendor lock-in, often tied to opinionated frameworks like the 12-factor application methodology. Piotr highlighted SAP’s initial adoption of Cloud Foundry, an open-source PaaS, to avoid vendor dependency while meeting multi-cloud requirements driven by legal and business needs, particularly in sectors like banking. Yet, Cloud Foundry’s constraints, such as single HTTP port exposure and reliance on outdated technologies like BOSH, prompted SAP to explore Kubernetes as a more flexible alternative.
Kubernetes: A Platform for Platforms
Kubernetes, as Piotr elucidated, is not a traditional PaaS but a container orchestration framework that serves as a foundation for building custom platforms. Its declarative API and extensibility distinguish it from predecessors, enabling consistent management of diverse resources like deployments, namespaces, and custom objects. Piotr illustrated this with the thermostat analogy: developers declare a desired state (e.g., 22 degrees), and Kubernetes controllers reconcile the actual state to match it. This pattern, applied uniformly across resources, empowers developers to extend Kubernetes with custom controllers, such as a hypothetical thermostat resource. The Kyma project, an open-source initiative led by SAP, builds on this extensibility, providing opinionated building blocks like Istio-based API gateways, NATS eventing, and serverless functions to bridge the gap between raw Kubernetes and a developer-friendly PaaS.
Overcoming Migration Challenges
The migration to Kubernetes presented multifaceted challenges, from technical complexity to cultural adoption. Piotr emphasized the steep learning curve associated with Kubernetes’ vast resource set, compounded by additional components like Prometheus and Istio. To mitigate this, SAP employed Kyma to abstract complexities, offering simplified resources like API rules that encapsulate Istio configurations for secure service exposure. Another hurdle was ensuring multi-cloud compatibility. SAP’s Gardener project, a managed Kubernetes solution, addressed this by providing a consistent, Kubernetes-compliant layer across providers like AWS, Azure, and Google Cloud. Piotr also discussed operational scalability, managing thousands of clusters for hundreds of teams. By applying the Kubernetes controller pattern, SAP automated cluster provisioning, upgrades, and security patching, reducing manual intervention and ensuring reliability.
Lessons from the Journey
Reflecting on the migration, Piotr candidly shared missteps that shaped SAP’s approach. Early attempts to shield users from Kubernetes’ complexity by mimicking Cloud Foundry’s API failed, as developers craved direct control over Kubernetes resources. Similarly, restricting cluster admin roles to prevent misconfigurations stifled innovation, leading SAP to grant greater flexibility. Some technology choices, like the Service Catalog project, proved inefficient, underscoring the importance of aligning with Kubernetes’ operator pattern. License changes in tools like Grafana also necessitated pivots, highlighting the need for vigilance in open-source dependencies. Piotr’s takeaways resonate broadly: Kubernetes is a long-term investment, requiring a balance of opinionated tooling and developer freedom, with automation as a cornerstone for scalability.