Posts Tagged ‘CICD’
[DevoxxFR2025] Dagger Modules: A Swiss Army Knife for Modern CI/CD Pipelines
Continuous Integration and Continuous Delivery (CI/CD) pipelines are the backbone of modern software development, automating the process of building, testing, and deploying applications. However, as these pipelines grow in complexity, they often become difficult to maintain, debug, and port across different execution platforms, frequently relying on verbose and platform-specific YAML configurations. Jean-Christophe Sirot, in his presentation, introduced Dagger as a revolutionary approach to CI/CD, allowing pipelines to be written as code, executable locally, testable, and portable. He explored Dagger Functions and Dagger Modules as key concepts for creating and sharing reusable, language-agnostic components for CI/CD workflows, positioning Dagger as a versatile “Swiss Army knife” for modernizing these critical pipelines.
The Pain Points of Traditional CI/CD
Jean-Christophe began by outlining the common frustrations associated with traditional CI/CD pipelines. Relying heavily on YAML or other declarative formats for defining pipelines can lead to complex, repetitive, and hard-to-read configurations, especially for intricate workflows. Debugging failures within these pipelines is often challenging, requiring pushing changes to a remote CI server and waiting for the pipeline to run. Furthermore, pipelines written for one CI platform (like GitHub Actions or GitLab CI) are often not easily transferable to another, creating vendor lock-in and hindering flexibility. This dependency on specific platforms and the difficulty in managing complex workflows manually are significant pain points for development and DevOps teams.
Dagger: CI/CD as Code
Dagger offers a fundamentally different approach by treating CI/CD pipelines as code. It allows developers to write their pipeline logic using familiar programming languages (like Go, Python, Java, or TypeScript) instead of platform-specific configuration languages. This brings the benefits of software development practices – such as code reusability, modularity, testing, and versioning – to CI/CD. Jean-Christophe explained that Dagger executes these pipelines using containers, ensuring consistency and portability across different environments. The Dagger engine runs the pipeline logic, orchestrates the necessary container operations, and manages dependencies. This allows developers to run and debug their CI/CD pipelines locally using the same code that will execute on the remote CI platform, significantly accelerating the debugging cycle.
Dagger Functions and Modules
Key to Dagger’s power are Dagger Functions and Dagger Modules. Jean-Christophe described Dagger Functions as the basic building blocks of a pipeline – functions written in a programming language that perform specific CI/CD tasks (e.g., building a Docker image, running tests, deploying an application). These functions interact with the Dagger engine to perform container operations. Dagger Modules are collections of related Dagger Functions that can be packaged and shared. Modules allow teams to create reusable components for common CI/CD patterns or specific technologies, effectively creating a library of CI/CD capabilities. For example, a team could create a “Java Build Module” containing functions for compiling Java code, running Maven or Gradle tasks, and building JAR or WAR files. These modules can be easily imported and used in different projects, promoting standardization and reducing duplication across an organization’s CI/CD workflows. Jean-Christophe demonstrated how to create and use Dagger Modules, illustrating their potential for building composable and maintainable pipelines. He highlighted that Dagger’s language independence means that modules can be written in one language (e.g., Python) and used in a pipeline defined in another (e.g., Java), fostering collaboration between teams with different language preferences.
The Benefits: Composable, Maintainable, Portable
By adopting Dagger, teams can create CI/CD pipelines that are:
– Composable: Pipelines can be built by combining smaller, reusable Dagger Modules and Functions.
– Maintainable: Pipelines written as code are easier to read, understand, and refactor using standard development tools and practices.
– Portable: Pipelines can run on any platform that supports Dagger and containers, eliminating vendor lock-in.
– Testable: Individual Dagger Functions and modules can be unit tested, and the entire pipeline can be run and debugged locally.
Jean-Christophe’s presentation positioned Dagger as a versatile tool that modernizes CI/CD by bringing the best practices of software development to pipeline automation. The ability to write pipelines in code, leverage reusable modules, and execute locally makes Dagger a powerful “Swiss Army knife” for developers and DevOps engineers seeking more efficient, reliable, and maintainable CI/CD workflows.
Links:
- Jean-Christophe Sirot: https://www.linkedin.com/in/jcsirot/
- Decathlon: https://www.decathlon.com/
- Dagger: https://dagger.io/
- Devoxx France LinkedIn: https://www.linkedin.com/company/devoxx-france/
- Devoxx France Bluesky: https://bsky.app/profile/devoxx.fr
- Devoxx France Website: https://www.devoxx.fr/
[DefCon32] OH MY DC: Abusing OIDC All the Way to Your Cloud
As organizations migrate from static credentials to dynamic authentication protocols, overlooked intricacies in implementations create fertile ground for exploitation. Aviad Hahami, a security researcher at Palo Alto Networks, demystifies OpenID Connect (OIDC) in the context of continuous integration and deployment (CI/CD) workflows. His examination reveals vulnerabilities stemming from under-configurations and misconfigurations, enabling unauthorized access to cloud environments. By alternating perspectives among users, identity providers, and CI vendors, Aviad illustrates attack vectors that compromise sensitive resources.
Aviad begins with foundational concepts, clarifying OIDC’s role in secure, short-lived token exchanges. In CI/CD scenarios, tools like GitHub Actions request tokens from identity providers (IdPs) such as GitHub’s OIDC provider. These tokens, containing claims like repository names and commit SHAs, are validated by workload identity federations (WIFs) in clouds like AWS or Azure. Proper configuration ensures tokens originate from trusted sources, but lapses invite abuse.
Common pitfalls include wildcard allowances in policies, permitting access from unintended repositories. Aviad demonstrates how fork pull requests (PRs) exploit these, granting cloud roles without maintainer approval. Such “no configs” scenarios, where minimal effort yields high rewards, underscore the need for precise claim validations.
Advanced Configurations and Misconfigurations
Delving deeper, Aviad explores “advanced configs” that inadvertently become misconfigurations. Features like GitHub’s ID token requests for forks introduce risks if not explicitly enabled. He recounts discovering a vulnerability in CircleCI, where reusable configurations allowed token issuance to forks, bypassing protections.
Shifting to the IdP viewpoint, Aviad discloses a real-world flaw in a popular CI vendor, permitting token claims from any repository within an organization. This enabled cross-project escalations, compromising clouds via simple PRs. Reported responsibly, the issue prompted fixes, emphasizing the cascading effects of IdP errors.
He references Tinder’s research on similar WIF misconfigurations, reinforcing that even sophisticated setups falter without rigorous claim scrutiny.
Exploitation Through CI Vendors
Aviad pivots to CI vendor responsibilities, highlighting how their token issuance logic influences downstream security. In CircleCI’s case, a bug allowed organization-wide token claims, exposing multiple projects. By requesting tokens in fork contexts, attackers could satisfy broad WIF conditions, accessing clouds undetected.
Remediation involved opt-in mechanisms for fork tokens, mirroring GitHub’s approach. Aviad stresses learning claim origins per IdP, avoiding wildcards, and hardening pipelines to prevent trivial breaches.
His tool for auditing Azure CLI configurations exemplifies proactive defense, aiding in identifying exposed resources.
Broader Implications for Secure Authentication
Aviad’s insights extend beyond CI/CD, advocating holistic OIDC understanding to thwart supply chain attacks. By dissecting entity interactions—users, IdPs, and clouds—he equips practitioners to craft resilient policies.
Encouraging bounty hunters to probe these vectors, he underscores OIDC’s maturity yet persistent gaps. Ultimately, robust configurations transform OIDC from vulnerability to asset, safeguarding digital infrastructures.
Links:
[DevoxxFR2014] Tips and Tricks for Releasing with Maven, Hudson, Artifactory, and Git: Streamlining Software Delivery
Lecturer
Michael Hüttermann, a freelance DevOps consultant from Germany, specializes in optimizing software delivery pipelines. With a background in Java development and continuous integration, he has authored books on Agile ALM and contributes to open-source projects. His expertise lies in integrating tools like Maven, Jenkins (formerly Hudson), Artifactory, andទ
System: ## Git, and Maven to create efficient release processes. His talk at Devoxx France 2014 shares practical strategies for streamlining software releases, drawing on his extensive experience in DevOps consulting.
Abstract
Releasing software with Maven can be a cumbersome process, often fraught with manual steps and configuration challenges, despite Maven’s strengths as a build tool. In this lecture from Devoxx France 2014, Michael Hüttermann presents a comprehensive guide to optimizing the release process by integrating Maven with Hudson (now Jenkins), Artifactory, and Git. He explores the limitations of Maven’s release plugin and offers lightweight alternatives that enhance automation, traceability, and efficiency. Through detailed examples and best practices, Hüttermann demonstrates how to create a robust CI/CD pipeline that leverages version control, binary management, and continuous integration to deliver software reliably. The talk emphasizes practical configurations, common pitfalls, and strategies for achieving seamless releases in modern development workflows.
The Challenges of Maven Releases
Maven is a powerful build tool that simplifies dependency management and build automation, but its release plugin can be rigid and complex. Hüttermann explains that the plugin often requires manual version updates, tagging, and deployment steps, which can disrupt workflows and introduce errors. For example, the mvn release:prepare and mvn release:perform commands automate versioning and tagging, but they lack flexibility for custom workflows and can fail if network issues or repository misconfigurations occur.
Hüttermann advocates for a more integrated approach, combining Maven with Hudson, Artifactory, and Git to create a streamlined pipeline. This integration addresses key challenges: ensuring reproducible builds, managing binary artifacts, and maintaining version control integrity.
Building a CI/CD Pipeline with Hudson
Hudson, now known as Jenkins, serves as the orchestration hub for the release process. Hüttermann describes a multi-stage pipeline that automates building, testing, and deploying Maven projects. A typical Jenkins pipeline might look like this:
pipeline {
agent any
stages {
stage('Checkout') {
steps {
git url: 'https://github.com/example/repo.git', branch: 'main'
}
}
stage('Build') {
steps {
sh 'mvn clean package'
}
}
stage('Deploy') {
steps {
sh 'mvn deploy -DskipTests'
}
}
}
}
The pipeline connects to a Git repository, builds the project with Maven, and deploys artifacts to Artifactory. Hüttermann emphasizes the importance of parameterized builds, allowing developers to specify release versions or snapshot flags dynamically.
Leveraging Artifactory for Binary Management
Artifactory, a binary repository manager, plays a critical role in storing and distributing Maven artifacts. Hüttermann highlights its ability to manage snapshots and releases, ensuring traceability and reproducibility. Artifacts are deployed to Artifactory using Maven’s deploy goal:
mvn deploy -DaltDeploymentRepository=artifactory::default::http://artifactory.example.com/releases
This command uploads artifacts to a specified repository, with Artifactory providing metadata for dependency resolution. Hüttermann notes that Artifactory’s cloud-based hosting simplifies access for distributed teams, and its integration with Jenkins via plugins enables automated deployment.
Git Integration for Version Control
Git serves as the version control system, managing source code and enabling release tagging. Hüttermann recommends using Git commit hashes to track builds, ensuring traceability. A typical release process involves creating a tag:
git tag -a v1.0.0 -m "Release 1.0.0"
git push origin v1.0.0
Jenkins’ Git plugin automates checkout and tagging, reducing manual effort. Hüttermann advises using a release branch for stable versions, with snapshots developed on main to maintain a clear workflow.
Streamlining the Release Process
To overcome the limitations of Maven’s release plugin, Hüttermann suggests custom scripts and Jenkins pipelines to automate versioning and deployment. For example, a script to increment version numbers in the pom.xml file can be integrated into the pipeline:
mvn versions:set -DnewVersion=1.0.1
This approach allows fine-grained control over versioning, avoiding the plugin’s rigid conventions. Hüttermann also recommends using Artifactory’s snapshot repositories for development builds, with stable releases moved to release repositories after validation.
Common Pitfalls and Best Practices
Network connectivity issues can disrupt deployments, as Hüttermann experienced during a demo when a Jenkins job failed due to a network outage. He advises configuring retry mechanisms in Jenkins and using Artifactory’s caching to mitigate such issues. Another pitfall is version conflicts in multi-module projects; Hüttermann suggests enforcing consistent versioning across modules with Maven’s versions plugin.
Best practices include maintaining a clean workspace, using Git commit hashes for traceability, and integrating unit tests into the pipeline to ensure quality. Hüttermann also emphasizes the importance of separating source code (stored in Git) from binaries (stored in Artifactory) to maintain a clear distinction between development and deployment artifacts.
Practical Demonstration and Insights
During the lecture, Hüttermann demonstrates a Jenkins pipeline that checks out code from Git, builds a Maven project, and deploys artifacts to Artifactory. The pipeline includes parameters for release candidates and stable versions, showcasing flexibility. He highlights the use of Artifactory’s generic integration, which supports any file type, making it versatile for non-Maven artifacts.
The demo illustrates a three-stage process: building a binary, copying it to a workspace, and deploying it to Artifactory. Despite a network-related failure, Hüttermann uses the opportunity to discuss resilience, recommending offline capabilities and robust error handling.
Broader Implications for DevOps
The integration of Maven, Hudson, Artifactory, and Git aligns with DevOps principles of automation and collaboration. By automating releases, teams reduce manual errors and accelerate delivery, critical for agile development. Hüttermann’s approach supports both small startups and large enterprises, offering scalability through cloud-based Artifactory and Jenkins.
For developers, the talk provides actionable strategies to simplify releases, while organizations benefit from standardized pipelines that ensure compliance and traceability. The emphasis on lightweight processes challenges traditional heavy release cycles, promoting continuous delivery.
Conclusion: A Blueprint for Efficient Releases
Michael Hüttermann’s lecture offers a practical roadmap for streamlining software releases using Maven, Hudson, Artifactory, and Git. By addressing the shortcomings of Maven’s release plugin and leveraging integrated tools, developers can achieve automated, reliable, and efficient release processes. The talk underscores the importance of CI/CD pipelines in modern software engineering, providing a foundation for DevOps success.
Links
[DevoxxFR2014] Runtime stage
FROM nginx:alpine
COPY –from=builder /app/dist /usr/share/nginx/html
EXPOSE 80
This pattern reduces final image size from hundreds of megabytes to tens of megabytes. **Layer caching** optimization requires careful instruction ordering:
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
Copying dependency manifests first maximizes cache reuse during development.
## Networking Models and Service Discovery
Docker’s default bridge network isolates containers on a single host. Production environments demand multi-host communication. **Overlay networks** create virtual networks across swarm nodes:
docker network create –driver overlay –attachable prod-net
docker service create –network prod-net –name api myapp:latest
Docker’s built-in DNS enables service discovery by name. For external traffic, **ingress routing meshes** like Traefik or NGINX provide load balancing, TLS termination, and canary deployments.
## Persistent Storage for Stateful Applications
Stateless microservices dominate container use cases, but databases and queues require durable storage. **Docker volumes** offer the most flexible solution:
docker volume create postgres-data
docker run -d \
–name postgres \
-v postgres-data:/var/lib/postgresql/data \
-e POSTGRES_PASSWORD=secret \
postgres:13
For distributed environments, **CSI (Container Storage Interface)** plugins integrate with Ceph, GlusterFS, or cloud-native storage like AWS EBS.
## Orchestration and Automated Operations
Docker Swarm provides native clustering with zero external dependencies:
docker swarm init
docker stack deploy -c docker-compose.yml myapp
“`
For advanced workloads, Kubernetes offers:
– Deployments for rolling updates and self-healing.
– Horizontal Pod Autoscaling based on CPU/memory or custom metrics.
– ConfigMaps and Secrets for configuration management.
Migration paths typically begin with stateless services in Swarm, then progress to Kubernetes for stateful and machine-learning workloads.
Security Hardening and Compliance
Production containers must follow security best practices:
– Run as non-root users: USER appuser in Dockerfile.
– Scan images with Trivy or Clair in CI/CD pipelines.
– Apply seccomp and AppArmor profiles to restrict system calls.
– Use RBAC and Network Policies in Kubernetes to enforce least privilege.
Production Case Studies and Operational Wisdom
Spotify manages thousands of microservices using Helm charts and custom operators. Airbnb leverages Kubernetes for dynamic scaling during peak booking periods. The New York Times uses Docker for CI/CD acceleration, reducing deployment time from hours to minutes.
Common lessons include:
– Monitor with Prometheus and Grafana.
– Centralize logs with ELK or Loki.
– Implement distributed tracing with Jaeger or Zipkin.
– Use chaos engineering to validate resilience.
Strategic Impact on DevOps Culture
Docker fundamentally accelerates the CI/CD pipeline and enables immutable infrastructure. Success requires cultural alignment: developers embrace infrastructure-as-code, operations teams adopt GitOps workflows, and security integrates into every stage. Orchestration platforms bridge the gap between development velocity and operational stability.