Jonathan Lalou's Blog

Posts Tagged ‘GitOps’

Lecturers

The lecture was co-delivered by Marion Chineaud and Elise Souvannavong, both Full Stack Developers at Takima, a French software engineering and consulting firm. Marion and Elise specialize in building robust, high-volume Java/Spring and Angular applications and implementing modern DevOps practices, including trunk-based development and continuous deployment.

Abstract

In modern software engineering, delaying releases until large feature sets are completed introduces integration risks, complex rebasing conflicts, and production instability. This talk addresses how to exit the binary “all-or-nothing” deployment model by implementing feature toggling and Trunk-Based Development. Grounded in real-world scenarios from Takiship—a logistics microservices ecosystem built with Java Spring, Angular, Kubernetes, and ArgoCD—the session outlines five architectural flag categories: Release Flags, Ops Flags, Experimental Flags (A/B Testing), Shadow Toggles, and Canary Releases.

The speakers detail practical implementation patterns—ranging from Spring properties and @RefreshScope to database-backed administration panels—while confronting the operational overhead of flag pollution. Finally, the presentation connects deployment frequency directly to Google’s DORA metrics, demonstrating how structured flag lifecycles create a robust safety net for modern continuous delivery and AI-assisted development workflows.

The Monolithic Branching Dilemma vs. Trunk-Based Development

Traditional GitFlow strategies often isolate large features on long-lived branches over extended periods. When multiple engineers alter overlapping microservices, merging results in severe rebase friction, missed edge cases, and high-risk releases.

+---------------------------------------------+
|        Traditional GitFlow Risk             |
|                                             |
|  Dev Branch 1: [--- 2 Months Dev ---]       |
|                                     \       |
|  Dev Branch 2: [--- Rebase Friction -\--->  |
|                                       \     |
|  Main Branch:  ========================(FAIL)
+---------------------------------------------+
|        Trunk-Based + Feature Flags          |
|                                             |
|  Small Batch:  --+---+---+---+---> Main     |
|                  |   |   |   |              |
|  Release Flag:  [OFF][OFF][OFF][ON]         |
+---------------------------------------------+

Transitioning to Trunk-Based Development shortens iteration cycles. Code is integrated into the main branch frequently in small batches. To prevent incomplete features from exposing half-finished functionality to end-users, teams decouple physical code deployment from logical feature activation through Release Flags.

Core Benefits of Feature Toggling

• Decoupled Lifecycle: Code can be safely pushed to production while dormant, awaiting business approval or QA validation.
• Instant Rollbacks: When an incident occurs in production, disabling a flag replaces frantic hotfix deployments with an instant configuration change.
• Granular Task Decomposition: Epics spanning multiple microservices can be split into small, trackable tasks that can be developed, merged, and tested in parallel.

Architectural Taxonomy of Feature Flags

Feature flags are not monolithic; they serve distinct technical and business stakeholders across different lifecycles.

Technical Implementations in Java Spring & Kubernetes

Depending on security constraints and autonomy requirements, flag state management can be implemented across three distinct layers.

+---------------------------------------------+
|          Flag Management Taxonomy           |
|                                             |
|  1. Static Application Configuration        |
|     - Spring @ConfigurationProperties       |
|                                             |
|  2. Dynamic GitOps & Hot Reloading          |
|     - Kubernetes ConfigMaps + ArgoCD        |
|     - Spring Cloud @RefreshScope Proxy      |
|                                             |
|  3. DB-Backed Admin Portal                  |
|     - Relational Toggles Table              |
|     - REST Control Endpoints (GET/PUT)      |
+---------------------------------------------+

Option 1: Static Application YAML Configuration

The simplest implementation encapsulates toggles into dedicated configuration properties, separating configuration parameters from domain logic.

@Configuration
@ConfigurationProperties(prefix = "delivery.feature")
public class FeatureFlagsProperties {
    private boolean expressDeliveryEnabled;

    public boolean isExpressDeliveryEnabled() {
        return expressDeliveryEnabled;
    }

    public void setExpressDeliveryEnabled(boolean expressDeliveryEnabled) {
        this.expressDeliveryEnabled = expressDeliveryEnabled;
    }
}

@Service
public class ExpressDeliveryService {
    private final FeatureFlagsProperties properties;

    public ExpressDeliveryService(FeatureFlagsProperties properties) {
        this.properties = properties;
    }

    public void processDelivery(Order order) {
        // Centralized evaluation entry point
        if (properties.isExpressDeliveryEnabled()) {
            executeExpressWorkflow(order);
        } else {
            executeStandardWorkflow(order);
        }
    }
}

• Limitation: Toggling state requires a Git commit, triggering full application rebuilding and pod redeployment.

Option 2: Dynamic GitOps with Spring Cloud @RefreshScope

To achieve zero-downtime hot reloading without restarting JVM instances, configuration properties are stored in a dedicated GitOps repository managed by ArgoCD and mapped into Kubernetes ConfigMaps.

@Component
@RefreshScope
@ConfigurationProperties(prefix = "delivery.ops")
public class OpsFlagsProperties {
    private int maxHistoricalFetchDays = 7;

    public int getMaxHistoricalFetchDays() {
        return maxHistoricalFetchDays;
    }

    public void setMaxHistoricalFetchDays(int maxHistoricalFetchDays) {
        this.maxHistoricalFetchDays = maxHistoricalFetchDays;
    }
}

• Mechanism: Spring Cloud wraps the bean within a dynamic proxy. Invoking POST /actuator/refresh invalidates the target proxy cache, forcing subsequent calls to pull updated parameters directly from the configuration server.
• Critical Restrictions: @RefreshScope cannot be used on scheduled tasks (@Scheduled) or stateful bean dependencies, as forced cache eviction can cause runtime crashes.

Option 3: Database-Backed Administration Portal

When non-technical stakeholders (Product Managers, QA leads) require direct runtime control, flags can be persisted in a database and modified via an administrative dashboard.

CREATE TABLE feature_toggles (
    id VARCHAR(64) PRIMARY KEY,
    toggle_code VARCHAR(64) NOT NULL UNIQUE,
    is_enabled BOOLEAN NOT NULL DEFAULT FALSE,
    display_title VARCHAR(128) NOT NULL
);

To prevent performance degradation from frequent database checks, frontend applications should retrieve the complete active toggle state array alongside user authentication payloads during initial load.

Operational Scenarios and Deployment Patterns

Scenario 1: Managing Traffic Volatility with Ops Flags

During high-volume periods (such as Black Friday or Cyber Week), legacy queries or third-party dependencies can experience severe performance degradation. Ops Flags convert rigid values into dynamic system tuners.

+---------------------------------------------+
|         Ops Flag Runtime Throttling         |
|                                             |
|  Normal Operations  ---> Fetch 30 Days Data |
|                                             |
|  Traffic Surge      ---> Adjust Ops Flag    |
|                          (Fetch 7 Days Data)|
|                                             |
|  System Degradation ---> Trigger Kill Switch|
|                          (Bypass Dependency)|
+---------------------------------------------+

Instead of hardcoding limits, parameters such as query batch sizes, external API timeouts, retry limits, and database pagination bounds are evaluated at runtime.

Scenario 2: Statistical Validation via A/B Testing

When evaluating architectural or UI choices (e.g., standard form vs. multi-step wizard), teams can run both variants concurrently in production.

+---------------------------------------------+
|        Deterministic A/B Routing            |
|                                             |
|  Inbound Request                            |
|        |                                    |
|        v                                    |
|  [ API Gateway ]                            |
|        |-- Cookie Present? -> Route to Variant
|        |-- Cookie Missing? -> Hash User ID  |
|                                (Assign A/B) |
|        v                                    |
|  [ Sticky Session Cookie Set ]              |
+---------------------------------------------+

• Sticky Sessions: Random allocation must be bound deterministically using session cookies or hashed user IDs. Once assigned, a user must consistently see the same variant to prevent confusing user experiences.
• Analytics Integration: Key performance metrics—conversion rates, system performance, error logs, and navigation paths—must be tagged with the active variant ID.

Scenario 3: Zero-Tolerance Domain Changes with Shadow Mode

For critical subsystems where failure presents direct business risk (e.g., billing engine updates), Shadow Mode (Dry-Run) runs both legacy and new implementations concurrently.

+---------------------------------------------+
|           Shadow Mode Execution             |
|                                             |
|                Inbound Transaction          |
|                         |                   |
|           +-------------+-------------+     |
|           |                           |     |
|           v                           v     |
|     [ Legacy Engine ]           [ New Engine ]|
|           |                           |     |
|           v                           v     |
|    (Return Result)             (Log Analytics)|
|           |                           |     |
|           +-------------+-------------+     |
|                         |                   |
|                         v                   |
|             [ Differential Audit Log ]      |
+---------------------------------------------+

1. Incoming requests enter the production gateway.
2. The legacy engine calculates the result and returns it directly to the customer.
3. The secondary engine executes the transaction asynchronously in isolation.
4. Output values, execution traces, and performance characteristics are sent to diagnostic logging systems to surface unexpected discrepancies.

Scenario 4: Risk Mitigation via Canary Releases

During major infrastructure overhauls (such as simultaneous ORM migrations, frontend framework updates, and UI redesigns), changes can be rolled out progressively across user tiers.

+---------------------------------------------+
|           Canary Release Phasing            |
|                                             |
|  Phase 1:  [ Beta Testers ]                 |
|            -> Collect feedback & telemetry  |
|                                             |
|  Phase 2:  [ Standard B2B Customers ]       |
|            -> Validate performance load     |
|                                             |
|  Phase 3:  [ High-Value Key Accounts ]      |
|            -> Complete feature cutover      |
+---------------------------------------------+

Technical Debt Management and Lifecycle Cleanup Strategy

Unmanaged feature flags can lead to operational complexity. Accumulated flag combinations increase test surface areas, complicate local debugging, and increase code clutter.

+---------------------------------------------+
|         Flag Lifecycle Governance           |
|                                             |
|  Release/Experimental Toggles               |
|  [ Created ] -> [ Validated ] -> [ REMOVED ]|
|                                             |
|  Operational Toggles                        |
|  [ Created ] -> [ Maintained Long-Term ]    |
+---------------------------------------------+

Protocol for Technical Debt Mitigation

1. Centralized Evaluation: Restrict flag conditional evaluation (if/else) to a single service layer or facade point. Avoid scattering flags across nested domain methods.
2. Automated Cleanup Tickets: Whenever a new temporary toggle is created, an associated cleanup task must be filed immediately in the sprint backlog.
3. Traceable Annotations: Include explicit inline markers (e.g., // TODO: TOGGLE_CLEANUP_KEY) within target code repositories to streamline string-search audits.
4. Lifecycle Separation: Maintain a clear operational distinction between temporary release toggles (which are decommissioned post-rollout) and permanent operational controls.

Industrial Context: DORA Metrics and AI Integration

Continuous delivery performance relies on key operational metrics evaluated by Google’s DevOps Research and Assessment (DORA) team:

• Deployment Frequency: How often code is successfully deployed to production.
• Lead Time for Changes: The duration required for a committed feature to reach production.
• Change Failure Rate: The percentage of deployments causing production defects.
• Failed Service Recovery Time (MTTR): The time required to restore service stability following an outage.

By decoupling deployment from feature activation, teams can increase deployment frequency while keeping change failure rates low. Toggles also provide an instant recovery mechanism (reducing MTTR) by converting complex rollback procedures into configuration changes.

+---------------------------------------------+
|         AI-Assisted CI/CD Guardrails        |
|                                             |
|  Autonomous Agent Code Generation           |
|                     |                       |
|                     v                       |
|        [ Feature Flag Enclosure ]           |
|                     |                       |
|                     v                       |
|  [ Automated Pipeline & Observability ]     |
|                     |                       |
|             +-------+-------+               |
|             |               |               |
|      (Stable Stream)  (Anomalies)           |
|             |               |               |
|             v               v               |
|      Keep Feature     Disable Flag          |
+---------------------------------------------+

As autonomous AI agents generate larger portions of application code, feature flags serve as a key runtime safety boundary. Enclosing AI-generated code within dynamic feature toggles provides an immediate circuit breaker to isolate anomalies, lower integration costs, and maintain production stability.

Links

• Voxxed Days Luxembourg Video Presentation
• Feature Flags: The Secret Behind Safe Deployments
• Stop Deploying Without Feature Flags — Seriously
• Feature Flags for Stress-Free Continuous Deployment
• 4 Types of Feature Flags, Challenges, and Best Practices
• 8 Types of Deployment Strategies & How Feature Flags Help
• Feature Flags as a Deployment Strategy: Deploy Dark, Release When Ready

Introduction to GitOps and ArgoCD

In the ever-evolving landscape of Kubernetes management, GitOps has emerged as a cornerstone for streamlined application deployment. At KCDUK2024, Farah Adbib and Antonio Alferez, both esteemed professionals from The Workshop, delivered an insightful session titled “An Odyssey with ArgoCD: From Git to Helm.” Their talk elucidated the journey of implementing GitOps using ArgoCD within their organization, navigating through initial challenges and innovative solutions. Farah, a DevOps Solution Architect, and Antonio, a Platform Engineer, shared their expertise on leveraging Git repositories and Helm charts to manage Kubernetes clusters efficiently, offering a narrative rich with practical insights.

The session began with an overview of their ecosystem, managing 55 Kubernetes clusters across public cloud and on-premises infrastructure, supporting 1,600 nodes. These clusters cater to 350 software engineers across various business units, each responsible for deploying their applications. The need for simplicity and security in deployment processes was paramount, given the isolation requirements for some clusters. Farah and Antonio’s narrative underscored the importance of aligning technological solutions with organizational needs, setting the stage for their exploration of ArgoCD’s capabilities.

Initial Approach: Git as the Source of Truth

Initially, Farah and Antonio adopted Git repositories as the primary source for ArgoCD, a logical choice given GitOps’ emphasis on declarative configuration. They opted for a decentralized approach, deploying an ArgoCD instance per cluster to meet stringent security and isolation requirements. Each application’s Helm chart was separated from the application code, stored in distinct Git repositories to avoid replication across data centers. This separation was driven by the differing lifecycles of application code and Helm charts, aiming to streamline management.

However, this approach revealed several pain points. The necessity for Git mirrors introduced additional infrastructure complexity, requiring maintenance and coordination between development and operations teams. The use of Git branches as target revisions led to confusion, as development and operational branches coexisted, complicating version control. Moreover, rendering Helm charts within ArgoCD itself delayed feedback loops, causing failures late in the pipeline. Farah and Antonio’s candid reflection on these challenges highlighted the need for a more robust solution, prompting a strategic pivot.

Evolving to Helm Rendered Manifests

Recognizing the limitations of their initial setup, Farah and Antonio transitioned to a Helm-rendered manifest pattern. This innovative approach involved pre-rendering Helm manifests in the CI/CD pipeline and storing them in an artifact registry, such as Nexus, rather than Git mirrors. By pointing ArgoCD to these pre-rendered manifests, they eliminated the need for ArgoCD to handle rendering, significantly reducing complexity and accelerating feedback loops. This shift also unified application code and Helm charts into a single repository, simplifying versioning and pipeline management.

A key enhancement was the introduction of an external Helm values repository per cluster, allowing operational changes like scaling without triggering a full pipeline rebuild. This decoupling of configuration from code enhanced flexibility and developer experience. Antonio emphasized the adoption of semantic versioning (SemVer) for both Docker images and Helm charts, incorporating commit hashes for traceability. This meticulous versioning strategy ensured clarity on deployed versions across 55 clusters, leveraging Git notes for additional auditing information.

Lessons Learned and Developer Experience

The transition to Helm-rendered manifests yielded significant improvements. By moving rendering logic to the CI/CD pipeline, Farah and Antonio achieved a “shift-left” approach, enabling earlier failure detection and faster iterations. The elimination of Git mirrors reduced infrastructure overhead, while unified repositories streamlined development workflows. The external Helm values repository facilitated rapid operational adjustments, enhancing agility.

Farah and Antonio underscored the importance of tailoring solutions to the organizational ecosystem. Their journey highlighted the pitfalls of adhering to industry defaults without considering specific requirements. They emphasized that developer experience is paramount, advocating for solutions that empower engineers while minimizing management overhead. Their narrative serves as a testament to the value of iterative improvement, encouraging practitioners to reassess and redesign when initial solutions fall short.

Links:

• The Workshop company website