Jonathan Lalou's Blog

Lecturer

Kevin Holditch serves as the Vice President of Platform at Form3, where he oversees the architectural evolution and engineering strategies of cloud-native financial infrastructure. Having joined Form3 in 2016 as one of its founding engineers, Kevin has guided the company’s scaling journey from a small initial team to a major technology provider powering critical payment systems across the United Kingdom and Europe. His career prior to Form3 spans technical leadership roles across high-scale engineering organizations, including EasyJet and ASOS.

Abstract

Modern financial technology demands uninterrupted availability, tight latency constraints, and operational independence from single cloud provider failures. This paper explores how Form3 transitioned its payment gateway architecture from a single-cloud AWS Java application to an active-active-active, cloud-agnostic platform engineered in Go across AWS, Google Cloud Platform (GCP), and Microsoft Azure. The analysis focuses on three specific technical challenges inherent in multi-cloud topology: cross-cluster Pod Disruption Budgets, automated rolling node-pool upgrades, and continuous disaster recovery through automated production-grade chaos engineering. By treating entire cloud providers as independent availability zones, the system achieves deterministic resilience while fulfilling regulatory mandates on zero vendor lock-in.

Architecture Evolution: From Single-Cloud Java to Cloud-Agnostic Go

Form3 operates as a unified RESTful API layer that abstracts the intricate protocol differences of global financial payment schemes. Payment networks such as BACS and Faster Payments (FPS) in the United Kingdom, alongside European payment rails, differ fundamentally across network transport channels, payload specifications, and state transitions. The platform insulates financial institutions from these underlying variations through a standardized interface.

+-----------------------------------------------+
|             Client Ingress (API)              |
+-----------------------------------------------+
                        |
      +-----------------+-----------------+
      |                 |                 |
      v                 v                 v
+-----------+     +-----------+     +-----------+
|    AWS    |     |    GCP    |     |   Azure   |
| Load Bal. |     | Load Bal. |     | Load Bal. |
+-----------+     +-----------+     +-----------+
      |                 |                 |
      v                 v                 v
+-----------+     +-----------+     +-----------+
|    K8s    |     |    K8s    |     |    K8s    |
| Cluster A |     | Cluster B |     | Cluster C |
+-----------+     +-----------+     +-----------+
      |                 |                 |
      +--------+--------+--------+--------+
               |                 |
               v                 v
     +-------------------+ +-------------------+
     |   CockroachDB     | |   NATS JetStream  |
     | (Distributed SQL) | | (Message Fabric)   |
     +-------------------+ +-------------------+

In its initial implementation, the platform utilized an AWS-native architecture built with Java microservices running on Amazon Elastic Container Service (ECS). Asynchronous service-to-service communication was managed via Amazon SQS, with relational data stored in Amazon RDS PostgreSQL. This architectural posture prioritized rapid market entry by delegating operational heavy lifting to managed AWS services.

// Sample multi-region ping handler in Go
package main

import (
    "encoding/json"
    "net/http"
)

type HealthResponse struct {
    Status string `json:"status"`
    Region string `json:"region"`
}

func HealthCheck(w http.ResponseWriter, r *http.Request) {
    resp := HealthResponse{
        Status: "HEALTHY",
        Region: "multi-cloud-node",
    }
    w.Header().Set("Content-Type", "application/json")
    _ = json.NewEncoder(w).Encode(resp)
}

The onboarding of Tier-1 retail banks introduced stricter regulatory and throughput constraints. Regulatory guidance from the UK Prudential Regulation Authority (PRA) requires financial institutions to maintain explicit exit strategies from cloud vendors to mitigate systemic concentration risk. To fulfill high-throughput targets exceeding 1,000 transactions per second alongside stringent Service Level Agreements (SLAs), Form3 abandoned single-cloud fallback strategies in favor of an active-active-active topology across three distinct cloud providers.

The system uses private network cross-links between AWS, GCP, and Azure, enabling pods in separate managed Kubernetes clusters to communicate via direct IP routing. In this topology, entire cloud providers function as virtual availability zones. Transitioning the microservice codebase from Java (Spring Boot) to Go delivered immediate operational benefits: startup times dropped from tens of seconds to near-instantaneous container initialization, eliminating test execution bottlenecks and reducing runtime memory footprints. Additionally, Go’s explicit error handling and minimal runtime abstraction eliminated hidden reflection mechanisms, improving codebase maintainability.

Distributed Consensus and State Preservation Across Provider Boundaries

Sustaining ACID transactional guarantees across disparate cloud vendors requires database and messaging infrastructure capable of consensus-based replication over public cloud interconnections. Form3 replaced proprietary cloud messaging and database layers with cloud-agnostic, distributed Go-native software components: CockroachDB for relational state storage and NATS JetStream for high-throughput event streaming.

CockroachDB implements the PostgreSQL wire protocol, allowing existing SQL schemas and drivers to transition seamlessly. Underneath, it relies on Raft consensus protocols to maintain consistency across distributed nodes. To prevent double-spending and ensure financial data integrity, the cluster operates in a right-consistent mode spanning all three cloud providers. A write operation is committed only after achieving write quorum across a minimum of two separate cloud vendors.

// Example CockroachDB transaction retry logic in Go
package main

import (
    "context"
    "database/sql"
    "fmt"

    _ "github.com/lib/pq"
)

func ExecutePayment(ctx context.Context, db *sql.DB, id string, amt int64) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    query := `UPDATE accounts SET balance = balance - $1 WHERE id = $2`
    if _, err := tx.ExecContext(ctx, query, amt, id); err != nil {
        return fmt.Errorf("update failed: %w", err)
    }

    return tx.Commit()
}

NATS JetStream acts as the persistence-backed messaging fabric. Originally authored in Go by the team behind high-frequency trading platforms at Tibco, NATS provides low-latency, lightweight message distribution. JetStream extends base NATS with Raft-driven state replication, allowing message streams to survive the total failure of any single cloud provider without loss of message ordering or state corruption.

Cross-Cluster Pod Disruption Budgets (XPDB)

Standard Kubernetes Pod Disruption Budgets (PDB) operate strictly within the boundaries of a single cluster control plane. In an architecture comprising three uncoordinated Kubernetes clusters sharing a single logical CockroachDB deployment, native PDB mechanisms introduce systemic availability risks.

If a nine-node CockroachDB cluster is distributed evenly with three nodes per cloud provider, maintaining database quorum requires a minimum of five operational nodes at all times. If each independent cluster configures a standard PDB allowing one pod to be unavailable (maxUnavailable: 1), concurrent node maintenance or rolling updates across clusters could simultaneously terminate one pod per cloud provider. This aggregate loss of three pods reduces active cluster capacity to six nodes—dangerously close to losing quorum.

To solve this cross-cluster coordination failure, Form3 developed and open-sourced Cross-Cluster Pod Disruption Budget (XPDB). XPDB implements a centralized or peer-coordinated budget counter across N distinct Kubernetes clusters.

“`