Posts Tagged ‘MultiCloud’
[GopherConUK2025] Conceptual XPDB Custom Resource definition
apiVersion: policy.form3.tech/v1alpha1
kind: CrossClusterPodDisruptionBudget
metadata:
name: cockroachdb-global-pdb
spec:
maxUnavailable: 1
clusters:
– aws-region-1
– gcp-region-1
– azure-region-1
selector:
matchLabels:
app: cockroachdb
XPDB enforces global disruption caps across cluster boundaries. When node drains or pod evasions occur in one cloud provider, XPDB evaluates the global state across all environments. By guaranteeing that only a single CockroachDB pod is disrupted across the entire global infrastructure at any given moment, XPDB ensures data consensus remains fully protected during routine infrastructure maintenance.
## Operator-Driven Infrastructure and Rolling Node-Pool Management
Managing multi-tenant infrastructure across multiple jurisdictions—each containing distinct development, staging, and production environments—results in a massive expansion of node pools. Updating Kubernetes worker nodes across this matrix using traditional Infrastructure-as-Code tools like Terraform creates extreme operational friction, requiring dozens of sequential pull requests and complex deployment pipelines.
To address this management overhead, Form3 built a custom Kubernetes operator called the Cluster Lifecycle Operator. The operator runs natively inside each managed cluster and abstracts raw node-pool management behind Custom Resource Definitions (CRDs).
// Conceptual snippet of CRD controller reconciliation loop
package main
import (
“context”
“fmt”
)
type ClusterSpec struct {
Version string json:"version"
}
func ReconcileNodePool(ctx context.Context, spec ClusterSpec) error {
fmt.Printf(“Reconciling node pools to version: %s\n”, spec.Version)
// Operator logic handles sequential node drains
return nil
}
Instead of modifying declarative infrastructure files for every individual node pool across every provider, engineers update a single CRD spec controlling the target cluster version. The Cluster Lifecycle Operator handles the rolling replacement of worker nodes asynchronously, adhering to defined disruption parameters and health checks. Upgrading the global fleet requires only three sequential pull requests—promoted systematically through development, staging, and production.
## Continuous Disaster Recovery via Automated Production Chaos Injection
Conventional disaster recovery (DR) practices often rely on periodic manual failover tests driven by static documentation. In rapidly changing microservice environments, compliance-focused DR exercises fail to validate real-world resilience, as system changes can render manual playbooks obsolete immediately after testing.
Form3 enforces continuous disaster recovery validation directly within staging environments through automated fault injection. A dedicated test harness deploys synthetic client applications outside the primary infrastructure perimeter. These synthetic actors continuously execute end-to-end payment workflows against simulated payment scheme interfaces at fixed intervals.
// Custom chaos injection test runner in Go
package main
import (
“context”
“log”
“time”
)
func InjectProviderOutage(ctx context.Context, targetCloud string) error {
log.Printf(“Simulating complete network partition for provider: %s”, targetCloud)
// Inject network block rules via Chaos Mesh API
time.Sleep(30 * time.Second)
return nil
}
“`
During automated test windows, the platform uses Chaos Mesh alongside custom Go-based orchestrators to introduce disruptive scenarios:
- Severing inter-cloud network connectivity.
- Terminating entire database nodes or message brokers.
- Programmatically isolating an entire cloud provider for 24 hours.
If synthetic payment processing encounters errors or breaches latency thresholds, automated alerts notify on-call platform teams. Automated daily reports detail platform behavior during fault injection cycles, verifying that the loss of an entire cloud vendor produces zero client impact.
Links:
[DevoxxPL2022] How We Migrate Customers and Internal Teams to Kubernetes • Piotr Bochyński
At Devoxx Poland 2022, Piotr Bochyński, a seasoned cloud native expert at SAP, shared a compelling narrative on transitioning customers and internal teams from a Cloud Foundry-based platform to Kubernetes. His presentation illuminated the strategic imperatives, technical challenges, and practical solutions that defined SAP’s journey toward a multi-cloud Kubernetes ecosystem. By leveraging open-source projects like Kyma and Gardener, Piotr’s team addressed the limitations of their legacy platform, fostering developer productivity and operational scalability. His insights offer valuable lessons for organizations contemplating a similar migration.
Understanding Platform as a Service
Piotr began by contextualizing Platform as a Service (PaaS), a model that abstracts infrastructure complexities, allowing developers to focus on application development. Unlike Infrastructure as a Service (IaaS), which provides raw virtual machines, PaaS delivers managed runtimes, middleware, and automation, accelerating time-to-market. However, this convenience comes with trade-offs, such as reduced control and potential vendor lock-in, often tied to opinionated frameworks like the 12-factor application methodology. Piotr highlighted SAP’s initial adoption of Cloud Foundry, an open-source PaaS, to avoid vendor dependency while meeting multi-cloud requirements driven by legal and business needs, particularly in sectors like banking. Yet, Cloud Foundry’s constraints, such as single HTTP port exposure and reliance on outdated technologies like BOSH, prompted SAP to explore Kubernetes as a more flexible alternative.
Kubernetes: A Platform for Platforms
Kubernetes, as Piotr elucidated, is not a traditional PaaS but a container orchestration framework that serves as a foundation for building custom platforms. Its declarative API and extensibility distinguish it from predecessors, enabling consistent management of diverse resources like deployments, namespaces, and custom objects. Piotr illustrated this with the thermostat analogy: developers declare a desired state (e.g., 22 degrees), and Kubernetes controllers reconcile the actual state to match it. This pattern, applied uniformly across resources, empowers developers to extend Kubernetes with custom controllers, such as a hypothetical thermostat resource. The Kyma project, an open-source initiative led by SAP, builds on this extensibility, providing opinionated building blocks like Istio-based API gateways, NATS eventing, and serverless functions to bridge the gap between raw Kubernetes and a developer-friendly PaaS.
Overcoming Migration Challenges
The migration to Kubernetes presented multifaceted challenges, from technical complexity to cultural adoption. Piotr emphasized the steep learning curve associated with Kubernetes’ vast resource set, compounded by additional components like Prometheus and Istio. To mitigate this, SAP employed Kyma to abstract complexities, offering simplified resources like API rules that encapsulate Istio configurations for secure service exposure. Another hurdle was ensuring multi-cloud compatibility. SAP’s Gardener project, a managed Kubernetes solution, addressed this by providing a consistent, Kubernetes-compliant layer across providers like AWS, Azure, and Google Cloud. Piotr also discussed operational scalability, managing thousands of clusters for hundreds of teams. By applying the Kubernetes controller pattern, SAP automated cluster provisioning, upgrades, and security patching, reducing manual intervention and ensuring reliability.
Lessons from the Journey
Reflecting on the migration, Piotr candidly shared missteps that shaped SAP’s approach. Early attempts to shield users from Kubernetes’ complexity by mimicking Cloud Foundry’s API failed, as developers craved direct control over Kubernetes resources. Similarly, restricting cluster admin roles to prevent misconfigurations stifled innovation, leading SAP to grant greater flexibility. Some technology choices, like the Service Catalog project, proved inefficient, underscoring the importance of aligning with Kubernetes’ operator pattern. License changes in tools like Grafana also necessitated pivots, highlighting the need for vigilance in open-source dependencies. Piotr’s takeaways resonate broadly: Kubernetes is a long-term investment, requiring a balance of opinionated tooling and developer freedom, with automation as a cornerstone for scalability.