[KCDUK2024] The Operator Antipattern | Gerald Schmidt
At KCDUK2024, Gerald Schmidt, data platform engineering lead at DS Smith, delivered a candid reflection on the Kubernetes operator pattern, questioning its efficacy in modern platform engineering. Drawing from his experience at Babylon Health, Go City, and Thoughtworks, Gerald shared a journey from enthusiasm to disillusionment, highlighting how operators, despite their promise, often introduce complexity and fragility that outweigh their benefits.
The Allure and Pitfalls of Operators
Gerald opened with his initial excitement for operators, which promised to extend Kubernetes’ API for managing complex stateful applications. He referenced Brandon Phillips’ vision of operators embedding domain expertise to automate tasks, such as managing databases or message queues. However, Gerald’s experience revealed that operators rarely delivered on this promise. Instead, they introduced tight coupling between custom resource definitions (CRDs) and controllers, creating operational challenges.
A pivotal anecdote from his time at a health startup illustrated this. Gerald’s team built operators for data flows, only to face disaster when CRDs were inadvertently deleted, requiring a weekend-long recovery. This incident led to their team losing deployment privileges, underscoring the risks of operators in environments with shrinking teams and heightened scrutiny. Gerald argued that operators, while powerful, often fail to match the robustness of managed services like AWS RDS, which offer superior durability and availability.
Persistent Volumes Over Stateful Applications
Gerald challenged the notion that Kubernetes struggles with stateful applications, suggesting the real issue lies with persistent volumes. Solutions like Thanos and WarpStream, which leverage object storage, address this effectively without relying on operators. He highlighted the maturity of the object storage ecosystem, with providers like AWS S3, Google Cloud Storage, and Wasabi offering cost-effective, reliable alternatives. The Container Object Storage Interface (COSI), though still emerging, promises further simplification, reducing the need for complex operator-based solutions.
He contrasted this with the operator pattern’s limitations, particularly around CRDs. Unlike controllers, which integrate seamlessly with Kubernetes’ control loop, CRDs require administrative privileges and complicate versioning and upgrades. Gerald cited the AWS Controllers for Kubernetes (ACK), still in v1alpha1, as an example of versioning challenges, where upgrades introduce new failure modes and complexity.
Toward Simpler, Developer-Friendly Solutions
Reflecting on developer experience, Gerald advocated for alternatives like config maps over CRDs. He praised Grafana’s approach, which uses config maps for dashboards, avoiding the need for administrative access and simplifying deployment. Similarly, older Prometheus setups allowed developers to control scraping with simple annotations, granting autonomy without the overhead of CRDs. Gerald argued that controllers alone, paired with config maps, provide most of the operator’s benefits without the downsides.
He concluded with a decision framework: only pursue the operator pattern if the use case justifies a domain-specific language. For operational tasks with low blast radius, like Kyverno’s policy enforcement, operators excel. However, for development teams, Gerald recommended avoiding custom operators, as they often lead to wasted effort and fragility, as seen in his team’s abandoned six-month project.