How to Stop a Rogue Database From Taking Writes

When a database fails over, the old primary can cause data corruption. Here's how ClickHouse's control plane safely isolates it.

Ensuring a database stays online is hard, but ensuring its data remains correct during a failure is even harder. In a new blog post, the team behind ClickHouse Managed Postgres detailed their solution to a classic database problem: what to do with an old primary server after a failover. When a primary PostgreSQL database goes down and a standby replica is promoted to take its place, the old primary might not be truly dead. If it comes back online, it may not know it has been replaced and could start accepting new data. This creates a dangerous "split-brain" scenario, where two databases think they are in charge, leading to data divergence, corruption, and a massive cleanup headache. To prevent this, systems need a way to reliably isolate, or "fence," the old primary before the new one takes over.

ClickHouse's solution is a control plane that orchestrates this fencing process across multiple environments. The core idea is to block all incoming write traffic to the old primary server before promoting the new one. The implementation is tailored to the infrastructure: on AWS, the system modifies security groups to deny traffic; on Google Cloud, it uses firewall rules; and for bare-metal servers, it relies on a custom agent that directly manipulates local `iptables` firewall rules. This multi-pronged approach ensures that no matter where the database is running, the control plane has a reliable, out-of-band method to isolate the potentially rogue server. This guarantees that once the failover is complete, only the new, correct primary database is accepting application writes, preserving data integrity.

This challenge of fencing is a fundamental issue in designing any high-availability distributed system, not just databases. The most extreme and traditional solution is known as STONITH, or "Shoot The Other Node In The Head," which involves forcibly powering down the old machine. ClickHouse’s software-defined approach is more nuanced and flexible, adapting to modern cloud environments. A key architectural choice they highlighted is their "fail-open" design. This means if the fencing mechanism itself fails for any reason, the system will proceed with the failover anyway. This prioritizes availability over perfect consistency, accepting a small risk of a split-brain in order to prevent a complete service outage. The alternative, a "fail-closed" system, would halt the failover if fencing failed, guaranteeing consistency but causing downtime.

For engineering teams managing their own PostgreSQL clusters, this post serves as a valuable, real-world case study. It demonstrates that a simple failover script is insufficient for a production system; a robust fencing mechanism is a critical component for true reliability. The discussion of a fail-open versus fail-closed strategy is particularly important, as it forces teams to confront a difficult trade-off between uptime and data consistency. The right answer depends entirely on the specific application's requirements. Ultimately, ClickHouse's deep dive reveals the hidden complexities behind providing a reliable managed database service and offers a clear blueprint for others tackling the same difficult problem.

Data corruption from a "split-brain" scenario is a nightmare for any engineering team. This post details a real-world, multi-cloud solution to a fundamental high-availability problem in PostgreSQL, offering a practical blueprint for SREs and database administrators managing critical systems.

A database split-brain can lead to silent data loss, inconsistent application behavior, and significant financial and reputational damage. Implementing robust fencing strategies like this is crucial for business continuity, protecting revenue-generating applications and maintaining customer trust during infrastructure failures.