Disaster recovery for a check-clearing hub: keeping a payment database and its check images in step

Brandon Eilers · Lead Software Engineer

Early in my career, at US Dataworks, I worked on Clearingworks, an enterprise payment-processing platform: a Java application running on Unix servers with a PostgreSQL database behind it. Its clients processed real money through it. Verizon's paper-check payments ran through Clearingworks, integrated with BNY Mellon, and The Bankers Bank (TBB), a correspondent bank, ran its check-image exchange network for community banks on the platform.

I built and documented the disaster recovery setup for TBB's environment. This is what that involved, and what it taught me about recovering systems that hold more than one kind of state.

Why a clearing hub raises the stakes

A correspondent bank isn't a single customer. When its image exchange is down, every community bank that depends on it is stuck too, and check processing runs against daily deadlines. Recovery had to be fast, repeatable, and something the operations team could run without me in the room.

Two kinds of state, two replication paths

The platform kept its data in two places, and both had to survive a site loss together:

• Transactional records in PostgreSQL. These were replicated to a standby database at the DR site, first with WAL log shipping and then with streaming replication.
• Check images and application files on the Unix file systems. These were mirrored to the DR site by rsync cron jobs on a schedule, copying only what had changed.

Backups ran alongside both, so recovery didn't depend on replication alone.

From log shipping to streaming replication

The database replication started as log shipping: PostgreSQL writes every change to its write-ahead log (WAL) in 16 MB segments, and each completed segment was copied to the DR site and replayed there. It works, but the standby only learns about changes once a segment fills and ships, so it can trail production by up to a whole segment's worth of activity.

When PostgreSQL 9 introduced built-in streaming replication, I moved the DR setup to it. The standby connects to the primary and receives WAL records as they're written, instead of waiting for whole files. For a platform processing checks for other banks, shrinking that window of work that could be lost in a site failure was the point.

The hard part: the two paths don't move at the same speed

Streaming replication keeps the standby only moments behind production; a scheduled file sync runs on its own clock. At any moment, the DR database can reference a check image that hasn't reached the DR file system yet. That isn't a bug to eliminate so much as a fact to plan around: the failover procedure has to account for the gap, by bringing the file mirror fully up to date (or confirming it is) before the DR site takes traffic, rather than assuming both paths stopped at the same instant.

The runbooks were the deliverable

Replication you can't operate under pressure isn't disaster recovery. I documented every process end to end: how each environment was built, how replication and the file sync were monitored, and the step-by-step procedures for failing over and failing back. The goal was that whoever was on call could recover the platform by following the document, in order, on a bad day.

Then we proved it: we ran a failover test, bringing the platform up at the DR site from the replicated database and mirrored files, to verify that everything worked end to end before we ever needed it for real.

The same habit carried through my career. Years later, at JPMorgan Chase, the documentation and knowledge transfer I led let a new Site Reliability Engineering team support our application on their own.

What I'd tell someone building DR for a system like this

• Inventory every kind of state, not just the database. Files, images, and configuration need their own path to the DR site.
• Know how far behind each path can be, and write the failover steps around the slowest one.
• Keep backups even with replication. Replication faithfully copies mistakes too.
• Make the runbook the product. If recovery depends on one person's memory, it isn't recovery.
• Test the failover before you need it. A procedure that has never been run is a hypothesis; ours was verified with a failover test.