# Runbook: Booking store failover to the standby region

- Owner: Platform team (Tobias Okonjo)
- Last reviewed: 2024-07-29
- Related: ADR-005 (regional standby)

## When to use it

The primary region is unavailable or the booking store there cannot accept writes, and the outage is
expected to last longer than the thirty-minute recovery time objective.

## Approval

The on-call site reliability engineer proposes the failover in the incident channel. The platform
team lead or the chief technology officer approves it. Do not start without that approval.

## Steps

1. Stop the booking services in the primary region, so that nothing writes to the old primary if it
   comes back.
2. Record the standby's replication lag from the dashboard. This is the window of writes that may be
   lost.
3. Promote the standby to primary.
4. Point the booking services in the standby region at the new primary and start them.
5. Reconcile payments: list every Larchpay charge in the lag window and compare it with bookings in
   the new primary. For each charge without a booking, settle it as in the payment webhook runbook.
6. Announce the failover to operators through the status page.

## Failing back

Failing back is a planned change, not part of the incident: rebuild the old region as a standby, let
it catch up, and switch during a quiet hour.
