# Incident report: Duplicate refunds and charges during the August failover

- Date of incident: 2024-08-14
- Incident commander: Priya Ramanathan
- Status: Closed on 2024-09-02

## Summary

On 14 August 2024 the primary region lost its storage network for 52 minutes. We failed over to the
standby region following the database failover runbook. During payment reconciliation, 37 passengers
were refunded twice, and 6 passengers were charged twice when their bookings were settled by hand.

## Timeline

- 09:12 Storage alerts in the primary region; bookings start failing.
- 09:31 Failover proposed by the on-call engineer and approved by Tobias Okonjo.
- 09:48 Standby promoted; bookings resume. Replication lag at promotion: 3 minutes 40 seconds.
- 10:05 to 11:20 Payment reconciliation from the console.
- 12:40 Support reports passengers with two refunds.

## What went wrong

1. **Refunds were not idempotent.** ADR-004 requires an idempotency key on every refund, but the
   console's bulk refund action generated a new key on each click. Two engineers refunded the same
   list of bookings.
2. **Hand settlement skipped the charge reference.** Six bookings were settled by issuing new charges
   instead of using the existing charge's reference, against step 4 of the payment webhook runbook.
3. **Old logs were missing.** One passenger's chargeback concerned a booking from early June. Payment
   logs older than 30 days had already been deleted under ADR-003, so we could not show the original
   payment flow and lost the chargeback.

## What went well

The failover itself took 17 minutes from approval to bookings resuming, inside the thirty-minute
recovery time objective.

## Actions

- Make the console's bulk refund derive its idempotency key from the booking, as every other call
  does. (Done 2024-08-21.)
- Remove "charge again" from the hand-settlement screen. (Done 2024-08-23.)
- Review how long payment logs are kept. (See the payment log retention follow-up.)
