# Runbook: Payment webhook failures

- Owner: Payments (Hana Sato)
- Last reviewed: 2024-07-22
- Related: ADR-004 (payment provider and idempotency)

## Symptoms

- Larchpay's dashboard shows webhooks to Ferrylog failing or retrying.
- Bookings stay in "payment pending" for more than 15 minutes.
- Passengers report being charged without receiving a ticket.

## Steps

1. Confirm the webhook endpoint answers. If it returns errors, check the booking service's
   deployment and roll back a deployment made in the last hour.
2. Once the endpoint answers, Larchpay retries failed webhooks on its own for up to three days. Each
   retried webhook is processed at most once, because `payment_event` rejects an event identifier it
   has already recorded.
3. For a booking still pending after the endpoint recovers, replay its payment from the payments
   console. The replay reuses the original idempotency key, so it returns the original result and
   cannot charge the passenger again.
4. If Larchpay shows a successful charge and Ferrylog shows no booking, settle the booking from the
   console with the charge's reference rather than issuing a new charge.

## Escalation

If charges and bookings disagree for more than ten passengers, declare an incident and page the
payments engineer on call.
