Building a Payment Incident Response Process for Trading Platforms

Header image: conceptual illustration.

Payment incidents are inevitable: a provider has an outage, webhooks stop arriving, a payout batch fails. What separates a short disruption from a crisis is a process prepared in advance. This guide adapts general incident-management practice — such as the approach described in Google's Site Reliability Engineering book chapter on managing incidents — to payments on trading platforms.

What counts as a payment incident

• A payment method failing or degraded.
• Webhooks or callbacks not received.
• Deposits confirmed but not credited.
• Payouts failing or stuck in unknown status.
• Settlement missing or short.
• Suspected duplicate credits or payments.

Severity levels (example)

• Sev 1: clients' money at risk or widespread failure of deposits or withdrawals.
• Sev 2: one method or provider degraded; workarounds exist.
• Sev 3: limited impact, e.g. delayed reports.

Define yours in writing, with examples.

Roles

Following the SRE model of clear separation:

• Incident lead: coordinates, decides, keeps the timeline.
• Operations/engineering: investigates and fixes.
• Communications: updates support, clients and management.
• Provider liaison: contacts the payment provider and tracks their response.

In a small team one person may hold several roles, but name them at the start.

The response steps

1. Detect. Alerts from payment dashboard metrics, support tickets, or provider notices.
2. Declare. Open an incident channel and assign roles.
3. Contain. Hide affected methods at checkout, pause automated payouts on an affected rail, stop retries that could create duplicates.
4. Communicate. Give support approved wording; update the status page where appropriate.
5. Resolve. Fix or wait for the provider; then process the backlog carefully, using status checks rather than blind resends.
6. Reconcile. Run matching for the incident window to find any missed credits or duplicates.
7. Review. Hold a blameless review and track follow-up actions.

Client communication templates

• During: "Some deposits by [method] are currently delayed. Your funds are not lost; we will update balances as soon as payments are confirmed. Please don't pay again."
• After: "The issue affecting [method] has been resolved. Delayed payments have been processed. If something still looks wrong, contact support with your reference."

Never ask clients for PINs, OTPs or passwords during an incident; fraudsters exploit outages to impersonate support.

Hypothetical incident

*Invented.* Webhooks from one provider stop at 20:05. The pending-age alert fires at 20:20. The lead hides the method, the provider liaison opens a ticket, and support receives the template. The provider fixes its sender at 21:10; a status-check job processes 87 pending payments; reconciliation the next morning confirms no duplicates. The review adds a "zero webhooks" alert to catch the problem in five minutes next time.

Frequently asked questions

Should we automatically switch to another provider?

Automatic failover helps if you have a tested second provider; otherwise hiding the method is safer. See payment uptime and failover.

Who writes the post-incident review?

Usually the incident lead, with input from everyone involved.

Sources

• Google SRE Book, "Managing Incidents": https://sre.google/sre-book/managing-incidents/