Payment Webhooks & Reconciliation

Verify every webhook to prevent spoofed events from attacking your system.

Signature verification:

Stripe:

• Uses HMAC-SHA256 with webhook secret
• Signature in Stripe-Signature header
• Timestamp prevents replay attacks
• Use Stripe's SDK verification methods

MangoPay:

• Uses webhook signature header
• Verify using shared secret
• Check timestamp freshness

Implementation rules:

• NEVER skip signature verification
• HTTPS only (reject HTTP)
• Consider IP whitelisting (additional layer)
• Reject invalid signatures immediately

Example (Stripe):

1. Extract signature from header
2. Compute expected signature using raw body + secret
3. Compare using constant-time comparison
4. Reject if mismatch

Always use the SDK's built-in verification—don't implement your own.

Webhooks may be delivered multiple times. Your handler must produce the same result regardless.

Why duplicates happen:

• Your server was slow to respond (timeout → retry)
• Network issues during acknowledgment
• Payment provider retry logic
• Manual retries during debugging

Idempotency implementation:

Event ID tracking:

• Store processed event IDs
• Check before processing
• Skip if already processed
• Use database unique constraint

State machine approach:

• Define valid state transitions
• Ignore events that don't advance state
• subscription: active → canceled (valid)
• subscription: canceled → active (check carefully)

Database transaction:

• Wrap processing in transaction
• Rollback on any failure
• Prevents partial updates

Example flow:

1. Receive webhook
2. Verify signature
3. Check if event_id in processed_events
4. If exists, return 200 (acknowledge, don't reprocess)
5. If new, process and store event_id atomically

Production webhook handling requires robust architecture.

Acknowledge quickly:

• Return 200 within 5-10 seconds
• Providers retry on timeout
• Queue heavy processing for async

Async processing pattern:

1. Receive webhook
2. Verify signature
3. Store raw event in database
4. Return 200 immediately
5. Background job processes event
6. Retry logic for processing failures

Error handling:

• Return 4xx for invalid signatures (don't retry)
• Return 5xx for server errors (do retry)
• Never return 200 if you didn't process successfully

Dead letter queue:

• Store events that fail processing
• Alert on accumulation
• Investigate and replay manually
• Don't lose events

Monitoring:

• Track webhook volume
• Alert on delivery failures
• Monitor processing lag
• Dashboard for operational visibility

Reconciliation ensures your data matches the payment provider's records.

Why reconciliation matters:

• Webhooks can be missed (outages, bugs)
• Processing errors may occur
• Prevents revenue leakage
• Required for accurate accounting

Reconciliation levels:

Real-time (per transaction):

• After webhook processing, verify via API
• Catch immediate discrepancies
• Expensive at scale

Batch (daily/weekly):

• Pull transaction list from provider
• Compare against internal records
• Flag mismatches for investigation

Settlement (monthly):

• Match bank deposits to expected payouts
• Verify fee calculations
• Generate audit reports

Implementation:

1. Pull transactions from payment provider API
2. Match against internal transaction table
3. Identify: missing, extra, amount mismatches
4. Generate exception report
5. Investigate and resolve discrepancies

Learn from common webhook and reconciliation problems.

Missed webhooks:

• Cause: Server downtime, network issues
• Solution: Daily reconciliation catches gaps
• Prevention: Redundant webhook endpoints

Out-of-order events:

• Cause: Async delivery, retries
• Solution: Check timestamps, use state machine
• Don't assume sequential delivery

Duplicate charges:

• Cause: Retry logic bugs, race conditions
• Solution: Idempotency keys on creation
• Reconciliation catches after the fact

Refund mismatches:

• Cause: Refund webhook missed
• Solution: Pull refund list in reconciliation
• Alert on unexpected balance changes

Fee discrepancies:

• Cause: Rate changes, volume tier changes
• Solution: Pull fee details from provider
• Monthly fee reconciliation

Currency conversion:

• Cause: Different conversion times
• Solution: Use consistent rate source
• Track conversion rate with transaction

Best practices:

• Log everything (requests, responses, decisions)
• Build reconciliation from day one
• Alert on any discrepancy
• Regular audits of payment data