Idempotent payouts

The same lesson as the task queue, applied to money

You've seen this shape before: a Solid Queue job can run at least once — a worker can crash right after finishing the work but before marking the job done, and the job gets retried. For most jobs (like BookingConfirmationJob sending an email) an extra retry is a harmless duplicate email at worst.

A payout job retrying is not harmless. If PayoutJob pays a walker $40 and then crashes before recording that it succeeded, a naive retry pays them ANOTHER $40. Nobody notices until the walker gets a $2,000 payout for a week of $40 walks — and by then the money is gone.

The fix is the same shape as before: an idempotency key. Give every payout attempt a unique key (one per real-world payout event — say, one per booking being paid out). Make that key a column with a UNIQUE database index. Before creating a payout, try to insert with that key; if the insert fails because the key already exists, the payout already happened — return the existing row instead of paying again.