Scaling background work

One big queue becomes one big bottleneck

Module 12 introduced Solid Queue for background jobs — email confirmations, webhook handling. At real volume, dumping every job into ONE queue means a burst of low-priority work (say, a batch of confirmation emails) can sit ahead of something time-critical (a payout), just because it got enqueued first. The fix is separate named queues: queue_as :payments, queue_as :mailers, queue_as :default, each with its own priority and its own concurrency limit — a cap on how many jobs from that queue run at once, so one noisy queue can't starve the workers every other queue needs.

Failures are a fact of background work — a network blip, a momentary database lock. retry_on tells a job how to react to a specific error instead of just dying: retry_on ActiveRecord::Deadlocked, wait: :polynomially_longer, attempts: 5 retries automatically, waiting longer between each attempt (exponential backoff) instead of hammering the same failure five times in a row.

Here's the rule that makes retries safe at all: every job MUST be idempotent — running it twice (or five times) has to produce the SAME result as running it once. Ties directly back to module 19's payout job: if a payout job isn't idempotent and Solid Queue retries it after a timeout that actually succeeded, a walker gets paid twice for the same booking. The unique idempotency key from module 19 is exactly what makes a retried payout safe — the second attempt recognizes the first one already happened and does nothing.

Once queues are split by priority, the next question is whether they're keeping up — monitoring queue depth (how many jobs are waiting, per queue) tells you that. A payments queue with a growing backlog is a signal to add workers or investigate a stuck job, long before a user notices anything's wrong.