Background Jobs That Survive a Crash

We shipped on a Thursday afternoon. Rolling deploy, two workers replaced one after the other, health checks green, nothing in the error tracker. On Friday morning a colleague asked why roughly two hundred users who signed up the day before never got their welcome email.

There was no exception to look at. The jobs had been picked up, the process received SIGTERM mid-send, and Kubernetes gave it the usual grace period before SIGKILL. The work vanished somewhere in the gap. From the queue's perspective those jobs had been handed out and never came back; from our perspective nothing had failed, because nothing had reported failing.

That is the shape of most background job incidents I've dealt with. Not a crash you can see in a stack trace — a crash you can only see by counting things that should exist and don't.

The request handler that owns work it can't finish

Here is the bug in its original form, and it is in almost every codebase at some point:

@Post('signup')
async signup(@Body() dto: SignupDto) {
  const user = await this.users.create(dto);
 
  await this.mailer.sendWelcome(user.email);   // 400ms, sometimes 9s, sometimes throws
  await this.crm.syncContact(user);            // third-party, no SLA you control
 
  return { id: user.id };
}

The user's HTTP request is now responsible for two things that have nothing to do with answering the user. If the mail provider is slow, signup is slow. If the CRM is down, signup returns 500 for a user who was, in fact, created — so they retry, and now you have a duplicate account problem on top of an email problem.

The deeper issue is ownership. A request lives for a few hundred milliseconds. The work of "make sure this person receives a welcome email" has to survive longer than that: across a provider outage, across a deploy, across the process being replaced. You cannot express that lifetime inside a request handler. The handler exits and takes its intentions with it.

So the request does the minimum that must be true before responding, and hands the rest over as a durable record:

@Post('signup')
async signup(@Body() dto: SignupDto) {
  const user = await this.users.create(dto);
 
  // The queue write is part of the same commit boundary in spirit:
  // if this throws, the caller sees an error and can retry safely.
  await this.emailQueue.add(
    'welcome',
    { userId: user.id },                       // an ID, not the user object
    { jobId: `welcome:${user.id}` },           // dedupe key — see below
  );
 
  return { id: user.id };
}

The response is now fast and honest. The email will happen, eventually, and "eventually" is a property the queue is designed to provide.

At-least-once is the floor, not a setting you can turn off

Every durable queue I've worked with — BullMQ, SQS, RabbitMQ — gives you at-least-once delivery. Not because the authors were lazy, but because exactly-once across a network with independently failing processes isn't something a queue can hand you. There is always a moment where the job has been done and the acknowledgement hasn't been recorded, and the only two options at that point are "possibly do it twice" or "possibly never do it at all."

Queues choose the first. That is the right choice, and it moves the burden to you: your consumer must be idempotent. Not "should be, when convenient." Must. A worker that charges a card without a guard will eventually charge twice, because a worker will eventually die between the charge and the ack.

For the email case, the guard is a claim in the database:

async function sendWelcomeOnce(userId: string) {
  // Claim first. Whoever wins the insert owns the send.
  const claimed = await db.$executeRaw`
    INSERT INTO email_sends (user_id, kind)
    VALUES (${userId}, 'welcome')
    ON CONFLICT (user_id, kind) DO NOTHING
  `;
 
  if (claimed === 0) return;   // already sent, or being sent — this run is a no-op
 
  await mailer.sendWelcome(userId);
}

The jobId in the producer helps too, but it solves a different problem: it stops the same job being enqueued twice. It does nothing about the same job being delivered twice, which is the case that at-least-once guarantees will happen.

Acknowledged is not the same as completed

This distinction is where most of the subtle bugs live.

When a worker picks up a job, the queue does not consider it finished — it considers it in progress and starts a clock. In BullMQ the job moves from wait to active and its lock is renewed every lockRenewTime. Only when your processor function resolves does the job move to completed. If the process dies while the job is active, nobody resolves anything, the lock expires, and the queue reclaims the job as stalled.

That reclaim is the entire reason the two hundred emails were recoverable in principle — and the reason they weren't in practice was that we had maxStalledCount at its default of 1 and a lockDuration shorter than our slowest send. A job that takes longer than the lock without renewing is not "slow," it is "presumed dead," and the queue will hand it to someone else while the original worker is still happily running it.

const worker = new Worker(
  'email',
  async (job) => {
    await sendWelcomeOnce(job.data.userId);
  },
  {
    connection,
    concurrency: 10,
    // Longer than the p99 of the job body, not the average.
    lockDuration: 60_000,
    // How many times a stalled job may be recovered before it goes to failed.
    maxStalledCount: 3,
    stalledInterval: 30_000,
  },
);

Two rules I now apply without thinking. Set lockDuration from the slow tail of the job, not the median. And if a job legitimately runs longer than the lock, renew it explicitly with job.extendLock() or job.updateProgress() — silence is indistinguishable from death.

Backoff, and why fixed intervals make outages worse

A retry policy that says "try again in 5 seconds" seems harmless until the failure is upstream and shared.

The mail provider goes down for two minutes. Five hundred jobs fail within the same second. All five hundred retry five seconds later, together. They all fail together, retry five seconds later, together. You have built a metronome that hits a struggling service with a synchronised wave every five seconds, and when the provider finally recovers, the first thing it receives is five hundred simultaneous requests — which may well knock it over again.

Exponential backoff spreads the attempts out over time. Jitter spreads them out within each attempt window, which is the part people skip and the part that actually breaks the synchronisation.

await emailQueue.add('welcome', { userId }, {
  attempts: 5,
  // 'custom' is the only value that dispatches to backoffStrategy.
  // The valid set is 'fixed' | 'exponential' | 'custom' — an arbitrary
  // name here is silently not what you meant.
  backoff: { type: 'custom' },
  removeOnComplete: { age: 3600, count: 1000 },
  removeOnFail: false,                  // keep failures for inspection
});
 
// The strategy lives on the Worker's settings, not on the Queue.
const worker = new Worker('email', processor, {
  connection,
  settings: {
    backoffStrategy: (attemptsMade: number) => {
      const base = Math.min(1000 * 2 ** attemptsMade, 5 * 60_000);  // cap at 5 min
      return Math.floor(base / 2 + Math.random() * (base / 2));     // full-ish jitter
    },
  },
});

Two things matter in that formula. The cap, because unbounded exponential backoff eventually schedules a retry for next week. And the randomness, because without it five hundred jobs that failed together will retry together forever, no matter how clever the curve is.

The other half of a retry policy is knowing what not to retry. A 500 from the provider is worth retrying. A 422 "invalid email address" is not — it will fail identically five times and burn five minutes of worker capacity to reach a conclusion you already had.

try {
  await mailer.sendWelcome(userId);
} catch (err) {
  if (err.status >= 400 && err.status < 500 && err.status !== 429) {
    // Permanent. Stop retrying and route it for a human to look at.
    throw new UnrecoverableError(`invalid recipient: ${err.message}`);
  }
  throw err;   // transient — let backoff do its job
}

The dead letter queue is a to-do list

A job that has exhausted its attempts should not disappear, and it should not sit in a failed set that nobody has ever opened. The failed set is a queue of work that needs a decision.

I treat it as an operational surface with three properties: it is alerted on when it grows, every entry carries enough context to act on, and there is a supported way to put a job back.

worker.on('failed', async (job, err) => {
  if (job.attemptsMade < job.opts.attempts) return;   // still retrying, not dead yet
 
  logger.error({
    queue: job.queueName,
    jobId: job.id,
    name: job.name,
    data: job.data,          // IDs only, so this is safe to log
    attempts: job.attemptsMade,
    reason: err.message,
  }, 'job exhausted retries');
 
  metrics.increment('jobs.dead', { queue: job.queueName, name: job.name });
});

Then replay is a script, not an archaeology project:

const dead = await emailQueue.getFailed(0, 500);
for (const job of dead) {
  // attemptsMade is NOT reset by default — a job that already exhausted
  // `attempts` would return to wait and fail again on the first error.
  await job.retry('failed', { resetAttemptsMade: true });
}

The reason replay has to be safe is the same reason as everything else here: consumers are idempotent, so re-running a job that partly succeeded is boring rather than dangerous. If replaying your DLQ is scary, the problem isn't the DLQ.

Small jobs, because you can't resume a monolith

There is a strong pull toward writing "process the monthly report for all 40,000 users" as one job. It's simple to reason about and simple to schedule. It is also unrecoverable: it fails at minute 38 of 40, the retry starts from user zero, and every retry has the same probability of hitting the same wall.

The alternative is a fan-out. One small coordinator job that enqueues many small unit jobs:

// Coordinator: cheap, fast, and safe to run twice.
async function scheduleMonthlyReports(job: Job<{ period: string }>) {
  const { period } = job.data;
  let cursor: string | undefined;
 
  do {
    const page = await db.user.findMany({
      where: { active: true, ...(cursor && { id: { gt: cursor } }) },
      orderBy: { id: 'asc' },
      take: 500,
      select: { id: true },
    });
 
    await reportQueue.addBulk(
      page.map((u) => ({
        name: 'report',
        data: { userId: u.id, period },
        opts: { jobId: `report:${period}:${u.id}` },   // one report per user per period
      })),
    );
 
    cursor = page.at(-1)?.id;
  } while (cursor);
}

Now a crash costs you one user's report, retried in seconds. Progress is durable because it's recorded in the queue itself rather than in the memory of a long-running loop. You get parallelism for free, you get a real progress number by reading queue counts, and the jobId makes the coordinator safe to re-run after its own crash.

The trade-off is honest: you've turned one log line into 40,000 jobs, which is more Redis memory and more noise. Aggressive removeOnComplete handles that. It's a good trade, but it is a trade.

Payloads carry IDs, not objects

Putting the whole entity in the payload feels efficient — the worker doesn't need to query. Two things go wrong.

The payload is a snapshot from enqueue time. If the job sits in a backlog for twenty minutes, or fails and retries an hour later, the worker acts on a version of reality that no longer exists. I've seen an email sent to an address the user had already corrected, because the correction happened after the job was queued.

And payloads are stored in Redis, serialised, for every job and every retry. Ten kilobytes of embedded object times a large queue is real memory, and it's memory you'll notice at exactly the wrong moment.

// Fragile: a snapshot that ages badly and costs memory.
await queue.add('welcome', { user, template, attachments });
 
// Durable: identifiers plus the small facts that must not change.
await queue.add('welcome', {
  userId: user.id,
  templateVersion: 'welcome_v4',   // pin behaviour, not data
});

The distinction I use: the payload carries what identifies the work and what pins its meaning. Everything mutable gets read at execution time, from the source of truth.

Shutting down on purpose

Back to the two hundred emails. The root cause wasn't the queue — it was that our process treated SIGTERM as "die now" instead of "stop taking new work and finish what you have."

async function shutdown(signal: string) {
  logger.info({ signal }, 'shutting down');
 
  // close() is already the graceful path: it stops pulling new jobs and
  // waits for active ones. close(true) forces and skips the wait.
  await worker.close();
  await queue.close();
  await redis.quit();
 
  process.exit(0);
}
 
process.on('SIGTERM', () => void shutdown('SIGTERM'));
process.on('SIGINT', () => void shutdown('SIGINT'));

In NestJS the same thing comes from enabling shutdown hooks and letting the module close the worker:

@Injectable()
export class EmailWorker implements OnApplicationShutdown {
  async onApplicationShutdown() {
    await this.worker.close();   // drains in-flight jobs before the process exits
  }
}
// main.ts
app.enableShutdownHooks();

Two constraints make this real. Your orchestrator's grace period must exceed your longest job — if Kubernetes SIGKILLs after 30 seconds and a job takes 90, graceful shutdown is a comment, not a behaviour. And when the deadline genuinely can't be met, you fall back to the stalled-job recovery from earlier. Graceful shutdown is the fast path; stall recovery is the guarantee.

When not to use a queue

The failure mode opposite to "await sendEmail in the handler" is queueing everything, and it has its own costs: an extra hop, an extra system to monitor, and a user experience where nothing appears to happen until it mysteriously does.

Keep it synchronous when the work is fast and the caller genuinely needs the result. Writing the row you're about to return. Validating a coupon. Resizing one small image that the response includes a URL for. Wrapping a 30-millisecond operation in a job gives you eventual consistency you didn't need and a "why isn't it there yet" bug you have to explain.

Queue it when the work is slow, when it talks to a system you don't control, when it must survive a restart, or when it can be retried independently of the caller. If the answer is genuinely "the user must see this succeed or fail right now," a queue is the wrong tool and no amount of polling on the frontend will make it the right one.

What this really trains

Every technique here is a variation of the same move: make the state of in-progress work durable and externally visible, so that a process dying is an interruption rather than an erasure.

Idempotent consumers, explicit acknowledgement, stall recovery, jittered backoff, small resumable units, ID-only payloads, deliberate shutdown — none of them prevent crashes. They make crashes uninteresting. A worker is allowed to die at any instant, and the system's answer should be a slightly later timestamp on some row, not a support ticket.

The test I now apply to any background pipeline is a single question: if I kill a worker with -9 right now, what does the system lose? If the answer is "a few seconds," it's finished. If the answer requires reading the code to work out, it isn't.