Setting Up Background Workers with Postgres, Redis, or Queues: 2026 Guide
August 4, 2026
Background workers are one of the simplest ways to make an application feel fast even when the work behind the scenes is not. Instead of making users wait while your API sends emails, generates reports, processes images, or syncs data with another service, you can hand that work off to a worker process and return control immediately. The result is lower request latency, fewer timeouts, better resilience under load, and a user experience that feels much more responsive.
But building background jobs well is not just about “putting stuff in a queue.” Reliable worker systems have to survive crashes, duplicate deliveries, temporary outages, and spikes in load. They also need clear operational visibility so you can tell whether jobs are actually moving, failing, or piling up. In practice, teams usually choose one of three paths:
use Postgres as a job store,
use Redis as a queue,
or adopt a dedicated broker/ecosystem like RabbitMQ, Kafka, SQS, or a framework-managed queue.
Each option can be the right answer, depending on scale, reliability needs, and team constraints. The trick is understanding what problem you’re solving and what trade-offs you are willing to accept.
1) Why background workers matter
The main reason background workers matter is straightforward: they move slow, non-interactive work out of the request path. When a user clicks “submit,” “upload,” or “checkout,” your API should spend its time validating input, writing the essential state change, and responding quickly. Anything that can safely happen later is a strong candidate for a worker.
This matters because request latency is often the first thing users notice. If a web request has to wait on third-party APIs, long database writes, file conversions, or complex business logic, the user experience degrades quickly. Workers reduce the chance that a temporary slowdown turns into a full outage. They also give you room to absorb bursts. A queue can buffer 10,000 incoming jobs far more gracefully than a synchronous endpoint can execute 10,000 long-running tasks.
Background workers are especially useful for:
sending email or SMS notifications,
resizing images or transcoding media,
syncing data to external systems,
generating exports and reports,
running cleanup and maintenance tasks,
processing webhooks,
and retrying flaky integrations.
A worker architecture also improves product reliability by separating concerns. The web request path can stay focused on user-facing correctness, while the worker path focuses on eventual completion. That separation is important because many background tasks do not need to be immediately visible to the user, only eventually successful. In systems like SQS, messages are intentionally processed with an at-least-once model and visibility timeouts to support this style of recovery and retry. (docs.aws.amazon.com)
The caveat is that asynchronous work introduces complexity. Once you hand work to a queue, you need to think about delivery guarantees, duplicate processing, and how to tell whether a job is still running, retrying, or dead. That is the price of moving from “simple sync code” to “reliable distributed execution.”
2) The core reliability requirements
Reliable background workers usually need the same core guarantees, no matter what technology you choose:
At-least-once delivery
At-least-once delivery means a job may be processed one or more times, but it should not be silently lost. This is the most common practical guarantee for worker systems. It is widely supported by queue systems such as SQS and by worker frameworks like Celery when configured for late acknowledgments. (docs.aws.amazon.com)
Atomic handoff
The handoff from “job is available” to “job is claimed by a worker” should be atomic. If two workers race for the same job, only one should win. In Postgres, this is often done with row locks and SKIP LOCKED; in Redis, it’s commonly done with atomic commands or consumer-group claim semantics; in dedicated brokers, the broker itself often owns the handoff protocol. PostgreSQL explicitly documents SKIP LOCKED as a way to avoid lock contention in queue-like tables. (postgresql.org)
Crash recovery
If a worker process dies halfway through a job, the system must recover. This usually means the job becomes visible again after a timeout, or the task remains unacknowledged and is redelivered. SQS uses visibility timeouts for this, and Celery’s late-ack mode acknowledges tasks after execution so a crash can trigger redelivery. (docs.aws.amazon.com)
Retries
Retries are not optional in distributed systems. Network failures, rate limits, dead APIs, and intermittent database issues happen. Good queues let you retry with backoff so you don’t hammer a failing dependency. SQS supports reappearance after visibility timeout, and frameworks such as Celery include retry primitives. (docs.aws.amazon.com)
Visibility timeouts and leases
A visibility timeout is a lease on a job. While a worker holds the lease, other workers should not process the same item. If the worker fails to finish in time, the job becomes visible again. SQS documents this directly and notes that the maximum visibility timeout is 12 hours from ReceiveMessage. (docs.aws.amazon.com)
Job status tracking
For real applications, you also need to know where a job stands: queued, running, succeeded, failed, retried, or dead-lettered. Status tracking can live in the queue system, in your application database, or both. Without status tracking, support teams and users are left guessing when a job “just hasn’t finished yet.”
3) Using Postgres as a job store
Using Postgres for background jobs is popular because it is already in the stack, familiar to the team, and transactional. A queue table can hold jobs with fields like id, payload, status, run_at, attempts, locked_at, and failed_at. Producers insert jobs inside the same transaction as the business record, which makes the “write business data + enqueue work” handoff much safer than writing to two unrelated systems.
Table-backed queues and polling
The simplest design is a table where workers poll for pending jobs. A worker repeatedly queries for ready rows, claims one, performs the work, and updates the row when complete. This approach is easy to understand and can work well at modest scale.
The downside is polling overhead. If many workers keep querying an empty table, you waste database resources. You can reduce that with smarter polling intervals, indexes on status and schedule columns, or notification mechanisms, but the basic trade-off remains: your queue traffic shares the same database as your application traffic.
Transactional locking and SKIP LOCKED
A more robust pattern is to select rows with FOR UPDATE SKIP LOCKED, claim them in a transaction, and mark them as running. PostgreSQL documents SKIP LOCKED as useful for queue-like tables because it avoids lock contention between multiple consumers. It is not a perfect general-purpose read pattern, but it is a practical tool for worker queues. (postgresql.org)
That pattern gives you atomic claim behavior: if one worker has locked a row, another worker skips it and moves on. This is one reason Postgres-backed queues remain attractive for teams that want simplicity without introducing a separate broker.
Where Postgres background worker processes fit in
There is another meaning of “background worker” in the Postgres world: server-side background worker processes. PostgreSQL supports background worker infrastructure inside the database server itself, along with advisory locks at session or transaction level. That is useful for extensions, maintenance tasks, and specialized internal workloads, but it is different from an application-level job queue. (postgresql.org)
For most application teams, the queue lives in tables, not inside the database server process. The important distinction is this: Postgres as a job store is usually about using SQL tables and locks; Postgres background workers are a lower-level database capability and are not the same thing as your app’s job processing layer.
Best fit
Postgres works best when:
your throughput is moderate,
you already rely heavily on Postgres,
you want transactional enqueue semantics,
and you prefer fewer moving parts over maximum queue specialization.
It becomes less attractive when queue volume is very high, when worker concurrency is large enough to stress the database, or when job processing must be isolated from primary OLTP workloads.
4) Using Redis as a job queue
Redis is attractive for queues because it is fast, simple to deploy, and flexible. It supports several queue patterns, from plain lists to sorted sets to streams. Each comes with different trade-offs.
Lists
The classic Redis queue pattern uses LPUSH to enqueue and BRPOP/BLPOP to consume. It is simple and fast, which is why it became a common choice for lightweight background tasks. The limitation is that you have to think carefully about failure recovery. If a worker pops an item and then crashes before finishing, the message may be lost unless you add an extra processing list, acknowledgment flow, or requeue mechanism.
Sorted sets
Sorted sets are useful when you need scheduled jobs, retries, or delayed execution. You can score jobs by due time and have workers claim jobs that are ready. This gives you more control than a raw list, but it also means you are building more queue logic yourself.
Streams and consumer groups
Redis Streams are the more modern, queue-like option. Redis explicitly supports consumer groups for Streams, which lets multiple workers split the workload and track pending messages. The Redis Streams model is especially useful when you want ordered append-only event storage plus group-based consumption and reclaim flows. Redis’s own material highlights consumer groups as a way to partition workloads and route messages to consumers. (redis.io)
Streams are a better fit than lists when you need:
message acknowledgment,
recovery of unacked messages,
consumer group coordination,
and more explicit tracking of pending work.
Atomic claim/reclaim flows
One of Redis’s strengths is that you can design atomic claim-and-reclaim flows around queues and streams. That matters because crash recovery depends on knowing whether a job is merely in flight or truly lost. Consumer-group pending entries, claim operations, and re-delivery flows give workers a way to recover items from dead consumers without manually scanning the whole queue.
Best fit
Redis is a strong choice when:
you want a fast queue with relatively simple operational overhead,
your jobs are short to medium in duration,
you need delayed work or consumer groups,
and you are comfortable running Redis as part of your infrastructure budget.
The trade-off is that Redis is often a better queue than a system-of-record. If you need deep auditability, strict transactional coupling to business writes, or complex long-term job state, Postgres or a dedicated broker may be a better fit.
5) Dedicated queue brokers and ecosystems
Sometimes the right answer is not “use the database you already have” or “use Redis because it’s convenient.” Sometimes you should choose a purpose-built broker.
RabbitMQ
RabbitMQ is a classic choice for message routing, acknowledgments, and dead-letter handling. RabbitMQ documents that messages can be acknowledged, negatively acknowledged, or rejected, and that rejected messages can be dead-lettered or requeued depending on configuration. (rabbitmq.com)
RabbitMQ is a strong fit when you need:
flexible routing,
per-message acknowledgement semantics,
retries and dead lettering,
and a mature AMQP-style messaging model.
Kafka
Kafka is less of a job queue in the traditional sense and more of an event streaming platform. Its documentation emphasizes topics, partitions, and consumer groups, and explains that consumer groups divide work across process pools while preserving partition ordering. Kafka also tracks committed offsets so a consumer can recover after failure. (kafka.apache.org)
Kafka is usually the right tool when:
you need durable streams of events,
you care about replay and ordered partitions,
multiple downstream consumers must read the same data,
and your team is already operating Kafka well.
SQS
Amazon SQS is a managed queue with visibility timeouts and at-least-once delivery characteristics. That makes it appealing when you want simplicity, durability, and minimal broker management. AWS documents the visibility timeout model, redelivery behavior, and retry-related constraints clearly. (docs.aws.amazon.com)
SQS is a good choice when:
you want a managed service,
you can accept the AWS ecosystem,
and you want solid queue semantics without running a broker yourself.
Framework-backed queues
Frameworks such as Celery also wrap queue systems with task routing, retries, acknowledgments, and result tracking. Celery’s documentation notes that late acknowledgment means a task is acknowledged after execution, which helps ensure redelivery if a worker dies mid-task. (docs.celeryq.dev)
Framework-backed queues are attractive when your team wants application-level abstractions rather than broker-level plumbing.
When to choose dedicated brokers
Choose a dedicated broker when:
throughput is high,
durability and operational isolation matter,
routing patterns are complex,
multiple teams or services depend on the same messaging fabric,
or your queue is becoming important enough to deserve its own infrastructure.
6) Architecture patterns for workers
A good worker system is more than a queue and a process loop. The architecture matters.
Producer/consumer separation
The producer creates jobs; the consumer processes them. Keeping these roles separate makes it easier to scale, test, and reason about failures. Producers should do as little as possible beyond validating input, writing state, and enqueueing work atomically if possible.
Worker pools
Most practical systems use worker pools instead of one process per job. A pool lets you control concurrency, CPU use, memory pressure, and downstream dependency load. Kafka consumer groups are a good example of coordinated parallelism, where multiple consumers in the same group divide the workload. (kafka.apache.org)
Idempotency
Idempotency is crucial because at-least-once delivery means duplicates happen. Every job handler should be written so that repeating the same message does not produce duplicate side effects. Common techniques include:
storing a job UUID,
checking whether the side effect has already been applied,
using unique constraints,
and making external API calls idempotent when possible.
Dead-letter handling
Some jobs fail because the input is bad, not because the system is temporarily unhealthy. These jobs should not retry forever. RabbitMQ and SQS both have dead-letter concepts, and dead-letter queues are a standard way to isolate poison messages for later inspection. (rabbitmq.com)
Retries with backoff
Retrying immediately can make outages worse. Backoff spreads retries over time and gives downstream systems room to recover. A good policy usually mixes:
short retries for transient errors,
longer backoff for repeated failures,
and a maximum retry count before dead-lettering.
Horizontal scaling
The easiest way to scale many worker systems is to add more consumers. That only works well if the queue supports safe concurrent claiming and if the downstream dependencies can handle the increased load. Scaling workers without scaling the databases, APIs, and storage systems they depend on is a common mistake.
7) Operational concerns
Workers are easy to start and harder to operate. Once they are in production, you need visibility.
Monitor queue depth
Queue depth tells you how far behind the system is. If job count is steadily rising, consumers are not keeping up. That can signal insufficient capacity, slow dependencies, or a poison-message pattern. For SQS, AWS also exposes in-flight message constraints that become relevant at scale. (docs.aws.amazon.com)
Track lag and age
A queue with 100 jobs is less concerning than a queue with 100 jobs that have been waiting 45 minutes. Job age and processing lag matter more than raw depth because they tell you whether customers are actually feeling the delay.
Watch failed jobs and retries
Failed job counts, retry frequency, and dead-letter volume are some of the best early warning signals you can have. Spikes often reveal external API instability, bad deploys, or data-quality problems.
Handle poison messages
A poison message is a job that fails every time. Without dead-letter handling, poison messages can cause worker loops, retry storms, and waste. RabbitMQ, SQS, and many frameworks provide patterns for isolating these messages. (rabbitmq.com)
Capacity planning
Plan for:
peak traffic bursts,
long-running tasks,
retries during outages,
and the fact that some jobs consume far more resources than others.
A queue system is not just about average load; it is The safest designs assume spikes will happen and that some tasks will take much longer than expected.
8) Security and failure modes
Worker systems can fail in subtle ways, and some of those failures are security-adjacent.
Prevent duplicate side effects
If a task sends an email, charges a card, or writes to an external system, duplicates can be expensive. Use idempotency keys, deduplication tables, or unique constraints so repeated processing does not repeat the side effect.
Handle worker crashes safely
Crashes are normal. The system should assume a worker may die after fetching a job but before acking or committing the result. This is why at-least-once semantics, visibility timeouts, and late acknowledgments matter. (docs.aws.amazon.com)
Protect shared database resources
When the queue is also your primary database, worker traffic can interfere with user traffic. Heavy polling, large payloads, and long transactions can create hot spots and lock contention. PostgreSQL notes that SKIP LOCKED is useful for queue-like tables but is not a general-purpose consistency solution. (postgresql.org)
Avoid hot spots
Hot spots happen when many workers hammer the same rows, keys, partitions, or Redis structures. Common fixes include:
sharding queues by tenant or task type,
using multiple job tables or streams,
spreading scheduled jobs over time,
and keeping payloads small.
Secure the job payload
Do not store secrets in job payloads unless you absolutely must. Jobs are often inspected in logs, dashboards, and dead-letter tools. If sensitive data is required, store a reference and fetch the secret from a secure system at execution time.
9) Implementation examples and library choices
The “right” library depends on the stack you already have and the kind of jobs you run.
Redis-backed examples
A modern Redis-backed setup often uses Redis Streams when you want acknowledgment, consumer groups, and recovery semantics. Redis’s own documentation and learning materials emphasize consumer groups as the mechanism for distributing work across multiple consumers. (redis.io)
For many teams, Redis-based workers are a good fit for:
lightweight async jobs,
notification delivery,
scheduled tasks,
and medium-scale work queues.
Common frameworks
Celery is a long-standing choice in Python ecosystems, and its docs describe late acknowledgment and result handling clearly. (docs.celeryq.dev)
Kafka consumers are often used directly or through stream-processing frameworks when the workload is event-driven and replayable. Kafka’s consumer-group model is central to scaling. (kafka.apache.org)
RabbitMQ consumers are a good fit when routing, acknowledgement, and dead-letter behavior matter more than event replay. (rabbitmq.com)
SQS workers are ideal when you want managed infrastructure and simple operational overhead. (docs.aws.amazon.com)
Mapping use cases to the right tool
A practical way to think about it:
Small team, moderate load, already on Postgres → start with Postgres.
Fast, simple queue with good throughput and limited infrastructure → Redis.
Message routing, dead-lettering, and broker semantics → RabbitMQ.
High-volume event streaming and replay → Kafka.
Managed cloud queue with minimal ops → SQS.
The best choice is often the one that minimizes the number of new systems your team must learn and operate while still meeting reliability needs.
10) Decision guide and conclusion
Choosing between Postgres, Redis, or a dedicated queue is mostly about matching the tool to the workload.
Choose Postgres if:
your app already depends on Postgres heavily,
job throughput is moderate,
you want transactional enqueueing with business data,
and you prefer simplicity over specialized messaging features.
Choose Redis if:
you need a fast queue with low operational friction,
your jobs are short or medium duration,
you want consumer groups or delayed work,
and you can tolerate Redis being part queue, part cache, part infrastructure layer.
Choose RabbitMQ if:
you need broker-level messaging behavior,
dead-lettering and acknowledgments are important,
or your workflow depends on routing and message control.
Choose Kafka if:
your system is event-driven,
multiple consumers need the same event stream,
ordering and replay matter,
and your team is ready to run a streaming platform.
Choose SQS if:
you want a managed queue,
you care about reliability without running brokers,
and you are comfortable with AWS’s semantics and limits. (docs.aws.amazon.com)
Final takeaways
Background workers are not just a performance optimization. They are an architectural boundary that keeps user-facing systems responsive while allowing slower work to happen safely in the background. The core design challenge is reliability: atomic handoff, recovery after crashes, retry strategy, duplicate protection, and visibility into job state. Postgres is excellent when you want transactional simplicity. Redis is attractive when you want speed and a lightweight queueing layer. Dedicated brokers like RabbitMQ, Kafka, and SQS become compelling when scale, routing, or operational isolation matter more than convenience.
If you remember only one rule, make it this: choose the smallest system that can still give you the delivery guarantees, recovery behavior, and observability your application needs. That choice will save you time, money, and a lot of incident response later.