Backend ServicesBy Abdul Rahman15 Jul 20263 min read
Message Queues and Event-Driven Architecture: SQS, RabbitMQ and Kafka
Moving slow or unreliable work out of the request path makes systems faster and more resilient. When to use queues, how SQS, RabbitMQ and Kafka differ, and how to process messages safely.
When a request triggers slow or unreliable work — sending emails, generating PDFs, calling third-party APIs, processing uploads — doing it inside the request makes the user wait and turns every downstream failure into an error on screen. Message queues decouple that work: the API records what needs to happen and returns immediately, and background workers do the rest.
The API responds immediately; workers process jobs at their own pace, with retries and a dead-letter queue.
When to Use a Queue
- Slow tasks — report generation, media processing, bulk imports.
- Unreliable dependencies — email, SMS, payment and AI providers that may time out or rate-limit.
- Traffic spikes — the queue absorbs bursts while workers process at a steady rate.
- Fan-out — one event ("student enrolled") triggering several independent reactions.
Queues vs Event Streams
Message queues (Amazon SQS, RabbitMQ) deliver each message to one consumer, which acknowledges and removes it. Ideal for jobs and tasks.
Event streams (Apache Kafka, Amazon Kinesis) keep an ordered, durable log that many consumers read independently, and can replay. Ideal for analytics pipelines, event sourcing and integrating many systems.
Choosing a Technology
- Amazon SQS — fully managed, virtually no operations, pay per use. A strong default on AWS.
- RabbitMQ — flexible routing (topics, fan-out, priorities) and mature tooling; you run it or use a managed service.
- Kafka — very high throughput and replayable history; operationally heavier, best when you genuinely need streams.
- Redis-based queues (such as BullMQ) — simple and fast for Node.js background jobs when you already run Redis.
Processing Messages Safely
Most queues guarantee at-least-once delivery: a message can arrive more than once. Workers must therefore be idempotent.
async function handle(message: { jobId: string; invoiceId: string }) {
// Skip if this job was already processed
const inserted = await db.processedJob.createIfNotExists(message.jobId);
if (!inserted) return;
await sendInvoiceEmail(message.invoiceId);
}
Other essentials:
- Retries with exponential backoff for transient failures.
- A dead-letter queue (DLQ) for messages that keep failing, with alerts so someone investigates.
- Visibility timeouts longer than the processing time, so a message isn't handed to a second worker mid-job.
- Small messages — send IDs and fetch data in the worker, rather than large payloads.
Designing Events
- Name events in the past tense:
student.enrolled,invoice.paid. - Include an event ID, timestamp, tenant ID and schema version.
- Keep events backward-compatible; consumers upgrade at different times.
- Use the outbox pattern — write the event to a database table in the same transaction as the business change, then publish it — so data changes and events never drift apart.
Key Takeaways
- Move slow and unreliable work out of the request path with queues.
- Use queues for jobs and streams for replayable, multi-consumer events.
- Make workers idempotent, retry with backoff and use dead-letter queues.
- Design versioned, past-tense events and use the outbox pattern for consistency.
