Amr Samir
Full-Stack Software Engineer
Building high-performance full-stack web applications, scalable architectures, and clean systems with Next.js & Node.js.
© 2026 Amr Samir. All rights reserved.
Next.js & React • TypeScript • Tailwind CSS
Amr Samir | Resilient Background Job Processing & Queues | Amr Samir
Architecting asynchronous job queues, streaming database queries, memory-bounded processing, failure retries, and job status polling/notifications.
By Amr Samir • August 19, 2026 • 2 min
Designing Robust Background Job Processing for Heavy Report Generation
Related Projects JobFlow Automation - Automated Freelance Scraper & WhatsApp Notification Engine
An automated job scraping daemon and WhatsApp event dispatcher built to monitor freelance job portals in real time and broadcast instant lead notifications to target developer groups.
Read more description Check Project
Related Certificates
1. Problem Statement & Interview Context A client clicks "Download Annual Financial Audit Report (500,000 transactions, PDF/CSV)" . If executed synchronously inside the HTTP request-response cycle:
The HTTP connection times out (e.g., Nginx 60s gateway timeout).
Server Node.js event loop blocks or exhausts memory loading half a million rows.
If the server restarts mid-generation, all progress is lost with no resumption mechanism.
[ Client ] -- HTTP POST /reports --> [ Web Server ] -- (Sync Generation) --> 💥 504 Gateway Timeout!
2. The Resilient Architecture Pattern +--------+ POST /reports +------------+ Enqueue Job +------------+
| Client | -----------------------> | API Server | ------------> | Redis / MQ |
+--------+ +------------+ +------------+
| | |
| Poll / WebSocket | Return 202 Accepted | Dequeue
| v v
| { jobId: "xyz" } +------------+
| | Worker Pool|
| +------------+
| |
| Stream to S3 | Chunk / Query
| <----------------------------------------------------------------+
| Presigned Download URL & Completed Status
3. Step-by-Step Implementation Strategy
Step 1: Immediate Job Acceptance (HTTP 202) The web server creates a pending record in MongoDB/PostgreSQL, pushes a message to BullMQ / Redis, and immediately returns a lightweight job descriptor:
@Post('generate-report')
@HttpCode(HttpStatus.ACCEPTED)
async createReportJob(@Body() dto: CreateReportDto, @Req() req) {
const jobRecord = await this.reportsService.createPending(dto, req.user.id);
await this.reportQueue.add('generate', {
reportId: jobRecord.id,
userId: req.user.id,
filters: dto.filters,
}, {
attempts: 3,
backoff: { type: 'exponential', delay: 5000 },
removeOnComplete: true,
});
return { reportId: jobRecord.id, status: 'queued' };
}
Step 2: Bounded Memory Querying (Streaming Cursors) Never run findMany() on 500,000 records. Stream data using database cursors or keyset batches to maintain a constant $O(1)$ memory footprint:
const cursor = db.transaction.findMany({
where: { accountId },
cursor: { id: lastSeenId },
take: 1000,
});
// Pipe stream chunks directly into CSV/PDF stream transformer
Step 3: Direct S3 Multipart Upload Stream output chunks directly to cloud object storage using multipart upload, avoiding storing multi-gigabyte files in worker RAM.
4. Failure Modes & Production Hardening
Worker Crash Mid-Job: Redis queue heartbeat detects unacknowledged jobs and re-queues them.Duplicate Execution: Jobs must use idempotent database write keys or file identifiers.Client Polling Fatigue: Support both WebSocket/SSE events and exponential backoff polling on /reports/:id/status.
5. Summary Checklist
Decouple long-running tasks from the HTTP thread.
Bound memory with cursor-based streaming.
Persist state machines with clear transitions: QUEUED -> PROCESSING -> COMPLETED | FAILED.
