Designing Robust Background Job Processing for Heavy Report Generation
1. Problem Statement & Interview Context
A client clicks "Download Annual Financial Audit Report (500,000 transactions, PDF/CSV)". If executed synchronously inside the HTTP request-response cycle:
- The HTTP connection times out (e.g., Nginx 60s gateway timeout).
- Server Node.js event loop blocks or exhausts memory loading half a million rows.
- If the server restarts mid-generation, all progress is lost with no resumption mechanism.
[ Client ] -- HTTP POST /reports --> [ Web Server ] -- (Sync Generation) --> 💥 504 Gateway Timeout!
2. The Resilient Architecture Pattern
+--------+ POST /reports +------------+ Enqueue Job +------------+
| Client | -----------------------> | API Server | ------------> | Redis / MQ |
+--------+ +------------+ +------------+
| | |
| Poll / WebSocket | Return 202 Accepted | Dequeue
| v v
| { jobId: "xyz" } +------------+
| | Worker Pool|
| +------------+
| |
| Stream to S3 | Chunk / Query
| <----------------------------------------------------------------+
| Presigned Download URL & Completed Status
3. Step-by-Step Implementation Strategy
Step 1: Immediate Job Acceptance (HTTP 202)
The web server creates a pending record in MongoDB/PostgreSQL, pushes a message to BullMQ / Redis, and immediately returns a lightweight job descriptor:
@Post('generate-report')
@HttpCode(HttpStatus.ACCEPTED)
async createReportJob(@Body() dto: CreateReportDto, @Req() req) {
const jobRecord = await this.reportsService.createPending(dto, req.user.id);
await this.reportQueue.add('generate', {
reportId: jobRecord.id,
userId: req.user.id,
filters: dto.filters,
}, {
attempts: 3,
backoff: { type: 'exponential', delay: 5000 },
removeOnComplete: true,
});
return { reportId: jobRecord.id, status: 'queued' };
}
Step 2: Bounded Memory Querying (Streaming Cursors)
Never run findMany() on 500,000 records. Stream data using database cursors or keyset batches to maintain a constant $O(1)$ memory footprint:
const cursor = db.transaction.findMany({
where: { accountId },
cursor: { id: lastSeenId },
take: 1000,
});
// Pipe stream chunks directly into CSV/PDF stream transformer
Step 3: Direct S3 Multipart Upload
Stream output chunks directly to cloud object storage using multipart upload, avoiding storing multi-gigabyte files in worker RAM.
4. Failure Modes & Production Hardening
- Worker Crash Mid-Job: Redis queue heartbeat detects unacknowledged jobs and re-queues them.
- Duplicate Execution: Jobs must use idempotent database write keys or file identifiers.
- Client Polling Fatigue: Support both WebSocket/SSE events and exponential backoff polling on
/reports/:id/status.
5. Summary Checklist
- Decouple long-running tasks from the HTTP thread.
- Bound memory with cursor-based streaming.
- Persist state machines with clear transitions:
QUEUED -> PROCESSING -> COMPLETED | FAILED.
