Building Resilience Against Third-Party Outages: Circuit Breakers & Exponential Backoff
Designing fault-tolerant distributed systems: Circuit Breaker states, exponential backoff with full jitter, graceful fallbacks, timeouts, and bulkheads.
Building Resilience Against Third-Party Outages: Circuit Breakers & Exponential Backoff
1. The Distributed Failure Cascading Problem
In modern architectures, backends integrate with multiple external dependencies: payment gateways, SMS providers, AI endpoints, and external APIs. When a third-party service slows down from 100ms to 30s:
- Web server threads hang waiting for external responses.
- Connection pools exhaust completely.
- Your entire application crashes, turning an external glitch into an internal total outage.
[ Client ] --> [ Your API ] --> [ Third-Party Gateway (Lagging 30s) ]
|
(All Worker Threads Blocked)
v
💥 TOTAL APPLICATION OUTAGE
2. Core Resilience Strategies
1. Circuit Breaker Pattern
+------------------+
| CLOSED | <--- Normal Operation (Requests Pass)
+------------------+
| ^
Failures exceed | Success in Half-Open
threshold |
v |
+------------------+
| OPEN | ---> Fast Fail (Fallback Triggered, No Outbound Call)
+------------------+
|
Timeout expires (Cooldown)
v
+------------------+
| HALF-OPEN | ---> Test with limited trial requests
+------------------+
2. Exponential Backoff with Full Jitter
Never retry on fixed intervals. Exponential backoff increases delay geometrically, while random jitter prevents thundering retry waves against a recovering service:
function calculateBackoff(attempt: number, baseMs = 200, maxMs = 10000): number {
const exponential = Math.min(maxMs, baseMs * Math.pow(2, attempt));
// Full Jitter
return Math.floor(Math.random() * exponential);
}
3. Bulkheading (Resource Isolation)
Assign separate thread pools or HTTP client connection limits for each external provider so that one slow service cannot starve the rest of the application.
3. NestJS / TypeScript Implementation Example
@Injectable()
export class ResilientPaymentClient {
private circuitBreaker = new CircuitBreaker(this.callExternalGateway.bind(this), {
timeout: 3000, // 3s timeout
errorThresholdPercentage: 50,
resetTimeout: 30000, // 30s cooldown before Half-Open
});
async processPayment(payload: PaymentPayload) {
this.circuitBreaker.fallback(() => this.handleFallback(payload));
return this.circuitBreaker.fire(payload);
}
private async callExternalGateway(payload: PaymentPayload) {
return axios.post('https://api.paymentprovider.com/v1/charge', payload, { timeout: 3000 });
}
private handleFallback(payload: PaymentPayload) {
// Return gracefully queued status
return { status: 'PENDING_ASYNC_RETRY', message: 'Payment queued for processing' };
}
}
4. Key Takeaways
- Always enforce strict timeouts on all external HTTP calls (never leave default unbounded timeouts).
- Fail fast with Circuit Breakers when external services are unhealthy.
- Use asynchronous outbox queues for non-blocking retry delivery.
Recommended Posts
Related Projects
A responsive e-commerce web storefront built with Next.js and React, featuring dynamic product catalog filtering, interactive cart state, and a streamlined multi-step checkout UI.