Designing Video Upload & Adaptive Bitrate Streaming Pipelines (HLS/DASH)
System design for large video ingestion: chunked S3 uploads, async worker transcoding clusters with FFmpeg, HLS manifest generation, and CDN edge delivery.
Designing Video Upload & Adaptive Bitrate Streaming Pipelines (HLS/DASH)
1. System Scale & Video Architecture Challenges
Video files are massive (hundreds of megabytes to gigabytes). Serving raw MP4 files directly from web servers causes buffering delays, high bandwidth costs, and poor user experience on mobile networks. A scalable video platform requires:
- Direct chunked uploads to object storage.
- Asynchronous transcoding pipelines generating multiple resolutions (1080p, 720p, 480p, 360p).
- Chunked adaptive bitrate streaming via HTTP Live Streaming (HLS).
- CDN edge caching.
+----------------------------------------------------------------------+
| End-to-End Video Pipeline |
| |
| [ Client ] ---> [ Direct S3 Upload ] ---> [ Transcoding Cluster ] |
| | |
| v |
| [ Player ] <--- [ CDN Edge (HLS) ] <--- [ HLS Manifests (.m3u8) ] |
+----------------------------------------------------------------------+
2. Adaptive Bitrate Streaming (HLS) Architecture
HLS splits a video into short 2–6 second segments (.ts / .m4s) and generates a master playlist (master.m3u8):
#EXTM3U (Master Manifest)
#EXT-X-STREAM-INF:BANDWIDTH=3000000,RESOLUTION=1920x1080
1080p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1500000,RESOLUTION=1280x720
720p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=854x480
480p/index.m3u8
The client video player dynamically shifts resolutions based on the viewer's current real-time bandwidth.
3. Worker Transcoding Pipeline with FFmpeg
export async function transcodeToHLS(inputPath: string, outputDir: string) {
// Transcode to 1080p, 720p, 480p with HLS segmentation
const ffmpegCommand = `
ffmpeg -i ${inputPath} \
-filter_complex "[0:v]split=3[v1][v2][v3]; \
[v1]scale=w=1920:h=1080[v1out]; \
[v2]scale=w=1280:h=720[v2out]; \
[v3]scale=w=854:h=480[v3out]" \
-map "[v1out]" -c:v:0 libx264 -b:v:0 3000k \
-map "[v2out]" -c:v:1 libx264 -b:v:1 1500k \
-map "[v3out]" -c:v:2 libx264 -b:v:2 800k \
-f hls -hls_time 4 -hls_playlist_type vod \
-master_pl_name master.m3u8 \
${outputDir}/%v/index.m3u8
`;
await execAsync(ffmpegCommand);
}
4. Key Takeaways
- Never transcode videos synchronously on web API servers.
- Use HLS/DASH chunking for smooth playback across varying bandwidth conditions.
- Cache video segment chunks aggressively at CDN edge caches.
Recommended Posts
Related Projects
A comprehensive learning management portal built with Express, React, and MongoDB, featuring signed Cloudinary video streaming URLs, nested curriculum models, student progress tracking, and PayPal course enrollments.