Streams and Backpressure
Design a small subsystem that relies on Streams and Backpressure. Outline components, data flow, failure modes, and metrics.
Answers use simple, clear English.
Quick interview answer
Use Streams and Backpressure as the core idea. Example shape: ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM..
Detailed answer
Use Streams and Backpressure as the core idea. Example shape: ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM.. Watch for: Error handling across piped streams needs explicit destroy/abort logic.. Measure success via latency/error/saturation. Core: Readable pushes data; Writable signals backpressure when internal buffer exceeds highWaterMark. Respect drain event before resuming writes to avoid memory blowup. Real-time example: ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM. Pros: Constant memory for large files; composable pipeline with pipe(). Cons: Error handling across piped streams needs explicit destroy/abort logic. Common mistakes: Ignoring write() return false; not awaiting 'drain' on backpressure. Best practices: Use pipeline() from stream/promises for automatic cleanup on error. Audience level: Mid-level.
Full explanation
Readable pushes data; Writable signals backpressure when internal buffer exceeds highWaterMark. Respect drain event before resuming writes to avoid memory blowup.
Real example & use case
ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM.
Pros & cons
Pros: Constant memory for large files; composable pipeline with pipe(). Cons: Error handling across piped streams needs explicit destroy/abort logic.
Common mistakes
Ignoring write() return false; not awaiting 'drain' on backpressure.
Best practices
Use pipeline() from stream/promises for automatic cleanup on error.
Follow-up questions
Only answered follow-ups are shown — click to open with full answers