Streams and Backpressure
Give a real production-style example of using Streams and Backpressure. Walk through the scenario end-to-end.
Answers use simple, clear English.
Quick interview answer
Scenario: ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM. Implementation notes: Use pipeline() from stream/promises for automatic cleanup on error.
Detailed answer
Scenario: ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM. Implementation notes: Use pipeline() from stream/promises for automatic cleanup on error. Core: Readable pushes data; Writable signals backpressure when internal buffer exceeds highWaterMark. Respect drain event before resuming writes to avoid memory blowup. Real-time example: ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM. Pros: Constant memory for large files; composable pipeline with pipe(). Cons: Error handling across piped streams needs explicit destroy/abort logic. Common mistakes: Ignoring write() return false; not awaiting 'drain' on backpressure. Best practices: Use pipeline() from stream/promises for automatic cleanup on error. Audience level: Junior.
Full explanation
Readable pushes data; Writable signals backpressure when internal buffer exceeds highWaterMark. Respect drain event before resuming writes to avoid memory blowup.
Real example & use case
ETL job pipes HTTP download → gzip transform → S3 upload without loading entire file into RAM.
Pros & cons
Pros: Constant memory for large files; composable pipeline with pipe(). Cons: Error handling across piped streams needs explicit destroy/abort logic.
Common mistakes
Ignoring write() return false; not awaiting 'drain' on backpressure.
Best practices
Use pipeline() from stream/promises for automatic cleanup on error.
Follow-up questions
Open one as its own read / solve / listen card