list vs generator memory
Compare list comprehensions vs generator expressions for processing a 10GB log file in Python.
Answers use simple, clear English.
Quick interview answer
List comps materialize all results in memory — can OOM on huge inputs. Generators yield lazily with near-constant memory. Trade-off: generators are one-shot (can't len()/reuse without regenerating).
Detailed answer
List comps materialize all results in memory — can OOM on huge inputs. Generators yield lazily with near-constant memory. Trade-off: generators are one-shot (can't len()/reuse without regenerating). For logs, stream line-by-line with a generator pipeline; materialize only small aggregates.
Real example & use case
(line for line in f if 'ERROR' in line) then count — never load full file as list.
Pros & cons
Pros: constant memory streaming. Cons: one-pass only unless reopened.
Code example
def error_lines(path: str):
with open(path) as f:
for line in f:
if 'ERROR' in line:
yield line.rstrip()
# count without loading all
print(sum(1 for _ in error_lines('app.log')))Practice code · python (view only · no execution)
def error_lines(path: str):
with open(path) as f:
for line in f:
if 'ERROR' in line:
yield line.rstrip()
# count without loading all
print(sum(1 for _ in error_lines('app.log')))