Skip to main content

09.1 - Iterators & Generators

Theory 25 min Advanced

The Iterator Protocol​

Any object is iterable if it implements __iter__() and __next__():

# Manual iterator implementation
class CountUp:
    def __init__(self, start, stop):
        self.current = start
        self.stop = stop

    def __iter__(self):
        return self   # self is both iterable and iterator

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration   # signals end of iteration
        value = self.current
        self.current += 1
        return value

for n in CountUp(1, 5):
    print(n)   # 1, 2, 3, 4

Generator Functions​

A generator function uses yield to produce values lazily — one at a time:

def count_up(start, stop):
    """Generator version — simpler and more memory-efficient."""
    current = start
    while current < stop:
        yield current
        current += 1

for n in count_up(1, 5):
    print(n)   # 1, 2, 3, 4

# Create a generator object
gen = count_up(1, 1_000_000)
next(gen)    # 1
next(gen)    # 2
# Only computes next value on demand!

How Generators Work​

Practical Generator Patterns​

# Fibonacci generator — infinite sequence
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
[next(fib) for _ in range(10)]
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

# Read large file line by line
def read_chunks(filepath, chunk_size=1024):
    with open(filepath, "rb") as f:
        while chunk := f.read(chunk_size):
            yield chunk

# Pipeline: generate → filter → transform
def process_logs(filepath):
    lines = (line.strip() for line in open(filepath))
    errors = (line for line in lines if "ERROR" in line)
    messages = (line.split("] ", 1)[-1] for line in errors)
    yield from messages

# yield from — delegate to sub-generator
def chain(*iterables):
    for it in iterables:
        yield from it

list(chain([1,2], [3,4], [5,6]))   # [1,2,3,4,5,6]

itertools — Generator Toolkit​

import itertools

# Infinite iterators
itertools.count(10, 2)         # 10, 12, 14, 16, ...
itertools.cycle("ABC")         # A, B, C, A, B, C, ...
itertools.repeat("x", 3)       # "x", "x", "x"

# Combinatorics
list(itertools.combinations([1,2,3], 2))
# [(1,2), (1,3), (2,3)]

list(itertools.permutations([1,2,3], 2))
# [(1,2), (1,3), (2,1), (2,3), (3,1), (3,2)]

list(itertools.product([0,1], repeat=2))
# [(0,0), (0,1), (1,0), (1,1)]

# Chaining
list(itertools.chain([1,2], [3,4]))   # [1,2,3,4]

# Grouping
data = [("A",1),("A",2),("B",3),("B",4)]
for key, group in itertools.groupby(data, key=lambda x: x[0]):
    print(key, list(group))

# Slicing
list(itertools.islice(fibonacci(), 10))   # first 10 Fibonacci

# Batches (Python 3.12+)
list(itertools.batched([1,2,3,4,5], 2))  # [(1,2),(3,4),(5,)]

Generator vs List — Memory Comparison​

import sys

# List — stores all values in memory
squares_list = [x**2 for x in range(1_000_000)]
sys.getsizeof(squares_list)   # ~8 MB

# Generator — computes on demand
squares_gen = (x**2 for x in range(1_000_000))
sys.getsizeof(squares_gen)    # ~200 bytes!

# Sum: both give same result, generator uses constant memory
sum(x**2 for x in range(1_000_000))   # 333333833333500000

Key Vocabulary​

TermDefinition
IteratorObject with __iter__ and __next__
Generator functionFunction with yield — returns a generator object
yieldPause execution and return a value; resume on next()
yield fromDelegate to a sub-generator
Lazy evaluationComputing values only when needed
StopIterationException signaling end of iteration
itertoolsStandard library module of iterator tools

Summary​

  • Generators produce values lazily with yield — no memory overhead for large sequences
  • Generator functions are simpler than manual iterator classes
  • Use yield from to delegate to sub-generators
  • itertools provides powerful combinations, chains, grouping, and batching
  • Use generators for data pipelines, large file processing, and infinite sequences