Generators, iterators, and the iterator protocol

What actually makes a for loop work on a list, a file, and a generator alike — and how yield turns an ordinary function into a resumable one.

Intermediate

3 min read

The protocol underneath every for loop

for x in something: works on lists, strings, dicts, files, and generators alike because Python doesn't special-case any of them — it asks something for an iterator, then repeatedly asks that iterator for the next value, until it signals there are none left:

it = iter([10, 20, 30])   # __iter__() — get an iterator
next(it)                   # 10 — __next__()
next(it)                   # 20
next(it)                   # 30
next(it)                   # raises StopIteration — the loop ends here

An iterable is anything with __iter__() (a list, a string, a custom class). An iterator is anything with __next__() (and its own __iter__() that returns itself). A for loop is exactly the code above, done automatically: call iter() once, call next() repeatedly, stop when StopIteration is raised.

Writing your own — the hard way

class CountUpTo:
    def __init__(self, limit):
        self.limit = limit
        self.n = 0
 
    def __iter__(self):
        return self
 
    def __next__(self):
        if self.n >= self.limit:
            raise StopIteration
        self.n += 1
        return self.n
 
for x in CountUpTo(3):
    print(x)   # 1, 2, 3

This works, but it's a lot of ceremony — manually tracking state (self.n) across calls, manually raising StopIteration — for something conceptually simple: "count up to a limit."

`yield` — the same thing, without the ceremony

def count_up_to(limit):
    n = 0
    while n < limit:
        n += 1
        yield n
 
for x in count_up_to(3):
    print(x)   # 1, 2, 3

Calling count_up_to(3) doesn't run the function body at all — it returns a generator object immediately. Each call to next() on it runs the function until the next yield, returns that value, and then pauses — with every local variable (n, in this case) frozen exactly where it was. The next next() call resumes right after that yield, not from the top. This is what a generator function actually is: ordinary function syntax that the interpreter turns into an object implementing the iterator protocol automatically, so you never write __iter__/__next__/StopIteration by hand.

Why this is the memory-efficient choice

def all_squares_list(n):          # builds the entire list in memory before returning
    return [i * i for i in range(n)]
 
def all_squares_gen(n):           # produces one value at a time, on demand
    for i in range(n):
        yield i * i

all_squares_list(10_000_000) allocates memory for ten million integers before you use a single one of them. all_squares_gen(10_000_000) holds only the current value and the loop's position — a handful of bytes, regardless of how large n is. If the caller is going to consume the sequence once, in order (a for loop, sum(), any()), a generator does the identical job for a fraction of the memory — this is the mechanism behind the list-vs-generator trade-off covered in the list-mechanics lesson.

The one real limitation

gen = count_up_to(3)
list(gen)   # [1, 2, 3]
list(gen)   # [] — already exhausted, nothing left to produce

A generator can only be iterated once. Once it's exhausted (or you stop pulling from it partway through), there's no way to "rewind" it — unlike a list, which you can iterate as many times as you want. If you need to iterate the same sequence multiple times, that's a real list, not a generator.