Chapter 26: Advanced Python Programming – Iterators and Generators

Abstract:

In Python, iterators and generators are both used for creating sequences of values, but they differ in their implementation and how they generate these values. Iterators are objects that provide sequential access to elements of an iterable, while generators are functions that yield values one at a time. 

Iterators:

• Definition:

  An iterator is an object that implements the __iter__ and __next__ methods. 
• Creation:

  You can create iterators from iterables (like lists, tuples, strings) using the iter() function. 
• State:

  Iterators maintain internal state, allowing them to track their position within the sequence. 
• Implementation:

  Iterators can be implemented as custom classes that define __iter__ and __next__ methods. 

Generators:

• Definition: Generators are special functions that use the yield keyword to produce values. 
• Creation: Generators are created by calling a generator function. 
• Lazy Evaluation: Generators don't store all values in memory; they generate values on demand when requested. 
• Implementation: Generators are implemented using a function containing yield statements. 
• Advantages: Generators are often preferred for their memory efficiency when dealing with large datasets or infinite sequences. 

Key Differences:

Feature	Iterator	Generator
Implementation	Class with __iter__ and __next__	Function with yield
State	Maintains state for sequential access	Pauses and resumes execution
Memory	Can store elements in memory	Generates elements on demand
Use Cases	When needing complex state management	When dealing with large datasets or infinite sequences

26.1 Introduction

Python offers powerful tools for handling iteration in an elegant and memory-efficient manner. Two of the most crucial concepts in this domain are Iterators and Generators. These tools allow developers to process large sequences of data without loading everything into memory at once. This chapter explores iterators and generators, how they work, their advantages, and how they can be used to write more Pythonic and efficient code.

---

26.2 Iterators in Python

26.2.1 What is an Iterator?

An iterator is an object that enables traversal through all the elements of a collection, such as a list or tuple, without needing to use indexing.

26.2.2 The Iterator Protocol

In Python, an object is an iterator if it implements two methods:

• __iter__() returns the iterator object itself.

• __next__() returns the next item in the sequence. When there are no more items, it raises the StopIteration exception.

Example: Custom Iterator

class CountDown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        else:
            num = self.current
            self.current -= 1
            return num

cd = CountDown(5)
for num in cd:
    print(num)

Output:

5
4
3
2
1

---

26.3 Generators in Python

26.3.1 What is a Generator?

A generator is a special type of iterator. It simplifies the process of writing iterators. Instead of using classes and maintaining state manually, Python allows defining an iterator in the form of a function using the yield keyword.

26.3.2 Generator Function

When a function contains a yield statement, it becomes a generator function. Each time yield is called, the function's state is preserved, allowing resumption later.

Example: Simple Generator

def count_up_to(max):
    count = 1
    while count <= max:
        yield count
        count += 1

for num in count_up_to(5):
    print(num)

Output:

1
2
3
4
5

---

26.4 Differences Between Iterators and Generators

Feature	Iterator	Generator
Implementation	Requires a class with __iter__ and __next__	Implemented using a function with yield
Syntax	More complex	Simpler and more concise
Memory Efficiency	Can be efficient	Highly efficient
State Management	Manually managed	Automatically managed

---

26.5 Generator Expressions

Just like list comprehensions, Python offers generator expressions for creating generators.

Example: Generator Expression

squares = (x * x for x in range(5))
for s in squares:
    print(s)

Output:

0
1
4
9
16

Generator expressions are enclosed in parentheses and are more memory-efficient than list comprehensions, especially for large datasets.

---

26.6 Use Cases of Iterators and Generators

• Reading large files line-by-line without loading the entire file into memory.

• Streaming data from a sensor or live feed.

• Infinite sequences (e.g., Fibonacci series, prime numbers).

• Lazy evaluation for computational efficiency.

---

26.7 Chaining Iterators and Composing Generators

Using built-in functions like itertools.chain() or writing custom generators, one can compose generators to build complex data processing pipelines.

Example: Generator Composition

def even_numbers(nums):
    for n in nums:
        if n % 2 == 0:
            yield n

def square(nums):
    for n in nums:
        yield n * n

numbers = range(10)
evens = even_numbers(numbers)
squared_evens = square(evens)

for num in squared_evens:
    print(num)

Output:

0
4
16
36
64

---

26.8 Built-in Generator Tools: itertools Module

The itertools module provides a collection of fast, memory-efficient tools for working with iterators.

Common Functions:

• count(start, step) – Infinite counter.

• cycle(iterable) – Repeats the iterable endlessly.

• repeat(elem, n) – Repeats an element n times.

• chain(iter1, iter2, ...) – Combines multiple iterables.

• islice(iter, start, stop, step) – Slice an iterator.

---

26.9 Advantages and Limitations

Advantages:

• Memory Efficient – Do not store entire sequences in memory.

• Cleaner Syntax – Especially with generator functions.

• Composable – Useful in building data pipelines.

Limitations:

• Single-use – Generators and iterators can’t be reused without re-instantiating.

• Lack of Random Access – Unlike lists or arrays, iterators do not support indexing.

---

26.10 Summary

Iterators and generators are key components of Python that enhance both performance and readability when dealing with sequences and data processing tasks. By mastering these tools, developers can write code that is both more efficient and Pythonic.

---

26.11 Exercises

Short Answer:

1. What are the two methods that define an iterator in Python?

2. What is the main difference between a generator and a list?

3. Why is StopIteration important?

Programming Exercises:

1. Write a generator function that yields the Fibonacci sequence up to n terms.

2. Create a custom iterator that returns only the prime numbers below n.

3. Use a generator expression to yield the cube of all odd numbers in a given list.

---

26.12 Sample Code Snippet for Practice

# Generator for Fibonacci sequence
def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

for num in fibonacci(10):
    print(num)

---

26.13 Conclusion

Understanding and effectively using iterators and generators allows Python programmers to handle large datasets, implement lazy evaluation, and build clean, efficient, and scalable code. They form the backbone of many Python features and libraries, making their mastery essential for advanced programming.