Python Intermediate: How to Use Generators in Python

Python generators

Have you ever encountered a dataset that required more RAM than your computer could handle? Or perhaps you have a sophisticated function that must keep an internal state every time it is called, but the function is too tiny to warrant establishing its own class. Generators and the Python yield statement are useful in these and other situations.

When we talk about generators, we use two different terminologies.

1. Generator-function: It is defined in the same way as a regular function, but whenever it needs to generate a value, it uses the yield keyword rather than the return keyword. If yield appears in the body of a def, the function is automatically converted to a generator function.
2. Generator-Object: A generator object is returned by generator functions. Calling the following method on the generator object or utilizing the generator object in a “for in” loop are two ways to use generator objects (as shown in the above program).

Generator-function

Example:

def generator_function():
   yield 'A'          
   yield 'B'      
   yield 'C'      
   yield 'D'      
 
for item in generator_function():
   print(item)

Output:

A
B
C
D

Generator-Object

You’ve already learned about the two primary methods for building generators: generator functions and generator expressions. You might even understand how generators work intuitively. Let us take a moment to make that understanding plain.

Generator functions look and behave just like regular functions, with one exception. Instead of returning, generator functions employ the Python yield keyword.

Example:

def generator_function():
   yield 'A'          
   yield 'B'      
   yield 'C'      
   yield 'D'      
 
item = generator_function() # Make the function as a generator object
 
print(item.__next__())
print(item.__next__())
print(item.__next__())
print(item.__next__())

Output:

A
B
C
D

As a result, a generator function returns an iterable generator object, which can be used as an Iterator.

Here’s another example: Factors generator of number.

Example:

 
def generator_function(limit):
   for i in range(1, limit + 1):
       if limit % i == 0:
           yield i
 
 
factors = generator_function(50)
 
print(factors.__next__())
print(factors.__next__())
print(factors.__next__())
print(factors.__next__())
print(factors.__next__())
print(factors.__next__())
 
print("\nUsing for in loop:\n")
for i in generator_function(50):
   print(i)

Output:

1
2
5
10
25
50
 
Using for in loop:
 
1
2
5
10
25
50

Handling large data files, such as log files, is a more practical application of stream processing. Generators offer a space-efficient solution for such data processing since only portions of the file are processed at any given moment. We could also use Iterators for this, but Generator is more convenient (we don’t need to write next and iter functions here).