Chapter 19: Profiling & Performance Optimization

Objectives

• Measure execution time with timeit.
• Profile functions using cProfile and pstats.
• Inspect line-by-line timings with line_profiler.
• Analyze memory usage via memory_profiler.
• Apply simple code-level optimizations and built-in enhancements.

1. Measuring Time with timeit

Use the timeit module for micro-benchmarks:

import timeit

# one-liner
print(timeit.timeit("sum(range(1000))", number=10000))

# using Timer object
timer = timeit.Timer("x*x for x in range(1000)")
print(timer.timeit(number=5000))

2. Profiling with cProfile

Collect function-level statistics:

import cProfile, pstats

def work():
    total = 0
    for i in range(100000):
        total += i
    return total

# run profiler
prof = cProfile.Profile()
prof.enable()
work()
prof.disable()

# print sorted stats
stats = pstats.Stats(prof)
stats.sort_stats("cumtime").print_stats(10)

3. Line-by-Line Profiling

Install pip install line_profiler and decorate:

@profile
def compute():
    total = 0
    for i in range(100000):
        total += i*i
    return total

if __name__ == "__main__":
    compute()

Then run kernprof -l -v chapter19.py to see line timings.

4. Memory Profiling

Install pip install memory_profiler and use:

from memory_profiler import profile

@profile
def load_data():
    data = [i for i in range(1000000)]
    return data

if __name__ == "__main__":
    load_data()

Run with python -m memory_profiler chapter19.py.

5. Simple Optimization Tips

• Prefer built-ins like sum(), map() and comprehensions over manual loops.
• Cache expensive calls with functools.lru_cache.
• Use local variables inside loops for speed.
• Avoid global lookups: assign frequently used functions to locals.
• Consider numpy or C extensions for numeric heavy loops.

Exercises

1. Benchmark two implementations of Fibonacci using timeit.
2. Profile a data transformation script with cProfile and optimize the top hotspot.
3. Use line_profiler on a nested-loop function and speed up its hottest line.
4. Measure memory usage of building a large list vs generator expression with memory_profiler.