1---
2name: python-performance-optimization
3description: Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance.
4---
5 
6# Python Performance Optimization
7 
8Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.
9 
10## When to Use This Skill
11 
12- Identifying performance bottlenecks in Python applications
13- Reducing application latency and response times
14- Optimizing CPU-intensive operations
15- Reducing memory consumption and memory leaks
16- Improving database query performance
17- Optimizing I/O operations
18- Speeding up data processing pipelines
19- Implementing high-performance algorithms
20- Profiling production applications
21 
22## Core Concepts
23 
24### 1. Profiling Types
25 
26- **CPU Profiling**: Identify time-consuming functions
27- **Memory Profiling**: Track memory allocation and leaks
28- **Line Profiling**: Profile at line-by-line granularity
29- **Call Graph**: Visualize function call relationships
30 
31### 2. Performance Metrics
32 
33- **Execution Time**: How long operations take
34- **Memory Usage**: Peak and average memory consumption
35- **CPU Utilization**: Processor usage patterns
36- **I/O Wait**: Time spent on I/O operations
37 
38### 3. Optimization Strategies
39 
40- **Algorithmic**: Better algorithms and data structures
41- **Implementation**: More efficient code patterns
42- **Parallelization**: Multi-threading/processing
43- **Caching**: Avoid redundant computation
44- **Native Extensions**: C/Rust for critical paths
45 
46## Quick Start
47 
48### Basic Timing
49 
50```python
51import time
52 
53def measure_time():
54    """Simple timing measurement."""
55    start = time.time()
56 
57    # Your code here
58    result = sum(range(1000000))
59 
60    elapsed = time.time() - start
61    print(f"Execution time: {elapsed:.4f} seconds")
62    return result
63 
64# Better: use timeit for accurate measurements
65import timeit
66 
67execution_time = timeit.timeit(
68    "sum(range(1000000))",
69    number=100
70)
71print(f"Average time: {execution_time/100:.6f} seconds")
72```
73 
74## Profiling Tools
75 
76### Pattern 1: cProfile - CPU Profiling
77 
78```python
79import cProfile
80import pstats
81from pstats import SortKey
82 
83def slow_function():
84    """Function to profile."""
85    total = 0
86    for i in range(1000000):
87        total += i
88    return total
89 
90def another_function():
91    """Another function."""
92    return [i**2 for i in range(100000)]
93 
94def main():
95    """Main function to profile."""
96    result1 = slow_function()
97    result2 = another_function()
98    return result1, result2
99 
100# Profile the code
101if __name__ == "__main__":
102    profiler = cProfile.Profile()
103    profiler.enable()
104 
105    main()
106 
107    profiler.disable()
108 
109    # Print stats
110    stats = pstats.Stats(profiler)
111    stats.sort_stats(SortKey.CUMULATIVE)
112    stats.print_stats(10)  # Top 10 functions
113 
114    # Save to file for later analysis
115    stats.dump_stats("profile_output.prof")
116```
117 
118**Command-line profiling:**
119 
120```bash
121# Profile a script
122python -m cProfile -o output.prof script.py
123 
124# View results
125python -m pstats output.prof
126# In pstats:
127# sort cumtime
128# stats 10
129```
130 
131### Pattern 2: line_profiler - Line-by-Line Profiling
132 
133```python
134# Install: pip install line-profiler
135 
136# Add @profile decorator (line_profiler provides this)
137@profile
138def process_data(data):
139    """Process data with line profiling."""
140    result = []
141    for item in data:
142        processed = item * 2
143        result.append(processed)
144    return result
145 
146# Run with:
147# kernprof -l -v script.py
148```
149 
150**Manual line profiling:**
151 
152```python
153from line_profiler import LineProfiler
154 
155def process_data(data):
156    """Function to profile."""
157    result = []
158    for item in data:
159        processed = item * 2
160        result.append(processed)
161    return result
162 
163if __name__ == "__main__":
164    lp = LineProfiler()
165    lp.add_function(process_data)
166 
167    data = list(range(100000))
168 
169    lp_wrapper = lp(process_data)
170    lp_wrapper(data)
171 
172    lp.print_stats()
173```
174 
175### Pattern 3: memory_profiler - Memory Usage
176 
177```python
178# Install: pip install memory-profiler
179 
180from memory_profiler import profile
181 
182@profile
183def memory_intensive():
184    """Function that uses lots of memory."""
185    # Create large list
186    big_list = [i for i in range(1000000)]
187 
188    # Create large dict
189    big_dict = {i: i**2 for i in range(100000)}
190 
191    # Process data
192    result = sum(big_list)
193 
194    return result
195 
196if __name__ == "__main__":
197    memory_intensive()
198 
199# Run with:
200# python -m memory_profiler script.py
201```
202 
203### Pattern 4: py-spy - Production Profiling
204 
205```bash
206# Install: pip install py-spy
207 
208# Profile a running Python process
209py-spy top --pid 12345
210 
211# Generate flamegraph
212py-spy record -o profile.svg --pid 12345
213 
214# Profile a script
215py-spy record -o profile.svg -- python script.py
216 
217# Dump current call stack
218py-spy dump --pid 12345
219```
220 
221## Optimization Patterns
222 
223### Pattern 5: List Comprehensions vs Loops
224 
225```python
226import timeit
227 
228# Slow: Traditional loop
229def slow_squares(n):
230    """Create list of squares using loop."""
231    result = []
232    for i in range(n):
233        result.append(i**2)
234    return result
235 
236# Fast: List comprehension
237def fast_squares(n):
238    """Create list of squares using comprehension."""
239    return [i**2 for i in range(n)]
240 
241# Benchmark
242n = 100000
243 
244slow_time = timeit.timeit(lambda: slow_squares(n), number=100)
245fast_time = timeit.timeit(lambda: fast_squares(n), number=100)
246 
247print(f"Loop: {slow_time:.4f}s")
248print(f"Comprehension: {fast_time:.4f}s")
249print(f"Speedup: {slow_time/fast_time:.2f}x")
250 
251# Even faster for simple operations: map
252def faster_squares(n):
253    """Use map for even better performance."""
254    return list(map(lambda x: x**2, range(n)))
255```
256 
257### Pattern 6: Generator Expressions for Memory
258 
259```python
260import sys
261 
262def list_approach():
263    """Memory-intensive list."""
264    data = [i**2 for i in range(1000000)]
265    return sum(data)
266 
267def generator_approach():
268    """Memory-efficient generator."""
269    data = (i**2 for i in range(1000000))
270    return sum(data)
271 
272# Memory comparison
273list_data = [i for i in range(1000000)]
274gen_data = (i for i in range(1000000))
275 
276print(f"List size: {sys.getsizeof(list_data)} bytes")
277print(f"Generator size: {sys.getsizeof(gen_data)} bytes")
278 
279# Generators use constant memory regardless of size
280```
281 
282### Pattern 7: String Concatenation
283 
284```python
285import timeit
286 
287def slow_concat(items):
288    """Slow string concatenation."""
289    result = ""
290    for item in items:
291        result += str(item)
292    return result
293 
294def fast_concat(items):
295    """Fast string concatenation with join."""
296    return "".join(str(item) for item in items)
297 
298def faster_concat(items):
299    """Even faster with list."""
300    parts = [str(item) for item in items]
301    return "".join(parts)
302 
303items = list(range(10000))
304 
305# Benchmark
306slow = timeit.timeit(lambda: slow_concat(items), number=100)
307fast = timeit.timeit(lambda: fast_concat(items), number=100)
308faster = timeit.timeit(lambda: faster_concat(items), number=100)
309 
310print(f"Concatenation (+): {slow:.4f}s")
311print(f"Join (generator): {fast:.4f}s")
312print(f"Join (list): {faster:.4f}s")
313```
314 
315### Pattern 8: Dictionary Lookups vs List Searches
316 
317```python
318import timeit
319 
320# Create test data
321size = 10000
322items = list(range(size))
323lookup_dict = {i: i for i in range(size)}
324 
325def list_search(items, target):
326    """O(n) search in list."""
327    return target in items
328 
329def dict_search(lookup_dict, target):
330    """O(1) search in dict."""
331    return target in lookup_dict
332 
333target = size - 1  # Worst case for list
334 
335# Benchmark
336list_time = timeit.timeit(
337    lambda: list_search(items, target),
338    number=1000
339)
340dict_time = timeit.timeit(
341    lambda: dict_search(lookup_dict, target),
342    number=1000
343)
344 
345print(f"List search: {list_time:.6f}s")
346print(f"Dict search: {dict_time:.6f}s")
347print(f"Speedup: {list_time/dict_time:.0f}x")
348```
349 
350### Pattern 9: Local Variable Access
351 
352```python
353import timeit
354 
355# Global variable (slow)
356GLOBAL_VALUE = 100
357 
358def use_global():
359    """Access global variable."""
360    total = 0
361    for i in range(10000):
362        total += GLOBAL_VALUE
363    return total
364 
365def use_local():
366    """Use local variable."""
367    local_value = 100
368    total = 0
369    for i in range(10000):
370        total += local_value
371    return total
372 
373# Local is faster
374global_time = timeit.timeit(use_global, number=1000)
375local_time = timeit.timeit(use_local, number=1000)
376 
377print(f"Global access: {global_time:.4f}s")
378print(f"Local access: {local_time:.4f}s")
379print(f"Speedup: {global_time/local_time:.2f}x")
380```
381 
382### Pattern 10: Function Call Overhead
383 
384```python
385import timeit
386 
387def calculate_inline():
388    """Inline calculation."""
389    total = 0
390    for i in range(10000):
391        total += i * 2 + 1
392    return total
393 
394def helper_function(x):
395    """Helper function."""
396    return x * 2 + 1
397 
398def calculate_with_function():
399    """Calculation with function calls."""
400    total = 0
401    for i in range(10000):
402        total += helper_function(i)
403    return total
404 
405# Inline is faster due to no call overhead
406inline_time = timeit.timeit(calculate_inline, number=1000)
407function_time = timeit.timeit(calculate_with_function, number=1000)
408 
409print(f"Inline: {inline_time:.4f}s")
410print(f"Function calls: {function_time:.4f}s")
411```
412 
413For advanced optimization techniques including NumPy vectorization, caching, memory management, parallelization, async I/O, database optimization, and benchmarking tools, see [references/advanced-patterns.md](references/advanced-patterns.md)
414 
415## Best Practices
416 
4171. **Profile before optimizing** - Measure to find real bottlenecks
4182. **Focus on hot paths** - Optimize code that runs most frequently
4193. **Use appropriate data structures** - Dict for lookups, set for membership
4204. **Avoid premature optimization** - Clarity first, then optimize
4215. **Use built-in functions** - They're implemented in C
4226. **Cache expensive computations** - Use lru_cache
4237. **Batch I/O operations** - Reduce system calls
4248. **Use generators** for large datasets
4259. **Consider NumPy** for numerical operations
42610. **Profile production code** - Use py-spy for live systems
427 
428## Common Pitfalls
429 
430- Optimizing without profiling
431- Using global variables unnecessarily
432- Not using appropriate data structures
433- Creating unnecessary copies of data
434- Not using connection pooling for databases
435- Ignoring algorithmic complexity
436- Over-optimizing rare code paths
437- Not considering memory usage
438