How do I know if my code is running fast enough? Is there a measurable way to test the speed & performance of my code?
For example, I have script that is reading CSV files and writing new CSV files while using Numpy to calculate statistics. Below, I'm using cProfiler for my Python script but after seeing resulting stats, what do I do next? In this case, I can see that the methods mean, astype, reduce from numpy, method writerow from csv and method append of python lists is taking a significant portion of the time.
How can I know if my code can improve or not?
python -m cProfile -s cumulative OBSparser.py
176657699 function calls (176651606 primitive calls) in 528.419 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.003 0.003 528.421 528.421 OBSparser.py:1(<module>)
1 0.000 0.000 526.874 526.874 OBSparser.py:45(start)
1 165.767 165.767 526.874 526.874 OBSparser.py:48(parse)
7638018 6.895 0.000 179.890 0.000 {method 'mean' of 'numpy.ndarray' objects}
7638018 56.780 0.000 172.995 0.000 _methods.py:53(_mean)
7628171 57.232 0.000 57.232 0.000 {method 'writerow' of '_csv.writer' objects}
7700878 52.580 0.000 52.580 0.000 {method 'reduce' of 'numpy.ufunc' objects}
7615219 50.640 0.000 50.640 0.000 {method 'astype' of 'numpy.ndarray' objects}
7668436 28.595 0.000 36.853 0.000 _methods.py:43(_count_reduce_items)
15323753 31.503 0.000 31.503 0.000 {numpy.core.multiarray.array}
45751805 13.439 0.000 13.439 0.000 {method 'append' of 'list' objects}
Can somebody explain the best practices?