Search code examples
pythonbenchmarkingmicrobenchmark

How to benchmark a single function call in Python?

How to interactively (micro)benchmark a code expression in Python (i.e. the equivalent of @btime/@benchmark in Julia) ?

I would like to benchmark interactively an expression (a function call), where the number of times this expression should be evaluated depends on its computational cost / variance....

I have tried timeit.timeit but I have some quesitons:

  1. How do I interpret the time output of the following results:
>>> a = 2.2
>>> timeit.repeat("round(a)", setup='from __main__ import a; gc.enable()', repeat=5,number=10)
[4.631001502275467e-06, 1.3809185475111008e-06, 1.2170057743787766e-06, 1.1718366295099258e-06, 1.1730007827281952e-06]
>>> timeit.timeit("round(a)", setup='from __main__ import a; gc.enable()', number=10)
5.741836503148079e-06
>>> timeit.timeit("round(a)", setup='from __main__ import a; gc.enable()')
0.11461802502162755
>>> timeit.Timer('round(a)', setup='from __main__ import a; gc.enable()').autorange()
(5000000, 0.4272152939811349)

It seems that when I go from number from 1 to 10, the timing remains pretty stable, but then it grows of different order of magnitude....

  1. Does it makes sense a workflow where I first create the Timer object once, and then I modify several times the code to evaluate and benchmark the modified Timer ? E.g.
import timeit
import gc

bclass = timeit.Timer('round(1.1)','gc.enable()', globals=globals()) 


[...]


bclass.stmt = "small_3x2_nash.vertex_enumeration()"
bclass.autorange()

bclass.stmt = "small_3x2_nash.lemke_howson_enumeration()"
bclass.autorange()

bclass.stmt = "small_3x2_nash.support_enumeration()"
bclass.autorange()

(seems this is not possible, as I have tried very different code and the result of the timing remains almost the same)

Solution

  • You can also use the Timer class to enable de GC, as explained in the docs here: https://docs.python.org/3/library/timeit.html#timeit.Timer.timeit

    And instead of explicit import every variable/function that you want to use from main, you could pass globals() in the Timer arguments. In that way, all the globals variables, functions, etc will be passed to de evaluated function.

    Python 3.10.10 (main, Mar  5 2023, 22:26:53) [GCC 12.2.1 20230201] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> 
>>> a = 2.2
>>> 
>>> t = timeit.Timer('round(a)', globals=globals())
>>> 
>>> t.timeit(number=1000)
0.00042562399994494626
>>> t.autorange()
(5000000, 0.46861540100007915)

    To enable GC you need also import the gclib

    import gc
t = timeit.Timer('round(a)', 'gc.enable()', globals=globals())