Measure instantiation + method execution time in Python

Question

I have a Python class and want to measure the time it takes to instantiate the class and execute a method across numerous, e.g., 100, runs.

I noticed that the first run takes considerably longer than consecutive runs. I assume that is caused by branch prediction since the input does not change. However, I want to measure the time it takes "from scratch", i.e., without the benefit of branch prediction. Note that constructing a realistic input is difficult in this case, thus the runs have to be executed on the same input.

To tackle this, I tried creating a new object on each run and delete the old object:

import time

class Myobject:

    def mymethod(self):
        """
        Does something complex.
        """
        pass

def benchmark(runs=100):
    """
    The argument runs corresponds to the number of times the benchmark is to be executed.
    """
    times_per_run = []
    r = range(runs)

    for _ in r:
        t2_start = time.perf_counter()

        # instantiation
        obj = Myobject()
        # method execution
        obj.mymethod()
        del obj

        t2_stop = time.perf_counter()

        times_per_run.append(t2_stop-t2_start)

    print(times_per_run)

benchmark(runs=10)

Executing this code shows that the average time per run varies significantly. The first run takes consistently longer. How do I eliminate the benefit of branch prediction when benchmarking across multiple runs?

Answer 1

To avoid the benefits of warmup (s. comments on post), I used the subprocess module to trigger the runs individually while measuring the time for each run and aggregate the results afterwards:

def benchmark(runs=100):
    times_per_run = []

    command = "python3 ./myclass.py"

    for _ in range(runs):
        t1_start = time.perf_counter()
        subprocess.run([command], capture_output=True, shell=True, check=False)
        t1_stop = time.perf_counter()

        times_per_run.append(t1_stop - t1_start)

    logging.info(f"Average time per run: {sum(times_per_run) / runs}")

benchmark()

This yields stable results.