I ran the CP-SAT solver three consecutive times on my model. The wall times were:
There is almost 18x difference in runtime from the fastest to the slowest. There were no other CPU-intensive jobs on the machine at the time.
I understand there's some randomness in the solver, but I tried to remove that variation by seeding:
solver = cp_model.CpSolver()
solver.parameters.random_seed = 11111
Is this level of performance variation, given a fixed seed, reasonable?
(My goal is to get before-and-after performance measurements for some model changes I am considering. These were supposed to be the "before" numbers, but with this much variation, I will not have any confidence that the effects of my change will be visible.)
I'm using ortools-9.10.4067 on Python 3.10.12 on Ubuntu Desktop 22.04.4.
git clone --branch inconsistent-solver-performance https://[email protected]/scottj97/builderment.git
cd builderment
rm seeds/map_50resources.plants.json; make MAP=seeds/map_50resources
(Or download this exported model)
Solver elapsed wall time will appear in the log:
[2024-06-15 13:16:33.131512] Constraint solver completed after 2:17:42.137048.
Yes. this is expected. This is a hard problem to make it fast and deterministic.
You can make the solver deterministic by setting the parameter interleave_search:true. But the impact on performance can be large.