Search code examples
optaplanner

BAVET Constraint streams seem to be non-reproducible

I have a working solution using Drools constraints streams with two kinds of tests:

  1. Constraint tests (both individual, as well as a combined one using ConstraintVerifier)
  2. Integration-style test which actually uses the solver to solve a set of golden problems (solutions are known and asserted on).

I'm using a step-based termination to have a fully reproducible setup (random seed=0, local-search=SA). I can see that it's reproducible, because multiple runs of those tests produce the same results in the same number of steps.

In the interested of performance optimization (and subsequently solution optimization) I tried to switch constraint streams from Drools (default?!) to Bavet. The result for my two kinds of tests is as follows:

  1. Still green, significant speed increase of ~20x.
  2. Some tests are failing because they no longer find the optimal solution (but they are way faster). What's especially interesting, is that the non-optimal solution that they find differs from one test run to the next.

Is there something inherit in the Bavet implementation that is non-reproducible, e.g. does it use its own Random instance or does it rely on System.identityHashCode() or something like that? I'd very much like to have a faster score calculation, but I favor reproducibility over speed.

EDIT

I forgot to mention that in order to make my constraints work with Bavet I had to rewrite them using groupBy a lot (instead of map which was failing with UnsupportedOperationException). Could it be a combination of that, i.e. groupBy + Bavet?

Solution

  • There is a lot to unpack here. First of all, Bavet is experimental and not (yet) feature complete. That is why you are seeing those exceptions, and it should probably be the answer to the reproducibility question too. Drools is our default CS implementation, and there are good reasons for it.

    With that out of the way, let's look deeper. The problem you are describing looks a lot like score corruption. Please see environment mode documentation for how to figure out if that is the case. If the default implementation shows score corruptions, most likely your constraint provider needs more work. If the Bavet implementation shows score corruptions, it can either be a problem in your constraint provider, or in Bavet itself - after all, it is still experimental.

    As a final point, I would not compare CS-D and CS-B so directly. If your constraint providers are not identical, you should not expect the two CS implementations to produce the same results, unless you have proven step by step that your constraint variants actually do result in the same constraint matches. If you intend to do that, Constraint Verifier will help you.

    As a side note, if map() was where you saw the UnsupportedOperationException, that has only been implemented in 8.27.0.Final, out soon. Please upgrade.