Search code examples
pysparkgenetic-algorithmdeap

Using DEAP (genetic algorithm library) with spark

Is IT possible to use DEAP ( http://deap.readthedocs.io/en/master/) with a spark cluster to map the fitness evaluation function. I would like to run a GA but the fitness function is rather long and I was planning on distributing it on a spark cluster.

Solution

  • You should look at the Using Multiple Processors section in the DEAP documentation and at this example. They explain how to replace the map function in the DEAP toolbox by a map function of your choice.

    To use pyspark to map the fitness evaluation function, you could do something like that:

    from pyspark import SparkContext

sc = SparkContext(appName="DEAP")

def sparkMap(algorithm, population):
    return sc.parallelize(population).map(algorithm)

toolbox.register("map", sparkMap)