Search code examples
machine-learninghyperparameters

How to express hyperparameter distributions in Spearmint?


I'm trying to use Spearmint, the Bayesian optimization library, to tune hyperparameters for a machine learning classifier. My question is how does one express parameter search spaces that does not follow a uniform distribution?

From the project's github page, here is an example of how to set two uniformly distributed parameter search spaces:

"variables": {
    "X": {
        "type": "FLOAT",
        "size": 1,
        "min":  -5,
        "max":  10
    },
    "Y": {
        "type": "FLOAT",
        "size": 1,
        "min":  0,
        "max":  15
    }
}

How would we define a search space like the one below in Spearmint?

SVC_PARAMS = [
    {
        "bounds": {
            "max": 10.0,
            "min": 0.01,
        },
        "name": "C",
        "type": "double",
        "transformation": "log",
    },
    {
        "bounds": {
            "max": 1.0,
            "min": 0.0001,
        },
        "name": "gamma",
        "type": "double",
        "transformation": "log",
    },
    {
        "type": "categorical",
        "name": "kernel",
        "categorical_values": [
            {"name": "rbf"},
            {"name": "poly"},
            {"name": "sigmoid"},
        ],
    },
]

Is there a place to look up all of the stochastic expressions (ie uniform, normal, log etc) currently being supported by Spearmint?


Solution

  • Spearmint learns these kinds of transformations automatically from the data. If you take a look here: https://github.com/HIPS/Spearmint/tree/master/spearmint/transformations you can see the implementation of the beta warping that is applied (detailed in this paper: http://arxiv.org/abs/1402.0929). Spearmint doesn't have a way to specify these a-priori, but you could have Spearmint operate on e.g. the log of the parameters (by giving the log of the parameter ranges and exponentiating on your end).