I'm trying to use Spearmint, the Bayesian optimization library, to tune hyperparameters for a machine learning classifier. My question is how does one express parameter search spaces that does not follow a uniform distribution?
From the project's github page, here is an example of how to set two uniformly distributed parameter search spaces:
"variables": {
"X": {
"type": "FLOAT",
"size": 1,
"min": -5,
"max": 10
},
"Y": {
"type": "FLOAT",
"size": 1,
"min": 0,
"max": 15
}
}
How would we define a search space like the one below in Spearmint?
SVC_PARAMS = [
{
"bounds": {
"max": 10.0,
"min": 0.01,
},
"name": "C",
"type": "double",
"transformation": "log",
},
{
"bounds": {
"max": 1.0,
"min": 0.0001,
},
"name": "gamma",
"type": "double",
"transformation": "log",
},
{
"type": "categorical",
"name": "kernel",
"categorical_values": [
{"name": "rbf"},
{"name": "poly"},
{"name": "sigmoid"},
],
},
]
Is there a place to look up all of the stochastic expressions (ie uniform
, normal
, log
etc) currently being supported by Spearmint?
Spearmint learns these kinds of transformations automatically from the data. If you take a look here: https://github.com/HIPS/Spearmint/tree/master/spearmint/transformations you can see the implementation of the beta warping that is applied (detailed in this paper: http://arxiv.org/abs/1402.0929). Spearmint doesn't have a way to specify these a-priori, but you could have Spearmint operate on e.g. the log of the parameters (by giving the log of the parameter ranges and exponentiating on your end).