Improvements of Random Search for Hyperparameter Optimization

Question

Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. Additional to the continuous Cost and gamma parameter, I have one discrete parameter and also an equality constraint over some parameters.

Now, I would like to develop random search further, e.g. through adaptive random search. That means for example adaptation of the search direction or of the search range.

Does somebody have an idea how this can be done or could reference to some existing work on this? Other ideas for improving random search are also welcome.

Answer 1

Why you try to reinvent the wheel? Hyperparameters optimization is well studied topic, with at least few of the state of the art method, which simply solve the problem for SVMs, including:

• Bayesian optimization (usually through modeling model quality with Gaussian processes), see for example bayesopt http://rmcantin.bitbucket.org/html/
• Tree of parzen estimators (sometimes better for discrete, complex hyperparameters spaces) included (in particular) in hyperopt http://hyperopt.github.io/hyperopt/