How to choose grid search (when working with trainer.hyperparameter_search)?

Question

I want to run trainer.hyperparameter_search (with grid search) and I haven't seen any HP algorithm type parameter.

How can I configure trainer.hyperparameter_search to run with grid-search ?

Answer 1

You can use Optuna for this:

def hp_search(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 5e-5, 5e-6, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 3,10),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [1,2,4,6,8,16,32]),
    }

trainer.hyperparameter_search(direction="maximize", hp_space=hp_space)

This thread should also bring more light to the task at hand.