How to implement randomised log space search of learning rate in PyTorch?

Question

I am looking to fine tune a GNN and my supervisor suggested exploring different learning rates. I came across this tutorial video where he mentions that a randomised log space search of hyper parameters is typically done in practice. For sake of the introductory tutorial this was not covered.

Any help or pointers on how to achieve this in PyTorch is greatly appreciated. Thank you!

Answer 1

Setting the scale in logarithm terms let you take into account more desirable values of the learning rate, usually values lower than 0.1

Imagine you want to take learning rate values between 0.1 (1e-1) and 0.001 (1e-4). Then you can set this lower and upper bound on a logarithm scale by applying a logarithm base 10 on it, log10(0.1) = -1 and log10(0.001) = -4. Andrew Ng provides a clearer explanation in this video.

In Python you can use np.random.uniform() for this

searchable_learning_rates = [10**(-4 * np.random.uniform(0.5, 1)) for _ in range(10)]
searchable_learning_rates
>>>
[0.004890650359810075,
 0.007894672127828331,
 0.008698831627963768,
 0.00022779163472045743,
 0.0012046829055603172,
 0.00071395500159473,
 0.005690032483124896,
 0.000343368839731761,
 0.0002819402550629178,
 0.0006399571804618883]

as you can see you're able to try learning rate values from 0.0002819402550629178 up to 0.008698831627963768 which is close to the upper bound. The longer the array the more values you will try.

Following the example code in the video you provided you can implement the randomized log search for the learning rate by replacing learning_rates for searchable learning_rates

for batch_size in batch_sizes:
    for learning_rate in searchable_learning_rates:
    ...
    ...