Search code examples
python-3.xhyperparametersgoogle-cloud-vertex-aispacy-3

Hyperparameter data types and scales not being validated

On past week, I was implementing some code to tune hyperparameters on a spaCy model, using Vertex AI. From that experience, I have several questions, but since they might no be directly related to each other, I decided to open one case per each question.

In this case, I would like to understand what is exactly going on, when I set the following hyperparameters, in some HP tuning job:

hyperparameters

Notice both examples have been purposedly written 'wrongly' to trigger an error but 'eerily', they don't (UPDATE: at least with my current understanding of the docs). I have the sensation that "Vertex AI does not make any validation of the inserted values, they just run whatever you write, and trigger an error only if the values don't actually make ANY sense". Allow me to insert a couple of comments on each example:

  • dropout: This variable should be "scaled linearly between 0 and 1" ... However what I can see in the HP tuning jobs, are values "scaled linearly between 0.1 and 0.3, and nothing in the interval 0.3 to 0.5". Now this reasoning is a bit naive, as I am not 100% sure if this algorithm had to do in the values selection, or "Google Console understood I only had the interval [0.1,0.3] to choose values from". (UPDATE) Plus, how can a variable be "discrete and linear" at the same time?
  • batch_size: I think I know what's going on with this one, I just want to confirm: 3 categorical values ("500", "1000" & "2000") are being selected "as they are", since they have a SHP of "UNESPECIFIED".

(*) Notice both the HP names, as well as their values, were just "examples on the spot", they don't intend to be "good starting points". HP tuning initial values selection is NOT the point of this query.

Thank you.

Solution

  • If the type is Categorical, then the scale type is irrelevant and ignored. If the type is DoubleValueSpec, IntegerValueSpec, or DiscreteValueSpec, then the scale type will govern which values are picked more often.

    Regarding how a variable can be both Discrete and have a scale: Discrete variables are still numeric in nature. For example, if the discrete values are [1, 10, 100], the ScaleType will determine whether the optimization algorithm considers "distance" between 1 and 10 versus 10 and 100 the same (if logarithmic) or smaller (if linear).