hydras sweep with Ax across different config groups

Question

I have a config group called scheduler which contains many optimizers scheduler, each one has it's own parameters. Is it possible to create a sweep such that I can explore many schedulers and in the same time at each scheduler explore different parameters of that scheduler.

What I have currently is the following:

  sweeper:
    ax_config:
      params:
        general.batch_size:
          type: choice
          values:
            - 4
            - 8
            - 16
            - 32
            - 64
        general.lr:
          type: range
          bounds: [0.00025, 0.025]
        scheduler:
          type: choice
          values:
            - step_lr
            - cyclic_lr
            - one_cycle_lr 

What I want something like this:

  values:
    - step_lr
      gamma:
       type: range
       bounds: [0.9, 0.99]

    - cyclic_lr
    - one_cycle_lr

Answer 1

Basically I think you want to use grid search sweeper to sweep over the AX sweeper . This is not supported. You will need to create different sweeps for each parameter in this case.