Kerastuner search doesn't get restarted even with Overwrite flag set to true

Question

Tuning is done as follows:

tuner = kt.RandomSearch(
    MyHyperModel(),
    objective="mae",
    max_trials=30,
    overwrite=True,
    directory=results_dir,
    project_name="tune_hypermodel",
)

And I'm iterating over how many features to use:

data[name]=pd.read_pickle(os.path.join(root, name)+'/'+name+'.data.reindexed_by_pc.pkl')
CpG_num_lst=[100,500,1000,5000,10000,20000,40000]
train_score, valid_score = [], []
hps_by_CpG_num = []

for CpG_num in CpG_num_lst:
    print("CpG_num:",CpG_num)
    # force overwrite tune search (to start new search). Cause even with overwrite=True it doesn't overwrite
    if  os.path.exists(results_dir+'/tune_hypermodel'):
        shutil.rmtree(results_dir+'/tune_hypermodel')
        
    # initialize 
    X1, y1, X2, y2 = [dict() for _ in range(4)]
    X1[name] = data[name][fold1_ids].head(CpG_num).values.T
    X2[name] = data[name][fold2_ids].head(CpG_num).values.T
    # get the ages of the corresponding persons. Notice info_1 and info_2 only contain "Ctrl" and not "Test" samples
    y1[name] = info_1[name].query("`Train.Test`=='Train'")['Age'].values.astype(float) 
    y2[name] = info_2[name].query("`Train.Test`=='Train'")['Age'].values.astype(float) 
    # Split the data
    X1_train, X1_valid, y1_train, y1_valid = train_test_split(X1[name], y1[name], test_size=0.2, shuffle= True)
    # Grid search
    tuner.search(X1_train, y1_train, validation_data = (X1_valid,y1_valid))
    best_hps = tuner.get_best_hyperparameters(num_trials=1)[0]
    # Get best hyperparameters
    hp_dict = dict()
    hp_dict['num_layers'] = best_hps.get('num_layers')
    hp_dict['batch_size'] = best_hps.get('batch_size')
    hp_dict['act_1'] = best_hps.get('act_1')
    hp_dict['act_2'] = best_hps.get('act_2')
    hp_dict['units_1'] = best_hps.get('units_1')
    hp_dict['units_2'] = best_hps.get('units_2')
    hps_by_CpG_num.append(hp_dict)
    
    # Build best model
    best_model = MyHyperModel().build(hp=best_hps)
    history = best_model.fit(X1_train, y1_train, validation_data = (X1_valid,y1_valid), batch_size=best_hps.get('batch_size'), epochs=200)
    

However for each element in the for loop the tuner search doesn't restart, it just uses the best hyperparameters from the first search (CpG_num = 500)

What am I missing? Why is Keras taking old hyperparameters?

Answer 1

The solution was to include the tuner instantiation within the for loop.

tuner = kt.RandomSearch(
    MyHyperModel(),
    objective="mae",
    max_trials=30,
    overwrite=True,
    directory=results_dir,
    project_name="tune_hypermodel",
    )

Not sure why though, but works... If somebody has any insight on this let me know. I thought the tuner would be overwritten. Is there a better way of doing this?