Search code examples
machine-learningpredictionxgboost

xgboost: what data to use in the watchlist

When running xgboost to optimize its hyperparameters, is it right that I use my dev (validation) set in the watchlist AND to draw predictions from to check its performance? I also had a question about following this same process when using hyperopt to search hyperparameters.

I am wondering if this is right because it is impressed that we never use the test set for anything other than to make a final prediction at the end. Thanks for your time.

# prepare xgboost data
xg_train = xgb.DMatrix(X_train, label=y_train)
xg_test = xgb.DMatrix(X_test, label=y_test)
xg_dev = xgb.DMatrix(X_dev, label=y_dev)

# validations set to watch performance
watchlist = [(xg_train,'train'), (xg_dev,'eval')]

# validation results
evals_results = {}

# boost
bst = xgb.train(params=param,
                dtrain=xg_train,
                num_boost_round=5000,
                evals=watchlist,
                evals_result=evals_results,
                early_stopping_rounds=100)

# generate scores
predicted_y = bst.predict(xg_dev, ntree_limit=bst.best_iteration)
current_score = f1_score(y_dev, np.round(predicted_y), average='weighted')
Solution

  • Chiming in to answer the question: yes the intuition is correct. Use the dev set to watch the training performance.