OOB score on train/test, accuracy and f1 score

Question

I'm really new to this and fairly confused. I'm training my model with a random forest (classification) and am trying to fully grasp the following concepts.

As far as I understand: you split you model with train/test split or cross validation or oob (bootstrapping methods) . Then the accuracy score or f1 score represents how well your model performs on the test set (accuracy being better for balances classes, f1 being better for unbalanced classes).

But then OOB score is a representation for how good your validation set is, so for how well the model is training on your data?

Am I misunderstanding soemthing here?

I'm mostly confused between the difference between accuracy/f1 scores and OOB scores.

Any input would be appreciated

Answer 1

These are 2 different aspects you're looking at:

1. Metrics, those are the mathematical formulas that you use to evaluate a model's performance on a set of data, so you'd give it the ground truth (real labels) and the predicted labels, and a metric score is computed, these metrics include:

   • Accuracy
   • Precision
   • Recall
   • F1
   • MSE
   • etc.

2. Variance-reduction, those are methods that you'd use to reduce the variance of the model, that is: prevent overfitting the model to the data, these methods include:

   • Using 2 different sets (i.e. train/test split)
   • Cross-validation (e.g. K-fold cross-validation, LOOCV, etc.)
   • Out of Bag, this one is particularly used in Random Forest algorithms to bootstrap the data that's used for each learner in the ensemble (forest).
   • etc.

So, basically, you use a method to try to reduce the variance of your model such that you'd improve the metrics.

As for your specific question: what is OOB score to the accuracy score? the OOB algorithm creates subsets of data that are used for training then computes the score using the metric against the predicted labels of these subsets.