Probably a very newbie question:
I'm working on a multi-class text classification project where all my features and labels are text based.
It has just came to my understanding that I'm not encoding the features and labels since I was relaying on the below:
def _create_transformer_from_feature_columns(columns):
tuples = []
for col in list(columns):
tfidf_kwargs = {'ngram_range': (1, 2), 'sublinear_tf': True}
if col not in NON_LEMMATIZED_COLUMN_NAMES:
tfidf_kwargs.update({'tokenizer': Tokenizer()})
tuples.append((f'vec_{col}', TfidfVectorizer(**tfidf_kwargs), col))
return ColumnTransformer(tuples, remainder='passthrough')
df_list = []
for bug in useful_bugs_dict.values():
# convert bug data into feature metric
bug_data = bug.get_data_as_df()
group_name = bug_data['group_name'][0]
if group_name not in group_owners_dict:
owner_id = len(group_owners_dict)
group_owners_dict[group_name] = owner_id
group_owner_id_dict[owner_id] = group_name
df_list.append(bug_data)
training_data = pd.concat(df_list)
training_data.reset_index(drop=True, inplace=True)
columns = training_data.drop('group_name', axis='columns').columns
transformer = _create_transformer_from_feature_columns(columns)
labels = training_data['group_name']
features = training_data.drop('group_name', axis='columns')
X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2)
Also I'm using XGBClassifier and I'm getting this warning:
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/xgboost/sklearn.py:1146:
UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release.
To remove this warning, do the following:
1) Pass option use_label_encoder=False when constructing XGBClassifier object;
and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
I was under the impression will do it for me.
Was I wrong?
The warning is unrelated to TfidfVectorizer. Its fit and fit_transform methods only rely on X to compute the tf-idf-weighted document-term matrix. y is ignored in both cases and its encoding is irrelevant.
For the scikit-learn classifiers, encoding y is also not mandatory. Passing string value objects in classification problems is usually not a problem. Note that the following code for a multiclass problem will execute without any issues:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.tree import DecisionTreeClassifier
X = ['doc one', 'doc two', 'number three']
y = [['yes', 'ok'], ['yes', 'not okay'], ['no', 'not okay']]
vec = TfidfVectorizer()
Xt = vec.fit_transform(X, y)
clf = DecisionTreeClassifier()
clf.fit(Xt, y)
The warning however is from the XGBClassifier which is not from scikit-learn. And apparently, the internal encoding of y is deprecated and will be removed in a future release. So in this particular case, you will have to do it explicitly yourself in the future, e.g when you use the next version(s).