How to fix ArrayMemoryError using BinaryRelevance even using csr_matrix?

Question

I am trying to predict toxic comments using Toxic Comment data from kaggle:

import skmultilearn, sys
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from scipy.sparse import csr_matrix, issparse
from sklearn.naive_bayes import MultinomialNB
from skmultilearn.problem_transform import BinaryRelevance

data_frame = pd.read_csv('data/train.csv')
corpus = data_frame['comment_text']
tfidf = TfidfVectorizer()
Xfeatures = csr_matrix(tfidf.fit_transform(corpus))
y = csr_matrix(data_frame[['toxic','severe_toxic','obscene','threat','insult','identity_hate']])
binary_rel_clf = BinaryRelevance(MultinomialNB())
binary_rel_clf.fit(Xfeatures,y)
predict_text = ['fuck die shit moron suck']
X_predict = tfidf.transform(predict_text)
br_prediction = binary_rel_clf.predict(X_predict)
br_prediction = br_prediction.toarray().astype(bool)
predictions = [y.columns.values[prediction].tolist() for prediction in br_prediction]
print(predictions)

However, I got this error:

Traceback (most recent call last):
  File "...\multi_label_toxic.py", line 15, in <module>
    binary_rel_clf.fit(Xfeatures,y)
  File "...\problem_transform\br.py", line 161, in fit
    classifier.fit(self._ensure_input_format(
  File "...\base\base.py", line 86, in _ensure_input_format
    return X.toarray()
  File "...\scipy\sparse\compressed.py", line 1031, in toarray
    out = self._process_toarray_args(order, out)
  File "...\scipy\sparse\base.py", line 1202, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 226. GiB for an array with shape (159571, 189775) and data type float64

And even if try to pass the param "require_dense=False" I got another error:

Traceback (most recent call last):
  File "...\multi_label_toxic.py", line 15, in <module>
    binary_rel_clf.fit(Xfeatures,y)
  File "...\skmultilearn\problem_transform\br.py", line 161, in fit
    classifier.fit(self._ensure_input_format(
  File "...\sklearn\naive_bayes.py", line 612, in fit
    X, y = self._check_X_y(X, y)
  File "...\sklearn\naive_bayes.py", line 477, in _check_X_y
    return self._validate_data(X, y, accept_sparse='csr')
  File "...\sklearn\base.py", line 433, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "...\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "...\sklearn\utils\validation.py", line 826, in check_X_y
    y = column_or_1d(y, warn=True)
  File "...\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "...\sklearn\utils\validation.py", line 864, in column_or_1d
    raise ValueError(
ValueError: y should be a 1d array, got an array of shape () instead.

How can I fix that and train using the entire model?

Answer 1

It seems that you specified the required_dense argument incorrectly. You need required_dense=[False, True] in order to specify the X values in sparse format but not the y values. In the second last row (predictions = ...) you need to use y before you convert it to a matrix so you can access the column names. The following code should work.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from scipy.sparse import csr_matrix, issparse
from sklearn.naive_bayes import MultinomialNB
from skmultilearn.problem_transform import BinaryRelevance
import numpy as np

data_frame = pd.read_csv('data/train.csv')
corpus = data_frame['comment_text']
tfidf = TfidfVectorizer()
Xfeatures = csr_matrix(tfidf.fit_transform(corpus))
cats = data_frame[['toxic','severe_toxic','obscene','threat','insult','identity_hate']]
y = csr_matrix(cats)
binary_rel_clf = BinaryRelevance(MultinomialNB(), require_dense = [False, True])
binary_rel_clf.fit(Xfeatures, y) # y[:,0].toarray().reshape(-1)
predict_text = ['fuck die shit moron suck']
X_predict = tfidf.transform(predict_text)
br_prediction = binary_rel_clf.predict(X_predict)
br_prediction = br_prediction.toarray().astype(bool)
predictions = [cats.columns[prediction].tolist() for prediction in br_prediction]
print(predictions)

Output:

[['toxic', 'obscene', 'insult']]