l have 66 classes of characters. l trained a multi layer perceptron feeding it with images and the class of each image is a character
Aa..Zz 0-9 ,;:! è à â ~
l applied a LabelBinarizer to transform each character to a vector of 66 classes, (e.g.[0 0 0 0 ....1 ...... 0]) because models don't accept non numerical data.
l was surprised to get dimension 61 for y_test (wrong) an y_train (66 correct).
what's wrong with my code ?
from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(data_pixels, classes_dataset, test_size=0.3)
print(x_train.shape)
print(y_train.shape)
print(x_test.shape)
print(y_test.shape)
print(x_test)
print("my y_train")
print(y_train[0:100])
print("my y_test ")
print(y_test[0:100])
**
(1708, 3072) # x_train shape
(1708,) # y_train shape
(732, 3072) #x_test shape
(732,) y_train shape
**
[[ 1. 1. 1. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
...,
[ 1. 1. 1. ..., 1. 1. 1.]
[ 1. 1. 1. ..., 1. 1. 1.]
[ 1. 1. 1. ..., 1. 1. 1.]]
y_train
['5' 'O' '9' '0' 'E' 'D' '9' ',' 'R' 'H' 'T' 'i' '\xc3\xa9' 'T' 'o' 'u' 'R'
'2' '0' 'L' 'o' '0' '6' 'q' 'P' '6' '2' 'T' '2' '0' 'i' '0' 'f' 'A' 'r'
'n' 'T' 'O' '8' 'B' 'T' 'd' '0' 'V' 'X' '9' '.' '6' 'J' 'S' 'E' 'O' 'T'
'4' 'E' '0' 'I' '3' 'o' 'E' '6' 'R' 'M' '0' 'E' '1' 'R' 'T' 'E' '.' '0'
'G' 'R' 'E' 'E' '0' '9' 'd' '1' '7' 'A' 'B' 'L' '4' 'l' 'O' '1' 'v' '3'
'%' 'd' '0' 'T' 's' 'A' '6' 'w' 'slash' '2' '9']
y_test
['N' '0' 'F' '4' 'U' 'C' 'u' 'e' '0' ',' 'G' 'C' 'T' '%' '-' 'V' '5' 'P'
'N' 'S' '8' '4' ',' 'm' '5' '3' 'e' 'I' 'i' 'M' 'I' '3' 'C' 'F' 'e' 'a'
'6' 'R' 'V' '4' '0' 'f' '9' 'E' '2' '0' 'E' 'N' 'I' '5' '0' 'A' '%' '-'
'G' '0' ',' 'O' 'Y' '\xc3\x89' 'R' 's' ',' 'A' 'I' '3' 'S' '2' 'P' '.' 'I'
',' 'r' 'I' 'i' '5' '5' 'R' 'C' 'e' '2' 'q' '.' 'R' 'O' 'n' 'S' '6' 'G'
'0' 'R' 'i' 't' 'i' '9' 'I' 'D' 'slash' '0' 'A']
when l apply LabelBinazer l get y_test dimension (732,61) rather than (732,66)
66 means the number of classes :
from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()
y_train = encoder.fit_transform(y_train)
y_test= encoder.fit_transform(y_test)
print(y_train.shape)
print(y_test.shape)
(1708, 66)
(732, 61) # How come l get 61 rather than 66
**Edit**
After making changes as suggested
y_test= encoder.transform(y_test)
l got the following error :
ValueError Traceback (most recent call last)
<ipython-input-109-b554d33049ab> in <module>()
1 y_said=encoder.fit(y_train)
2 y_said
----> 3 y_test= encoder.transform(y_test)
/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/label.pyc in transform(self, y)
333 pos_label=self.pos_label,
334 neg_label=self.neg_label,
--> 335 sparse_output=self.sparse_output)
336
337 def inverse_transform(self, Y, threshold=None):
/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/label.pyc in label_binarize(y, classes, neg_label, pos_label, sparse_output)
492 if (y_type == "multilabel-indicator" and classes.size != y.shape[1]):
493 raise ValueError("classes {0} missmatch with the labels {1}"
--> 494 "found in the data".format(classes, unique_labels(y)))
495
496 if y_type in ("binary", "multiclass"):
ValueError: classes [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65] missmatch with the labels [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59 60]found in the data
EDIT2 :
encoder = LabelBinarizer()
encoder.fit(y_train + y_test)
y_train= encoder.transform(y_train)
y_test= encoder.transform(y_test)
I got the following error :
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-125-45c4512ecc1f> in <module>()
1 from sklearn.preprocessing import LabelBinarizer
2 encoder = LabelBinarizer()
----> 3 encoder.fit(y_train + y_test)
4 y_train= encoder.transform(y_train)
5 y_test= encoder.transform(y_test)
TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('S5') dtype('S5') dtype('S5')
EDIT3: My code from jupyter :
from __future__ import print_function
from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as np
# fix random seed for reproducibility
numpy.random.seed(7)
batch_size = 128
num_classes = 66
epochs = 12
data_pixels=np.genfromtxt("pixels_dataset.csv", delimiter=',')
classes_dataset=np.genfromtxt("labels.csv",dtype=np.str , delimiter='\t')
from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(data_pixels, classes_dataset, test_size=0.3)
from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()
encoder.fit(y_train + y_test)
y_train= encoder.transform(y_train)
y_test= encoder.transform(y_test)
You need to transform your y_test using encoder fitted on your train data. So you need to change your code to:
y_test= encoder.transform(y_test)
Otherwise your encoder would be refitted what causes a problem.
edit:
You may also try:
encoder.fit(y_train + y_test) # in case of lists
encoder.fit(numpy.append(y_train, y_test)) # in case of numpy arrays
y_train= encoder.transform(y_train)
y_test= encoder.transform(y_test)
edit 2:
What finally succeed to work was:
encoder.fit(numpy.append(y_train, y_test)) # in case of numpy arrays
y_train= encoder.fit_transform(y_train)
y_test= encoder.transform(y_test)