How to Merge Numerical and Embedding Sequential Models to treat categories in RNN
I would like to build a one layer LSTM model with embeddings for my categorical features. I currently have numerical features and a few categorical features, such as Location, which can't be one-hot encoded e.g. using pd.get_dummies() due to computational complexity, which is what I originally intended to do.
Let's visualise an example:
Sample Data
data = {
'user_id': [1,1,1,1,2,2,3],
'time_on_page': [10,20,30,20,15,10,40],
'location': ['London','New York', 'London', 'New York', 'Hong Kong', 'Tokyo', 'Madrid'],
'page_id': [5,4,2,1,6,8,2]
}
d = pd.DataFrame(data=data)
print(d)
user_id time_on_page location page_id
0 1 10 London 5
1 1 20 New York 4
2 1 30 London 2
3 1 20 New York 1
4 2 15 Hong Kong 6
5 2 10 Tokyo 8
6 3 40 Madrid 2
Let's look at the person visiting a website. I'm tracking numerical data such as time on page and others. Categorical data includes: Location (over 1000 uniques), Page_id (> 1000 uniques), Author_id (100+ uniques). The simplest solution would be to one-hot encoding everything and put this into LSTM with variable sequence lengths, each timestep corresponding to a different page view.
The above DataFrame will generate 7 training samples, with variable sequence lengths. For example, for user_id=2 I will have 2 training samples:
[ ROW_INDEX_4 ] and [ ROW_INDEX_4, ROW_INDEX_5 ]
Let X be the training data, and let's look at the first training sample X[0].
From the picture above, my categorical features are X[0][:, n:].
Before creating sequences, I factorized the categorical variables into [0,1... number_of_cats-1], using pd.factorize() so the data in X[0][:, n:] is numbers corresponding to their index.
Do I need to create an Embedding for each of the Categorical Features separately? E.g. an embedding for each of x_*n, x_*n+1, ..., x_*m?
If so, how do I put this into Keras code?
model = Sequential()
model.add(Embedding(?, ?, input_length=variable)) # How do I feed the data into this embedding? Only the categorical inputs.
model.add(LSTM())
model.add(Dense())
model.add.Activation('sigmoid')
model.compile()
model.fit_generator() # fits the `X[i]` one by one of variable length sequences.
My solution idea:
Something that looks like:
I can train a Word2Vec model on every single categorical feature (m-n) to vectorise any given value. E.g. London will be vectorised in 3 dimensions. Let's suppose I use 3 dimensional embeddings. Then I will put everything back into the X matrix, which will now have n + 3(n-m), and use the LSTM model to train it?
I just think there should be an easier/smarter way.
One solution, as you mentioned, is to one-hot encode the categorical data (or even use them as they are, in index-based format) and feed them along the numerical data to an LSTM layer. Of course, you can also have two LSTM layers here, one for processing the numerical data and another for processing categorical data (in one-hot encoded format or index-based format) and then merge their outputs.
Another solution is to have one separate embedding layer for each of those categorical data. Each embedding layer may have its own embedding dimension (and as suggested above, you may have more than one LSTM layer for processing numerical and categorical features separately):
num_cats = 3 # number of categorical features
n_steps = 100 # number of timesteps in each sample
n_numerical_feats = 10 # number of numerical features in each sample
cat_size = [1000, 500, 100] # number of categories in each categorical feature
cat_embd_dim = [50, 10, 100] # embedding dimension for each categorical feature
numerical_input = Input(shape=(n_steps, n_numerical_feats), name='numeric_input')
cat_inputs = []
for i in range(num_cats):
cat_inputs.append(Input(shape=(n_steps,1), name='cat' + str(i+1) + '_input'))
cat_embedded = []
for i in range(num_cats):
embed = TimeDistributed(Embedding(cat_size[i], cat_embd_dim[i]))(cat_inputs[i])
cat_embedded.append(embed)
cat_merged = concatenate(cat_embedded)
cat_merged = Reshape((n_steps, -1))(cat_merged)
merged = concatenate([numerical_input, cat_merged])
lstm_out = LSTM(64)(merged)
model = Model([numerical_input] + cat_inputs, lstm_out)
model.summary()
Here is the model summary:
Layer (type) Output Shape Param # Connected to
==================================================================================================
cat1_input (InputLayer) (None, 100, 1) 0
__________________________________________________________________________________________________
cat2_input (InputLayer) (None, 100, 1) 0
__________________________________________________________________________________________________
cat3_input (InputLayer) (None, 100, 1) 0
__________________________________________________________________________________________________
time_distributed_1 (TimeDistrib (None, 100, 1, 50) 50000 cat1_input[0][0]
__________________________________________________________________________________________________
time_distributed_2 (TimeDistrib (None, 100, 1, 10) 5000 cat2_input[0][0]
__________________________________________________________________________________________________
time_distributed_3 (TimeDistrib (None, 100, 1, 100) 10000 cat3_input[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 100, 1, 160) 0 time_distributed_1[0][0]
time_distributed_2[0][0]
time_distributed_3[0][0]
__________________________________________________________________________________________________
numeric_input (InputLayer) (None, 100, 10) 0
__________________________________________________________________________________________________
reshape_1 (Reshape) (None, 100, 160) 0 concatenate_1[0][0]
__________________________________________________________________________________________________
concatenate_2 (Concatenate) (None, 100, 170) 0 numeric_input[0][0]
reshape_1[0][0]
__________________________________________________________________________________________________
lstm_1 (LSTM) (None, 64) 60160 concatenate_2[0][0]
==================================================================================================
Total params: 125,160
Trainable params: 125,160
Non-trainable params: 0
__________________________________________________________________________________________________
Yet there is another solution which you can try: just have one embedding layer for all the categorical features. It involves some preprocessing though: you need to re-index all the categories to make them distinct from each other. For example, the categories in first categorical feature would be numbered from 1 to size_first_cat and then the categories in the second categorical feature would be numbered from size_first_cat + 1 to size_first_cat + size_second_cat and so on. However, in this solution all the categorical features would have the same embedding dimension since we are using only one embedding layer.
Update: Now that I think about it, you can also reshape the categorical features in data preprocessing stage or even in the model to get rid of TimeDistributed layers and the Reshape layer (and this may increase the training speed as well):
numerical_input = Input(shape=(n_steps, n_numerical_feats), name='numeric_input')
cat_inputs = []
for i in range(num_cats):
cat_inputs.append(Input(shape=(n_steps,), name='cat' + str(i+1) + '_input'))
cat_embedded = []
for i in range(num_cats):
embed = Embedding(cat_size[i], cat_embd_dim[i])(cat_inputs[i])
cat_embedded.append(embed)
cat_merged = concatenate(cat_embedded)
merged = concatenate([numerical_input, cat_merged])
lstm_out = LSTM(64)(merged)
model = Model([numerical_input] + cat_inputs, lstm_out)
Model summary:
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
cat1_input (InputLayer) [(None, 100)] 0 []
cat2_input (InputLayer) [(None, 100)] 0 []
cat3_input (InputLayer) [(None, 100)] 0 []
embedding_14 (Embedding) (None, 100, 50) 50000 ['cat1_input[0][0]']
embedding_15 (Embedding) (None, 100, 10) 5000 ['cat2_input[0][0]']
embedding_16 (Embedding) (None, 100, 100) 10000 ['cat3_input[0][0]']
numeric_input (InputLayer) [(None, 100, 10)] 0 []
concatenate_26 (Concatenat (None, 100, 160) 0 ['embedding_14[0][0]',
e) 'embedding_15[0][0]',
'embedding_16[0][0]']
concatenate_27 (Concatenat (None, 100, 170) 0 ['numeric_input[0][0]',
e) 'concatenate_26[0][0]']
lstm_5 (LSTM) (None, 64) 60160 ['concatenate_27[0][0]']
==================================================================================================
Total params: 125160 (488.91 KB)
Trainable params: 125160 (488.91 KB)
Non-trainable params: 0 (0.00 Byte)
__________________________________________________________________________________________________
As for fitting the model, you need to feed each input layer separately with its own corresponding numpy array, for example:
X_tr_numerical = X_train[:,:,:n_numerical_feats]
# extract categorical features: you can use a for loop to this as well.
# note that we reshape categorical features to make them consistent with the updated solution
X_tr_cat1 = X_train[:,:,cat1_idx].reshape(-1, n_steps)
X_tr_cat2 = X_train[:,:,cat2_idx].reshape(-1, n_steps)
X_tr_cat3 = X_train[:,:,cat3_idx].reshape(-1, n_steps)
# don't forget to compile the model ...
# fit the model
model.fit([X_tr_numerical, X_tr_cat1, X_tr_cat2, X_tr_cat3], y_train, ...)
# or you can use input layer names instead
model.fit({'numeric_input': X_tr_numerical,
'cat1_input': X_tr_cat1,
'cat2_input': X_tr_cat2,
'cat3_input': X_tr_cat3}, y_train, ...)
If you would like to use fit_generator() there is no difference:
# if you are using a generator
def my_generator(...):
# prep the data ...
yield [batch_tr_numerical, batch_tr_cat1, batch_tr_cat2, batch_tr_cat3], batch_tr_y
# or use the names
yield {'numeric_input': batch_tr_numerical,
'cat1_input': batch_tr_cat1,
'cat2_input': batch_tr_cat2,
'cat3_input': batch_tr_cat3}, batch_tr_y
model.fit_generator(my_generator(...), ...)
# or if you are subclassing Sequence class
class MySequnece(Sequence):
def __init__(self, x_set, y_set, batch_size):
# initialize the data
def __getitem__(self, idx):
# fetch data for the given batch index (i.e. idx)
# same as the generator above but use `return` instead of `yield`
model.fit_generator(MySequence(...), ...)
- Change value in Ini to empty using Python configparser
- Script works with error line but won't run corectly when bad line is removed
- FFMPEG not saving logs when converting to audio format
- Numerically obtaining response of a damped driven oscillator gives peak at wrong frequency
- What is the deal with the pony in Python community?
- 'Area' object is not callable
- ValueError: Attribute Users.request is required
- How to place class in its own file when it appears to be inheriting from an instance?
- Python 3.6 with pony 0.7 gives error on commit to oracle db
- Why pyqt tablewigdet is only displaying row number and no data?
- How to check if a value exists in a dictionary?
- OpenCV: draw a rectangle around a region
- Python, Tkinter, trying to pull random numbers from a list based off user input for number and have results open in mew window
- How to remove repeated elements in a vector, similar to 'set' in Python
- Understanding descriptor protocol for 'wrapper-descriptor' itself
- Passing a NumPy 3d array to a C function with a triple pointer as an argument
- How do I create variable variables?
- Turn a tf.data.Dataset to a jax.numpy iterator
- Django 5.1 + Postgresql (debian server)
- Passing in arguments to Dependency function makes it recognized as a Query Parameter
- How to pass parameters to an endpoint using `add_route()` in FastAPI?
- How to make Depends optional in FastAPI?
- Streaming multiple videos through FastAPI to Web Browser causes HTTP requests to stall
- FastAPI error when handling file together with form-data defined in a Pydantic model
- Use serial port in Python without installing external packages
- How can i get the output to print 10 circles instead of 9 in python turtle module
- Unable to install TA-Lib on Ubuntu
- Unable to install Python without sudo access
- Size of an open file object
- two if or more with one ELSE