Input Layer for Conv1D in Keras

Let me explain my problem and the dataset a bit. In my dataset, i have hourly measurements of a variable x and 7 more columns representing the day of the week that that measurment was taken as dummy variables. So, it is something like this:

DateTime          x   Mon Tue Wed Thur Fri Sat Sun
2017/01/01 00:00  10   0   0   0   0    0   0   1
2017/01/01 01:00  15   0   0   0   0    0   0   1
2017/01/01 02:00  21   0   0   0   0    0   0   1
             ...
2017/01/01 23:00  32   0   0   0   0    0   0   1
2017/01/02 00:00  17   1   0   0   0    0   0   0
2017/01/02 01:00  19   1   0   0   0    0   0   0
2017/01/02 02:00  48   0   0   0   0    0   0   1
             ...
2022/08/15 00:00  43   0   0   0   0    0   0   1

This dataset has shape of (49249, 8).

I am using data from 2017~2021 to train, and only 2022 would be used for test.

Using two days of data (48 rows), i need to forecast x in the next day. After changing this dataset to look like a supervised learning, i got a new training dataset with the shape (1842, 55). 55 is because im using two days of data (24+24) plus the dummy variables represnting the day i want to make a forecast (7 variables) 24+24+7 = 55.

Now, according Keras documentation the InputShape of a Conv1D layer should have shape of (batch_size, steps, input_dim).

So i did the following:

max_batch_size = 1824
steps = 48
input_dim = 55

So, i tried to reshaped it:

dataset = dataset.reshape(max_batch_size, steps, input_dim)

And i got this error:

ValueError: cannot reshape array of size 100320 into shape (1824,48,55)

I know this have been asked a lot, but i still don't understand it and none of the answers i have seen here have worked for me. How should i set my InputLayer to use with a Conv1D layer?

Solution

in this case you are trying to solve a sequence problem, so if i got it true, you would need only the output values, Make a list of values which are the outputs and then by

dataset = tf.data.Dataset.from_tensor_slices(list_of_values)

take the list of values, you said you would need two days to forecast the next day; so based on that you should windowing the output of the function above with the window size of two days, and one day, so it would mean it will take 48 values and then forecast for 24 values on the next day. so then;

data = data.window(window_size = 48 + 24, drop_remainder = True)
#set drop_remainder to true to drop sequences that are not in the same length as the others

the flat map and shuffle data as like as below

data = data.flat_map(lambda window : window.batch(48+24))
data = data.shuffle(shuffle_buffer)

now you should map data to separate the inputs and the outputs, somthing like as below

data = data.map(window : (window[:48], window[48:])
#by this way you set window size 48 values as the input and the others 24 values as the outputs (forecast)

and now set its batch size

data = data.batch(batch_size).prefetch()

now your data in windowed dataset which could be fed into convolution 1D and for the input layer there are two methods; you could set input shape into (48,) and then define lambda layer with lambda which reshape or expand dimensions or even set the input shape equal to (48,1) and after that it would fed into Conv1D layer

Notice In your case, by this way, the model would get 48 values in any format as i mean from each time of a day, but if your model would get its inputs from certain time for example 12 AM, so it would need to define shift parameter in window function and set it equal to 24 to take certain 24 values of each day from 12 AM