Error in keras model for classification model with transformers

Question

I followed this tutorial: https://keras.io/examples/timeseries/timeseries_transformer_classification/ For classification model with transformers to my time-series. However, in the line:

x = layers.MultiHeadAttention(
            key_dim=head_size, num_heads=num_heads, dropout=dropout
        )(x, x)

I get the error:

{IndexError}tuple index out of range

Any ideas why?

Answer 1

Disclosure: I came here for the bounty, then I tried on Colab and everything worked fine..

Next I read the comments: "This question is a joke in its current state. There is no way to reproduce it." and at this point I agree. But as I am a Hans in Luck and obviously have to much time procrastinating, I started Pycharm following the OPs cue: "No, when I paste it to my pycharm I get the above error"

But this also worked for me, which makes me wonder whether you have touched something, so I am happy to provide a(n) (untouched) working version for you..

import numpy as np

def readucr(filename):
    data = np.loadtxt(filename, delimiter="\t")
    y = data[:, 0]
    x = data[:, 1:]
    return x, y.astype(int)


root_url = "https://raw.githubusercontent.com/hfawaz/cd-diagram/master/FordA/"

x_train, y_train = readucr(root_url + "FordA_TRAIN.tsv")
x_test, y_test = readucr(root_url + "FordA_TEST.tsv")

x_train = x_train.reshape((x_train.shape[0], x_train.shape[1], 1))
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1], 1))

n_classes = len(np.unique(y_train))

idx = np.random.permutation(len(x_train))
x_train = x_train[idx]
y_train = y_train[idx]

y_train[y_train == -1] = 0
y_test[y_test == -1] = 0


from tensorflow import keras
from tensorflow.keras import layers

def transformer_encoder(inputs, head_size, num_heads, ff_dim, dropout=0):
    # Normalization and Attention
    x = layers.LayerNormalization(epsilon=1e-6)(inputs)
    x = layers.MultiHeadAttention(
        key_dim=head_size, num_heads=num_heads, dropout=dropout
    )(x, x)
    x = layers.Dropout(dropout)(x)
    res = x + inputs

    # Feed Forward Part
    x = layers.LayerNormalization(epsilon=1e-6)(res)
    x = layers.Conv1D(filters=ff_dim, kernel_size=1, activation="relu")(x)
    x = layers.Dropout(dropout)(x)
    x = layers.Conv1D(filters=inputs.shape[-1], kernel_size=1)(x)
    return x + res

Also to make sure that we are talking of the same package versions

I used numpy (1.21.2) and tensorflow (2.6.0) - try with these versions or let me know in case you used different versions.