Why am I getting horizontal line (almost zero) from neural network instead of the desired curve?

Question

I am trying to use neural network for my regression problem in python but the output of the neural network is a straight horizontal line which is zero. I have one input and obviously one output. Here is my code:

def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(1, input_dim=1, kernel_initializer='normal', activation='relu'))
    model.add(Dense(4, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error',metrics=['mse'], optimizer='adam')

    model.summary()
    return model


# evaluate model
estimator = KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=64,validation_split = 0.2, verbose=1)
kfold = KFold(n_splits=10)
results = cross_val_score(estimator, X_train, y_train, cv=kfold)

Here are the plots of NN prediction vs. target for both training and test data.

Training Data Test Data

I have also tried different weight initializers (Xavier and He) with no luck! I really appreciate your help

Answer 1

First of all correct your syntax while adding dense layers in model remove the double equal == with single equal = with kernal_initilizer like below

 model.add(Dense(1, input_dim=1, kernel_initializer ='normal', activation='relu'))

Then to make the performance better do the followong

1. Increase the number of hidden neurons in the hidden layers

2. Increase the number of hidden layers.

If still you have same problem then try to change the optimizer and activation function. Tuning the hyperparameters may help you in converging to the solution

EDIT 1

You also have to fit the estimator after cross validation like below

estimator.fit(X_train, y_train)

and then you can test on the test data as follow

prediction = estimator.predict(X_test)

from sklearn.metrics import accuracy_score
accuracy_score(Y_test, prediction)