Scaling of stock data

Question

I am trying to apply machine learning on stock prediction, and I run into problem regarding scaling on future unseen (much higher) stock close value.

Lets say I use random forrest regression on predicting stock price. I break the data into train set and test set. For the train set, I use standardscaler, and do fit and transform And then I use regressor to fit For the test set, I use standardscaler, and do transform And then I use regressor to predict, and compare to test label

If I plot predict and test label on a graph, predict seems to max out or ceiling. The problem is that standardscaler fit on train set, test set (later in the timeline) have much higher value, the algorithm does not know what to do with these extreme data

def test(X, y):
    # split the data
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, shuffle=False)

    # preprocess the data
    pipeline = Pipeline([
    ('std_scaler', StandardScaler()),
    ])
    # model = LinearRegression()
    model = RandomForestRegressor(n_estimators=20, random_state=0)

    # preprocessing fit transform on train data
    X_train = pipeline.fit_transform(X_train)
    # fit model on train data with train label
    model.fit(X_train, y_train)
    # transform on test data
    X_test = pipeline.transform(X_test)
    # predict on test data
    y_pred = model.predict(X_test)
    # print(np.sqrt(mean_squared_error(y_test, y_pred)))

    d = {'actual': y_test, 'predict': y_pred}
    plot_data = pd.DataFrame.from_dict(d)
    sns.lineplot(data=plot_data)
    plt.show()

What should be done with the scaling?

This is what I got for plotting prediction, actual close price vs time

Answer 1

The problem mainly comes from the model you are using. RandomForest regressor is created upon Decision Trees. It is learning to map an input to an output for every examples in the training set. Consequently RandomForest regressor will work for middle values but for extreme values that it hasn't seen during training it will of course perform has your picture is showing. What you want, is to learn a function directly using linear/polynomial regression or more advanced algorithms like ARIMA.