How to deep-learn simple data in exponential form?

Question

I want to deep-learn data in the form of exponential functions but it doesn't seem to be properly learned. Changing the activation function hasn't helped at all.

# 딥러닝을 구동하는 데 필요한 케라스 함수 호출
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# 필요한 라이브러리 불러오기
import numpy as np
import tensorflow as tf

# 실행할 때마다 같은 결과를 출력하기 위해 설정하는 부분
np.random.seed(3)
tf.random.set_seed(3)

# 준비된 수술 환자 데이터를 불러오기
Data_set = np.loadtxt("size.csv", delimiter = ",")

# 환자의 기록과 수술 결과를 X와 Y로 구분하여 저장
X = Data_set[:,0]
Y = Data_set[:,1]

print(X)
print(Y)

Output:

[1992. 1994. 1996. 1998. 1999. 2001. 2004. 2006. 2007. 2009. 2011. 2014.
 2015. 2017. 2019. 2020. 2021. 2022. 2023.]
[800. 500. 350. 250. 180. 130.  90.  65.  45.  40.  28.  22.  14.  10.
   7.   6.   5.   4.   3.]

I think the model has too high a loss but I don't know how to make the right model. Even if I increase epochs to 10000 the loss doesn't decrease any more. How do I solve this problem?

# 딥러닝 구조를 결정 (모델을 설정하고 실행)
model = Sequential()
model.add(Dense(30, input_dim = 1, activation = 'relu'))
model.add(Dense(1, activation = 'sigmoid'))

# 딥러닝 실행
model.compile(loss = 'mean_squared_error', optimizer = 'adam', metrics = ['accuracy'])
model.fit(X, Y, epochs = 100, batch_size = 10)

Output:

2/2 [==============================] - 0s 9ms/step - loss: 60102.5781 - accuracy: 0.0000e+00
Epoch 99/100
2/2 [==============================] - 0s 8ms/step - loss: 60102.5781 - accuracy: 0.0000e+00
Epoch 100/100
2/2 [==============================] - 0s 8ms/step - loss: 60102.5781 - accuracy: 0.0000e+00

Answer 1

several issues to remember when training neural nets:

1. Use correct activation functions. In your case output activation function is sigmoid however the target Y is exponential. It is better to change output activation function to exponential or think about preprocessing target Y before passing it to the model. You can't use sidmoid because it outputs [-1,1], but all values in Y are very large and definitely above 1.
2. Input X should be properly scaled before entering the model. This is because initialialized NN weights are very small random numbers that are very close to zero. So it is best practice to scale input data in way that it is distributed around 0: part of the data in negative domain and part in positive. You can do this by applying standard scaler or similar preprocessing steps.

X = [1992, 1994, 1996, 1998, 1999, 2001, 2004, 2006, 2007, 2009, 2011, 2014,
 2015, 2017, 2019, 2020, 2021, 2022, 2023,]

Y = [800, 500, 350, 250, 180, 130,  90,  65,  45,  40,  28,  22,  14,  10,
   7,   6,   5,   4,   3,]

X = tf.constant(X)
Y = tf.constant(Y)


model = Sequential()
model.add(Dense(30, input_dim = 1, activation = 'relu'))
model.add(Dense(1,  activation =  'exponential'))

def scale_standard(Z):
    return (Z - tf.math.reduce_min(Z)) / (tf.math.reduce_max(Z) - tf.math.reduce_min(Z))    

def scale_max(Z):
    return Z / tf.math.reduce_max(Z)
                                          
X = scale_standard(X)

pred = model(X)
print(pred)


# 딥러닝 실행
model.compile(loss = 'mean_squared_error', 
              optimizer = 'adam', 
              metrics = ['r2_score'])

model.fit(X, Y, epochs = 1000, batch_size = 10)

Outputs:

Epoch 998/1000
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 140.9382 - r2_score: 0.9969 
Epoch 999/1000
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 140.9404 - r2_score: 0.9969 
Epoch 1000/1000
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 140.9404 - r2_score: 0.9969