How to find coefficients in multi-variable gradient descent?

Question

I need to implement gradient descent on my own. My task is to create an arbitrary function, add noise to it, and then find the coefficients values for that function. So first, I created a function and created some random values:

# Preprocessing Input data
function = lambda x: x ** 2 +(x)+1 
X=[]
Y=[]
for i in range(-100,100):
  X.append(i)
  Y.append(function(i) + random.randrange(-10,10)

then I normalized the values-

maxVal = np.max(np.hstack((X,Y)))
X = X/maxVal 
Y = Y/maxVal

X= np.asarray(X)
Y= np.asarray(Y)

and this is my code for gradient descent, using derivative to find the coefficients

w1Arr = []
w2Arr = []
bArr = []
lossArr = []

for i in range(epochs):
  Y_pred =w1*np.square(X)+w2*X+b

  D_w1 = (-2/n) * sum( np.square(X) * (Y - Y_pred))  # Derivative for w1

  D_w2 = (-2/n) * sum(X * (Y - Y_pred))  # Derivative for w2

  D_b = (-2/n) * sum(Y - Y_pred)        # Derivative for b

  w1 = w1 - L * D_w1  # Update w1
  w2 = w2 - L * D_w2  # Update w2
  b = b - L * D_b  # Update b

  loss = sum((Y - Y_pred) * (Y - Y_pred)) #MSE
  w1Arr.append(w1)
  w2Arr.append(w2)
  bArr.append(b)
  lossArr.append(loss)

when I try to plot the results:

# Making predictions
Y_pred = w1*(np.square(X))+w2*X+b
#print(Y_pred)
plt.scatter(X, Y)
plt.plot(X, Y_pred) # predicted
plt.legend()
plt.show()

I see that the coefficients are pretty much the same,and just looks like a linear line-

I'm pretty much stuck, and don't know what is wrong with my code or how to fix it. I looked online looking for solutions, but couldn't find any. any help would be appreciated.

Answer 1

Found out the problem!

The normalization you applied just messed up the relation between the x and y, in particular you skewed the domain with respect of the codomain:

maxVal = np.max(np.hstack((X,Y)))
X = X/maxVal 
Y = Y/maxVal

Just remove the normalization and you will find that you can learn the coefficients.

If you really want, you can normalize both the axis, but they have to be two proportional values:

X = X / np.max(X)
Y = Y / np.max(Y)