I wrote a linear regression from scratch but the loss is increasing. My data are the areas and the prices (as labels) of the houston housing dataset. I tried multiple learning-rates (from 10 to 0.00000000001), but its still not working. With every epoch, my fit-line/function keeps moving further away from the data points. There must be something wrong with the functions I guess, but I cant figure out what. Here is an example of the loss:
loss: 0.5977188541860982
loss: 0.6003449724263221
loss: 0.6029841845821928
loss: 0.6056365560589673
loss: 0.6083021525886172
loss: 0.6109810402314608
loss: 0.6136732853778034
loss: 0.6163789547495854
loss: 0.6190981154020385
loss: 0.6218308347253524
loss: 0.6245771804463445
And here the code:
from preprocessing import load_csv
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
# mean squared error
def MSE(y_prediction, y_true, deriv=(False, 1)):
if deriv[0]:
# deriv[1] is the derivitive of the fit_function
return 2 * np.mean(np.subtract(y_true, y_prediction) * deriv[1])
return np.mean(np.square(np.subtract(y_true, y_prediction)))
# linear function
def fit_function(theta_0, theta_1, x):
return theta_0 + (theta_1 * x)
# train model
def train(dataset, epochs=10, lr=0.01):
# loadinh and normalizing the data
x = (v := np.array(dataset["GrLivArea"].tolist()[:100])) / max(v)
y = (l := np.array(dataset["SalePrice"].tolist()[:100])) / max(l)
# y-intercept
theta_0 = random.uniform(min(y), max(y))
# slope
theta_1 = random.uniform(-1, 1)
for epoch in range(epochs):
predictions = fit_function(theta_0, theta_1, x)
loss = MSE(predictions, y)
delta_theta_0 = MSE(predictions, y, deriv=(True, 1))
delta_theta_1 = MSE(predictions, y, deriv=(True, x))
theta_0 -= lr * delta_theta_0
theta_1 -= lr * delta_theta_1
print("\nloss:", loss)
plt.style.use("ggplot")
plt.scatter(x, y)
x, predictions = map(list, zip(*sorted(zip(x, predictions))))
plt.plot(x, predictions, "b--")
plt.show()
train(load_csv("dataset/houston_housing/single_variable_dataset/train.csv"), epochs=500, lr=0.001)
Here is the plot after 500 epochs.
Thanks for your help :)
Quite an old post, but I thought I'd give an answer anyway.
You flipped the sign on the MSE derivative:
def MSE(y_prediction, y_true, deriv=(False, 1)):
if deriv[0]:
return 2 * np.mean(np.subtract(y_prediction, y_true) * deriv[1])
return np.mean(np.square(np.subtract(y_true, y_prediction)))
The partial derivatives w.r.t. you parameters are:
For conciseness:
def MSE(y_prediction, y_true, deriv=None):
if deriv is not None:
return 2 * np.mean((y_prediction - y_true)*deriv)
return np.mean((y_prediction - y_true)**2)
Which allows you to get the derivatives without passing in a tuple with a flag:
delta_theta_0 = MSE(predictions, y, deriv=1)
delta_theta_1 = MSE(predictions, y, deriv=x)
Here's an example using sklearn.datasets.load_boston
with LSTAT
(lower status of the population) and MEDV
(Median value of owner-occupied homes in $1000's) as target the last two data features as input and target respectively.
Trained with epochs=10000
and lr=0.001
: