I implemented a simple Linear Regression model using Tensor Flow. However it only works for around 10-15 data points. Any more than that and the the loss function starts drastically increasing until it reaches infinity. The data is correct because I have synthetically generated it. The sklearn Linear Regression model works perfectly for the same data.
size = 8
x = np.float32(np.arange(size).reshape((size,1)))
y = x*8
class Linear_Model():
def __init__(self,input_dim,lr=0.01):
self.w = tf.Variable(tf.ones(shape=(input_dim,1)))
self.b= tf.Variable(tf.zeros(shape=(input_dim)))
self.lr = lr
def predict(self,x):
return tf.matmul(x,self.w) + self.b
def compute_loss(self,label,predictions):
return tf.reduce_mean(tf.square(label-predictions))
def train(self,x,y,epochs=12,batch_size=64):
dataset = tf.data.Dataset.from_tensor_slices((x,y))
dataset = dataset.shuffle(buffer_size=1024).batch(batch_size)
for i in range(epochs):
start = time.time()
for step,(x,y) in enumerate(dataset):
with tf.GradientTape() as tape:
preds = self.predict(x)
loss = self.compute_loss(y,preds)
dw,db = tape.gradient(loss,[self.w,self.b])
self.w.assign_sub(self.lr*dw)
self.b.assign_sub(self.lr*db)
print("Epoch %d : loss = %.4f time taken = %.3f s"% (i,float(loss),time.time()-start))
model = Linear_Model(1,0.01)
model.train(x,y,epochs=15)
Edit - From playing around with the learning rate I saw that a learning rate of 0.01 is too big. However this is not a problem for all the implementations I've seen around the web. What's happening here?
The reason that your loss explodes is that your data is not normalized. As you increased the number of data-points, the magnitude of the input data becomes larger.
How can I fix it?
Normalized your data before feeding into the model:
x = (x - x.min()) / (x.max() - x.min())
y = (y - y.min()) / (y.max() - y.min())