Adding numbers using Neural network + back propagation
I am trying to use neural networks to add numbers toegther but i can't seem to make the back propagation function working.
This is how the neural network looks like where W1 = x1, W2 = x2, W3 = y1, W4 = y2, W5 = z1 and W6 = z2 This is my code so far:
from random import randint,random ,uniform
import numpy as np
class Data:
data_dict= {}
def __init__(self,limit):
self.limit = limit
'''creates data but beware that the limit may not be the same as the size of the dictionary'''
def create_data(self):
for i in range(self.limit):
num1 = randint(0,100)
num2 = randint(0,100)
self.data_dict[(num1,num2)] = num1+num2
''' you compare the error with every test in the data set and find weights that minimise the error'''
class Neural:
def __init__(self,data):
self.x1 = uniform(-1,1)
self.x2 = uniform(-1,1)
self.y1 = uniform(-1,1)
self.y2 = uniform(-1,1)
self.z1 = uniform(-1,1)
self.z2 = uniform(-1,1)
self.data=data
def relu(self,number):
return max(0,number)
def sigmoid(self,number):
return 1/(1 + np.exp(-number))
'''weighted summation with activation function to compute output '''
def compute_output(self,num1,num2):
hidden_layer_input1 = self.sigmoid((num1 * self.x1) + (num2 * self.y1))
hidden_layer_input2 = self.sigmoid((num1 * self.x2) + (num2 * self.y2))
return ((hidden_layer_input1*self.z1) + (hidden_layer_input2 * self.z2))
'''mean swaured error error between the actual output with the output generated by the algorithm '''
def compare_ouput(self,data):
'''actually,better to find error between all tests. add all the errors up'''
error = 0
for key in data.data_dict:
error += abs(data.data_dict[key] - self.compute_output(key[0],key[1])) **2
return error/len(data.data_dict)
# return abs(actual - self.compute_output(num1,num2))
'''TODO function that changes the weight depending on the errors using gradient descent'''
'''first make it random'''
'''next perhaps change weights for each test and average out the adjustments for each weight'''
def random_back_propagation(self):
error = 100000
while error>0.1:
self.x1 = random()
self.x2 = random()
self.y1 = random()
self.y2 = random()
self.z1 = random()
self.z2 = random()
error = self.compare_ouput(self.data)
print(error)
print(self.compute_output(140,15))
'''learning rate is the amount the weights are updated during training'''
def back_propagation(self, learning_rate):
for _ in range(1000):
for key in self.data.data_dict:
num1, num2 = key
target = self.data.data_dict[key]
hidden_layer1_output = self.sigmoid((num1 * self.x1) + (num2 * self.y1))
hidden_layer2_output = self.sigmoid((num1 * self.x2) + (num2 * self.y2))
output = ((hidden_layer1_output * self.z1) + (hidden_layer2_output * self.z2))
error = target - output
#check if you are happpy with the error
output_unit1 = output * (1 - output) * (error)
hidden_unit1 = hidden_layer1_output * (1 - hidden_layer1_output) * self.z1 * output_unit1
hidden_unit2 = hidden_layer2_output * (1 - hidden_layer2_output) * self.z2 * output_unit1
self.z1 += (learning_rate * output * hidden_layer1_output)
self.z2 += (learning_rate * output * hidden_layer2_output)
self.x2 += (learning_rate * hidden_unit2 * num1)
self.x1 += (learning_rate * hidden_unit1 * num1)
self.y1 += (learning_rate * hidden_unit1 * num2)
self.y2 += (learning_rate * hidden_unit2 * num2)
print(self.x1,self.x2,self.y1,self.y2,self.z1,self.z2)
print(num1,num2,self.compute_output(num1,num2))
data = Data(200)
data.create_data()
neural = Neural(data)
neural.back_propagation(0.01)
#print(data.data_array)
#print(uniform(-1,1))
print(neural.compute_output(15,7))
I tried changing the learning rate, number of itterations, number of items in the data set but I'm not sure if the problem is with trying to find correct values to use or if my function is just incorrect
There are some problems with your current approach. I'll try to elaborate in the following sections.
Loss function
The loss function you are currently using is error = target - output. At first glance it seems to work, because if target and output are equal, error will be 0. But remember that the network will try to minimize the loss function, and in this case one of the ways to do it is by generating an extremely large output, so that error will be negative.
I suggest using something like an MSE (Mean Squared Error): error = (target - output) ** 2. That way, if the network wants to minimize it, it will have no choice but to try to make target and output equal.
Gradient updates
Currently you are updating your parameters with the positive gradient self.z1 += (learning_rate * gradient). The gradient points to the direction of greatest increase of the loss function. We don't want to increase the loss. We want to go in the direction of greatest decrease, so we use the negative gradient self.z1 -= (learning_rate * gradient).
The sigmoid function
You are using a sigmoid as the activation function for some of your hidden units. Here's the sigmoid graph:
Notice that when X > 6 or X < -6, the sigmoid basically flatlines. This flatline translates into an extremely small gradient, practically 0, and that means the parameters update extremely slowly or not at all. The network is saturated.
The data that you are feeding to the network consists of numbers that are larger than what neural networks normally work with. Consider that one of your hidden units had its weights randomly initialized to 0.5 and 0.3. Now you feed the network with the numbers you wish to sum: 50 and 20. The computation at the hidden unit will be sigmoid(50 * 0.5 + 20 * 0.3) which is sigmoid(31), and its derivative is effectively 0.
If you still wanna use the sigmoid, I recommend that you preprocess your input data so that the network won't receive such large numbers. For example, you can divide every input by 100, as the largest possible integer in the training data is 100.
Putting it all together
Below is the code with the above mentioned problems fixed. As it uses a different loss function, all of the gradients are different (feel free to double check as I'm not that confident in calculus). I also added a print to keep track of the loss across epochs.
from random import randint, random, uniform
import numpy as np
class Data:
data_dict = {}
def __init__(self, limit):
self.limit = limit
"""creates data but beware that the limit may not be the same as the size of the dictionary"""
def create_data(self):
for i in range(self.limit):
num1 = randint(0, 100)
num2 = randint(0, 100)
self.data_dict[(num1, num2)] = num1 + num2
""" you compare the error with every test in the data set and find weights that minimise the error"""
class Neural:
def __init__(self, data):
self.x1 = uniform(-1, 1)
self.x2 = uniform(-1, 1)
self.y1 = uniform(-1, 1)
self.y2 = uniform(-1, 1)
self.z1 = uniform(-1, 1)
self.z2 = uniform(-1, 1)
self.data = data
def relu(self, number):
return max(0, number)
def sigmoid(self, number):
return 1 / (1 + np.exp(-number))
"""weighted summation with activation function to compute output """
def compute_output(self, num1, num2):
num1, num2 = num1 / 100, num2 / 100
hidden_layer_input1 = self.sigmoid((num1 * self.x1) + (num2 * self.y1))
hidden_layer_input2 = self.sigmoid((num1 * self.x2) + (num2 * self.y2))
return 100 * ((hidden_layer_input1 * self.z1) + (hidden_layer_input2 * self.z2))
"""mean swaured error error between the actual output with the output generated by the algorithm """
def compare_ouput(self, data):
"""actually,better to find error between all tests. add all the errors up"""
error = 0
for key in data.data_dict:
error += abs(data.data_dict[key] - self.compute_output(key[0], key[1])) ** 2
return error / len(data.data_dict)
# return abs(actual - self.compute_output(num1,num2))
"""TODO function that changes the weight depending on the errors using gradient descent"""
"""first make it random"""
"""next perhaps change weights for each test and average out the adjustments for each weight"""
def random_back_propagation(self):
error = 100000
while error > 0.1:
self.x1 = random()
self.x2 = random()
self.y1 = random()
self.y2 = random()
self.z1 = random()
self.z2 = random()
error = self.compare_ouput(self.data)
print(error)
print(self.compute_output(140, 15))
"""learning rate is the amount the weights are updated during training"""
def back_propagation(self, learning_rate):
for epoch in range(1000):
errors = []
for key in self.data.data_dict:
num1, num2 = key
target = self.data.data_dict[key]
# Rescaling everything
num1, num2 = num1 / 100, num2 / 100
target = target / 100
hidden_layer1_output = self.sigmoid((num1 * self.x1) + (num2 * self.y1))
hidden_layer2_output = self.sigmoid((num1 * self.x2) + (num2 * self.y2))
output = (hidden_layer1_output * self.z1) + (hidden_layer2_output * self.z2)
# loss
error = (target - output) ** 2
# check if you are happpy with the error
# derivative of error with respect to output
# d(error)/d(output) = -2 * (target - output)
# derivative of error with respect to x1
# d(error)/d(x1) = d(error)/d(output) * d(output)/d(x1)
# /-> hidden_layer1_output
# d(error)/d(x1) = d(error)/d(output) * ( d(output)/d(hl1) * d(hl1)/d(x1) )
# derivative of output with respect to hl1
# d(output)/d(hl1) = z1
# derivative of hl1 with respect to x1
# d(hl1)/d(x1) = hidden_layer1_output * (1 - hidden_layer1_output) * num1
# d(error)/d(x1) = -2 * (target - output) * z1 * hidden_layer1_output * (1 - hidden_layer1_output) * num1
self.x1 -= learning_rate * -2 * (target - output) * self.z1 * hidden_layer1_output * (1 - hidden_layer1_output) * num1
self.y1 -= learning_rate * -2 * (target - output) * self.z1 * hidden_layer1_output * (1 - hidden_layer1_output) * num2
self.x2 -= learning_rate * -2 * (target - output) * self.z2 * hidden_layer2_output * (1 - hidden_layer2_output) * num1
self.y2 -= learning_rate * -2 * (target - output) * self.z2 * hidden_layer2_output * (1 - hidden_layer2_output) * num2
# derivative of error with respect to z1
# d(error)/d(z1) = d(error)/d(output) * d(output)/d(z1)
# derivative of output with respect to z1
# d(output)/d(z1) = hidden_layer1_output
# d(error)/d(z1) = -2(target - output) * hidden_layer1_output
self.z1 -= learning_rate * -2 * (target - output) * hidden_layer1_output
self.z2 -= learning_rate * -2 * (target - output) * hidden_layer2_output
# print(self.x1, self.x2, self.y1, self.y2, self.z1, self.z2)
errors.append(error)
print(f"Mean error: {np.mean(errors)}")
data = Data(2000)
data.create_data()
neural = Neural(data)
neural.back_propagation(0.1)
print("#################################PREDICTIONS############################################")
print(f"15 + 7 = {neural.compute_output(15, 7)}")
print(f"3 + 2 = {neural.compute_output(3, 2)}")
print(f"50 + 70 = {neural.compute_output(50, 70)}")
- matplotlib 3D scatter plot alpha varies when viewing different angles
- How to write very long string that conforms with PEP8 and prevent E501
- Getting Home Directory with pathlib
- how to avoid bot detection on websites using selenium python
- Python mock to create a fake object return a dictionary when any of its attributes are used
- Polars vs. Pandas: size and speed difference
- How to mock.patch a class imported in another module
- Python - error cannot determine truth value of Relational (Newton-Raphson)
- ProcessPoolExecutor logging fails to log inside function on Windows but not on Unix / Mac
- SQLAlchemy ORM Insert or Update when importing from JSON
- django managers vs proxy models
- Pytroch clamp for complex values
- For every identifier select only rows with largest order column
- truth value for Expr is ambiguous in with_columns ternary expansion on dates
- Remove equal characters from two python strings
- Python pyad module can't set UPN
- Macro VS Micro VS Weighted VS Samples F1 Score
- Printing a Tree data structure in Python
- How to fix/reset decreasing timestamps while preserving gaps in time-series data for CNN training?
- Test that module is NOT imported
- Pyserial module isn't installed on PATH
- Print a multiplication table in Python
- Python: ModuleNotFoundError: No module named 'xyz'
- Receiving Import Error: No Module named ***, but has __init__.py
- PyQt5 QProgressBar border radius issue
- URL-encoding and -decoding a string in Python
- Fastest way to find the smallest possible sum of the absolute differences of pairs within a single array?
- Flask: Update Code Reference for: current_app._get_current_object()
- Export Charts from Excel as images using Python
- Align yaxis label spanning two axes with yaxis labels of one axes in subplots