What is the best way to implement a bias node to my python/numpy neural network?

Question

I built a numpy only neural network originally based on an online tutorial and have come to realise that I should have some kind of bias neuron. However I have really been struggling to figure out how to implement it into my code and would greatly appreciate some guidance.

import numpy as np

class NN():   
    def __init__(self, layers, type):
        """
        layers: a list of layers, eg:
              2 input neurons
              1 hidden layer of 3 neurons
              2 output neurons
              will look like [2,3,2]
        type: initialisation type, "random" or "uniform" distribution
        """

        self.p = 0.1

        self.layers = len(layers) - 1

        self.inputSize = layers[0]
        self.outputSize = layers[self.layers]

        self.layerSizes = layers[:-1] #input layer, hiddens, discard output layer

        self.inputs = np.zeros(self.inputSize, dtype=float)
        self.outputs = np.zeros(self.outputSize, dtype=float)

        self.L = {}

        if type == "random":
            for i in range(1,self.layers+1):
                if i < self.layers:
                    self.L[i] = (np.random.ranf(( self.layerSizes[i-1] , self.layerSizes[i] )).astype(np.float) - 0.5) * 2
                else:
                    self.L[i] = (np.random.ranf(( self.layerSizes[i-1] , self.outputSize )).astype(np.float) - 0.5)*2
        elif type == "uniform":            
            for i in range(1,self.layers+1):
                if i < self.layers:
                    self.L[i] = np.random.uniform( -1 , 1 , (self.layerSizes[i-1],self.layerSizes[i]) )
                else:
                    self.L[i] = np.random.uniform( -1 , 1 , (self.layerSizes[i-1],self.outputSize) )

        else:
            print("unknown initialization type")

    def updateS(self): #forward propogation Sigmoid
        for i in range(1,self.layers+1):
            if 1 == self.layers:  #dodgy no hidden layers fix
                self.z = np.dot(self.inputs, self.L[i])
                self.outputs = ( self.sigmoid(self.z) - 0.5)*2           
            elif i == 1:  #input layer
                self.z = np.dot(self.inputs, self.L[i])
                self.temp = self.sigmoid(self.z)
            elif i < self.layers: #hidden layers
                self.z = np.dot(self.temp, self.L[i])
                self.temp = self.sigmoid(self.z)
            else: #output layer
                self.z = np.dot(self.temp, self.L[i])
                self.outputs = ( self.sigmoid(self.z) - 0.5)*2

    def sigmoid(self, s):
        #activation funtion
        return 1/(1+np.exp(-s/self.p))

Answer 1

Bias is just a variable you add to each neuron as you progress through the neural network feed forward process. Thus the feed forward process from one neuron layer to the next will be the sum of all the weights multiplied by the previous neuron that feed into the next neuron, then the bias of that neuron will be added, or:

output = sum(weights * inputs) + bias

To put this into context, look at the picture below:

where:

X1: Input value 1.
X2: Input value 2.
B1n: Layer 1, neuron n bias.
H1: Hidden layer neuron 1.
H2: Hidden layer neuron 2.
a(…): activation function.
B2n: Layer 2, neuron n bias.
Y1: network output neuron 1.
Y2: network output neuron 2.
Y1out: network output 1.
Y2out: network output 2.
T1: Training output 1.
T2: Training output 2.

when working out H1, you would need to use the following formula:

H1 = (X1 * W1) + (X2 * W2) + B11    

Note that is is before the value for the neuron has been completely calculated via the activation function.

Therefore, im pretty sure that the bias would be entered within the feed forward function:

def updateS(self): #forward propogation Sigmoid
        for i in range(1,self.layers+1):
            if 1 == self.layers:  #dodgy no hidden layers fix
                self.z = np.dot(self.inputs, self.L[i])
                self.outputs = ( self.sigmoid(self.z) - 0.5)*2           
            elif i == 1:  #input layer
                self.z = np.dot(self.inputs, self.L[i])
                self.temp = self.sigmoid(self.z)
            elif i < self.layers: #hidden layers
                self.z = np.dot(self.temp, self.L[i])
                self.temp = self.sigmoid(self.z)
            else: #output layer
                self.z = np.dot(self.temp, self.L[i])
                self.outputs = ( self.sigmoid(self.z) - 0.5)*2

by adding a value to the end of the self.z values. I think these values can be anything you want as a bias simply moves the intercept of the linear equation