How is bias added to specific hidden layers of a Neural Network in PyBrain?

Question

I have a Neural Network with two hidden layers. I want to add a bias unit only to the second hidden layer. How do I do that?

The code for my network is as follows:

nn = FeedForwardNetwork()
inLayer = LinearLayer(numFeatures)
hiddenLayer1 = LinearLayer(numFeatures+1)
hiddenLayer2 = SigmoidLayer(numFeatures+1)
outLayer = LinearLayer(1)

nn.addInputModule(inLayer)
nn.addModule(hiddenLayer1)
nn.addModule(hiddenLayer2)
nn.addOutputModule(outLayer)

in_to_hidden1 = FullConnection(inLayer, hiddenLayer1)
hidden1_to_hidden2 = FullConnection(hiddenLayer1, hiddenLayer2)
hidden2_to_out = FullConnection(hiddenLayer2, outLayer)

nn.addConnection(in_to_hidden1)
nn.addConnection(hidden1_to_hidden2)
nn.addConnection(hidden2_to_out)
nn.sortModules()

Answer 1

This is fairly simple task. First you have to create Bias module:

bias = BiasUnit()

Then add it to your NeuralNetwork, so:

nn = FeedForwadNetwork()
nn.addModule(bias)

Then, assuming you already added other layers you have to connect bias to hidden layer of your choice:

bias_to_hiden = FullConnection(bias, hiden_layer)

And then add it to neural network:

nn.addConnection(bias_to_hiden)

Apart from that you do everything same as before.

For the reference check code of the buildNetwork function from pybrain.tools.shortcuts module. Here is some code that connects bias unit to other layers (lines 75-79):

if opt['bias']:
    # add bias module and connection to out module, if desired
    n.addModule(BiasUnit(name='bias'))
    if opt['outputbias']:
        n.addConnection(FullConnection(n['bias'], n['out']))

Hope that helped.