Search code examples
machine-learningthresholdbackpropagation

Thresholds in backpropagation


What is the use of thresholds in backpropagation algorithm. I wrote a java code for class label identification. I used some random thresholds (0-1) for the neurons. I trained the system and tested using some data.It worked pretty good. But what difference the algorithm makes with or without using Thresholds.


Solution

  • What you call "thresholds" are actually biases in affine transformations denoted by neurons

    f(w,x,b) = g(<w,x>+b)
    

    Biases should not be used as a constants as you suggest but rather - trained just as any other parameter in the network. Usually one simply adds a hypothethical "bias neuron" always equal to 1, and so bias becomes just another weight

    f(w,x,b) = g(<[w b], [x 1]>)
    

    Why is it important to have biases? In general having no bias means, that "filters" (feature detectors) trained in your neurons have to go through the origin. You can think about each neuron as a hyperplane in your input space around which you "fold" the space so your data becomes more separable. If you do not have biases - all these hyperplanes close at the origin. If you fix them as constants - you fix their distances from the origin. Finally if you train them - algorithm can freely place them in the input space (desired behaviour).