Using Sigmoid instead of Tanh activation function fails - Neural Networks

Question

I'm looking at the following code from this blog

It gives the option to use both the sigmoid and the tanh activation function.

The XOR test seems to work fine with the tanh function yielding ~ (0,1,1,0)

But upon changing to sigmoid I get the wrong output ~ (0.5,0.5,0.5,0.5)

I've tried this with another piece of code I found online and the exact same problem occurs.

It seems the only thing changing is the activation function (and its derivative). Does changing this require other changes, say in backpropogation?

Thanks a lot for any help!

Answer 1

Looks like the model you use doesn't train biases. The only difference between tanh and sigmoid is scaling and offset. Learning the new scaling will be done through the weights, but you'll also need to learn to compensate for the new offset, which should be done by learning the biases as well.