Search code examples
algorithmmachine-learningneural-networkartificial-intelligence

Is a multilayer neural net of 2 neurons just the same like 1 neuron

AND logic and OR logic can be solved by just 1 neuron. However, XOR logic needs a neural network of 3 neurons in 2 layers:

(neuron1)\
          \
           +----- (neuron3)
          /
(neuron2)/

Consider this form of neural network:

(neuron1) ------- (neuron2)

Is this kind of neural network with just 2 neurons connecting to each other able to solve anything better than just 1 single neuron?

Solution

  • 2 neurons can be more powerful than 1 neuron.

    For example consider two neurons with the standard rectifier nonlinearity max(0,x).

    Let the input be x.

    The first neuron computes y=max(0,x)

    The second neuron computes z=max(0,1-y)

    The picture plots y (green),1-y (red),z (blue) against x.

    enter image description here

    This shows how two neurons both using a simple rectifier can construct a more complicated saturating nonlinearity (represented by the blue line).

    There is no way to produce the blue line with a single rectifying neuron operating on a single output (because all such outputs have at most two linear segments, and our z output has three linear segments).