Are there alternatives to backpropagation?

Question

I know a neural network can be trained using gradient descent and I understand how it works.

Recently, I stumbled upon other training algorithms: conjugate gradient and quasi-Newton algorithms. I tried to understand how they work but the only good intuition I could get is that they use higher order derivative.

Are those alternative algorithms I mentioned fundamentally different from a backpropagation process where weights are adjusted by using the gradient of the loss function?

If not, is there an algorithm to train a neural network that is fundamentally different from the mechanism of backpropagation?

Answer 1

Conjugate gradient and quasi-Newton algorithms are still gradient descent algorithms. Backpropagation (or backprop) is nothing more than a fancy name to a gradient computation.

However, the original question of alternatives to backprop is very important. One of the recent alternatives, for example, is equilibrium propagation (or shortly eqprop).

Edit 28/02/2024: Since the question is quite general ("an algorithm to train a neural network"), here are a few more techniques to include:

• Reservoir computing (2000s) - train only the linear readout layer using e.g. ridge regression.
• Evolutionary computation (the idea originated back in 1950s) - inspired by biological evolution. Can work quite well to train small neural networks. One of the answers below mentioned NEAT.
• Direct feedback alignment (2016) - introducing asymmetry between forward and backward passes.
• Mentioned above equilibrium propagation (2016).
• Hinton's forward-forward algorithm (2022).
• And other methods which are being developed. The motivation why we look for backprop alternatives, among other reasons, is to address the question How does our own brain learn? So far, we believe the brain does not do backprop.