I'm searching for existing work on Neural Net architectures that grow based on need or complexity/variability of training data. Some architectures that I've found include self-organizing maps, and growing Neural gas. Are these the only one's out there?
What I'm searching for is best illustrated by a simple scenario; if the training data only has a few patterns, then the neural net would be 2-3 layers deep with a small set of nodes in each layer. If the training data was more convoluted, then we would see deeper networks.
Such work seems rare or absent in the AI literature. Is it because the performance is comparatively weak ? I'd appreciate any guidance.
An example of this is called neuro-evolution. What you could do is combine backprop with evolution to find the optimal structure for your dataset. Neataptic is one of the NN libraries which offers neuro-evolution. With some simple coding you could turn this into backprop + evolution.
The disadvantage of this is that it will require much more computation power as it requires a genetic algorithm to run an entire population. So using neuro-evolution does make the performance comparibly weak.
However, I think there are more techniques out there that disable certain nodes, and if there is no negative effect on the output, they will be removed. I'm not sure though.