Is there a good resource that clearly explains the difference between learning and memorisation of artificical neural networks - much better if the source contains mathematical explanations and/or proofs. I have looked online and through literature but most of them simply demonstrate so using examples, such as object detection or recognition. For example: this, this also this... the list is quite long. I find this insufficient. I know to some this question might sound a bit vague, but, does such a thing, as a threshold exist beyond which the system is simply memorising or has learned? Thank you very much.
I don't know about any source of proofs, but neural networks are in nature very good classifiers for being overfit/overtrained.if you use a more complex network than the one you need(and you don't usually know what is it of course) you will probably overtrain the model.regularization and dropout are used to prevent or at least decrease the overfitting of neural networks.
when your model overfits it means it has only found a pattern for the specific distribution of samples similar to the ones you trained your model with, hence loosing the ability to generalize for all unseen inputs. this behavior is called memorization, memorization can also happen in cases when your samples/inputs have a order in them but you don't want the model to learn the order, and neural network use that order to classify samples.(e.g. inputs be sorted from low to mid to high) . shuffling the data is a good solution for this matter.
I only know you can find out if the model has memorized the train set in this way: training accuracy is very high, maybe 100% but test accuracy is much lower, and this much doesn't have a threshold, it is empirical, for a specific problem you may be even ok with a 75% accuracy but for another one even 95% be low.