Perceptron training from a stream with limited memory

Question

I want to train a Perceptron using stochastic gradient rulefrom the stream data. I have very limited amount of memory and i can store only N$examples.

Suppose my population consist of point as show in the following picture:

Now suppose my first N examples come in following fashion and i can classify them correctly as show in the next picture:

Now the problem is if the next N examples come in this way:

and i have classified them as shown. The problem is that since i can't train perceptron for previous N examples(because i can store only N examples and previous N examples need to be thrown away) and training on next N examples contradicts the hyper-plane for previous N examples.

How to train a perceptron from stream data? Do i need to store all examples or there is an alternative way?

Answer 1

You need to summarize your prior training somehow, and then incorporate that summary into your new training.

One simplistic way to do this is to express each category as k (cluster population) copies of the cluster's centroid. When you train on each new data set, include code to weight the centroid properly.

A similar approach is to alter your training algorithm to incorporate that weight directly into the way you alter the evaluation coefficients. "Learning rate" would be useful here.

You might also search on line for the work done in training with streaming data; I've given you only a simple version of the start-up approaches.