Neural networking noob here.
I am using PyBrain to try and create a network that learns music.
My dataset consists of about a hundred songs where the input for the network is two notes and the target is the next two notes.
Each note is represented by an int for the note/chord combination, an int for octave of the note and a float for the duration of the note.
My network looks like this:
net = RecurrentNetwork()
net.addInputModule(LinearLayer(6, name='in'))
net.addModule(LSTMLayer(50, name='hidden1'))
net.addModule(LSTMLayer(50, name='hidden2'))
net.addOutputModule(LinearLayer(6, name='out'))
net.addConnection(FullConnection(net['in'], net['hidden1'], name='c1'))
net.addConnection(FullConnection(net['hidden1'], net['hidden2'], name='c3'))
net.addRecurrentConnection(FullConnection(net['hidden2'], net['hidden1'], name='c4'))
net.addConnection(FullConnection(net['hidden2'], net['out'], name='c5'))
net.sortModules()
With a dataset and trainer like so:
ds = SupervisedDataSet(6, 6)
trainer = BackpropTrainer(net, ds, verbose = True, momentum = 0.01)
My problem is, when I train the network, I get a huge error back (E.G. 24569847209.8) which never seems to go down, it changes with each epoch but it always hovers around the same number.
After the network is trained it creates a song by taking two random notes as input, then generating the target, then passing the target back as an input, and repeating this over and over until it has a full song. But I find all it ever does is just write output over and over again, like it just learns one fixed target.
I'm really not sure what's wrong with what I have that is causing this.If there is some information I should include please let me know.
I seem to have fixed the learning problem! My network error is now steadily decreasing as I train it! I was using the wrong trainer. As I am using a recurrent neural network, I should not have been using the back propagation trainer. I am now using the RPropMinusTrainer. To fix, the following line:
trainer = BackpropTrainer(net, ds, verbose = True, momentum = 0.01)
Was changed to
trainer = RPropMinusTrainer(net, dataset=ds, verbose = True)