Saving neural network testing outputs in Pybrain

Question

I made a supervised neural network with pybrain, it works great and when I test it with "trainer.testOnData(test_data, verbose=True)" I can see the output (and the error) but I would also like to save it for further analysis. I coudn't find how on pybrain documentation. Does anyone that has worked with pybrain know how I can do it? Thank you (I hope this is not an obvious thing).

Answer 1

I have same problem as you, and to quickly answer question: no there is no straight forward way to do it.
But it is of course doable.

Mess with pybrain code

That seem like easiest solution, here you have source code of BackpropTrainer.testOnData. As you can see, it prints all errors if verbose is set to True.

    if verbose:
        print('All errors:', ponderatedErrors)
    assert sum(importances) > 0
    avgErr = sum(errors) / sum(importances)
    if verbose:
        print('Average error:', avgErr)
        print(('Max error:', max(ponderatedErrors), 'Median error:',
               sorted(ponderatedErrors)[len(errors) / 2]))
    return avgErr

We could make it return all errors along avgErr by changing last line to:

return avgErr, ponderatedErrors

Then you catch values simply unpacking result:

avgErr, allErrors = trainer.testOnData(dataSet, verbose=True)

or when you don't want all errors:

avgErr, _ = trainer.testOnData(dataSet, verbose=True)

That's simplest solution. But no everyone like to mess with external libraries source code.

Change stdout, catch it to a file and transform it

This is few step procedure, because testOnData never returns all errors, just prints it, it means that you have to transform string into something useful (lets try with list).

Change stdout to print into file

That's easy:

import sys
sys.stdout = open('./OURFILE', 'w+')

So now when we run testOnData output is save in file.

Work that string

We are intrested in second line of our file, so lets just get it:

our_file = open('./OURFILE', 'r')
our_file.next()                      # get rid of first line
our_line = our_file.next()           # save second line

Because how pybrain is written our line looks like this:

('All errors:', HERE_IS_LIST_OF_ERRORS)

Now, I'm not regex wizard so I'll just count when list starts.

still_string = our_line[16:-1]

It will give us string that includes only a list. And by now you cane use eval to change sting into proper list:

list_of_errors = eval(still_string)

From here, you cane use numpy or pandas to play with it.

I hope that helped.