I have a bug in my code, but finding the exact cause of it is difficult because of how theano works.
Following the tips in the exception details, I set theano.config.optimizer='None' and theano.config.exception_verbosity='high', but that doesn't tell me enough.
In my case, for example, there is a problem with the dot product between two tensors. The stacktrace leads me through a lot and to a particular function which seems to contain in it, somewhere, the problematic call to theano.tensor.dot, but I can't find where exactly that part of the code is, and since I'm trying to implement things through keras, it gets even more complicated and tangled up.
Is there any way to get more details on an apply node? I've tried using StepMode, as it seems to be attached to the nodes, but if there is a way of making that tool print the exact lines from which the code in the node is executed, I don't know what it is. I tried using that to print a stacktrace when the problem occurs, but it prints just about the same stacktrace as the exception.
If you want to find the spots in your code that use theano.tensor.dot you can monkeypatch it with wrapper code that uses traceback.print_stack:
import traceback
original_dot = theano.tensor.dot
def debug_wrapper(*args,**kw):
traceback.print_stack()
return original_dot(*args,**kw)
theano.tensor.dot = debug_wrapper
This way any time theano.tensor.dot is called (after it is patched) it will show you the stack like the one in a traceback message and still do it's job. Note that I am not very familiar with theano so this is a general python debugging solution, there might well be ways specific to theano that let you do similar.