I would like to create my own operations to merge networks. So I've taken a look to the code, and I modified engine/topology.py to create my new operation.
I didn't modified layers/wrappers.py because it's only for RNN and when I modify it I get an error.
Are there other files/classes to modify? Don't I have to do something else somewhere else to specify what to do during the backward pass?
You don't have to change any other files if you have implemented your operation properly with backend operations only. The backend is clever and takes care of the computation of the gradients for the backpropagation by itself.
This means that all parameters that will change over time have to be defined with K.variable, and you only use mathematical operations defined in keras.backend. Otherwise the backend will not be able to perform the backpropagation properly.
Side-note: Instead of modifying the source code of keras, you could implement your own class that extends the Merge class and override the call function for your custom operation.