output layer regularization implementation

Question

I’m building a NN model using keras, and I wish to impose a constraint on it that doesn’t (directly) have to do with the weights. Would be very grateful for some help / points me towards some relevant keywords to look up. The constraint I wish to impose is a bit complex, but it can be simplified in the following manner: I wish to impose a constraint on the output of certain inputs of the net. For the sake of simplicity, let’s say the constraint looks like NN(3)+NN(4) < 10, where NN is the neural net, which can be seen as a function. How can I impose such a constraint? Thank you very much in advance for any help on the subject!

edit: A more detailed explanation of what I'm trying to do and why. The theoretical model I'm building is this: I'm feeding the output of the first net into the input of the second net, along with an additive gaussian noise. The constraint I wish to impose is on the output of the first NN (g). Why? Without a constraint, the net maps the inputs to outputs as high as it possibly can in order to make the additive noise as insignificant as possible. And rightly so, this is the optimal encoding function g, but it's not very interesting :) And so I wish to impose a constraint on the output of the first NN (g). More specifically, the constraint is on the total power of the function: integral{ fX(x) * g(x)^2 dx }. But this can be simplified more or less, to a function that looks something like what I described earlier - g(3)+g(4)<10. More specifically, the function is sum { fX(x) * g(i)^2 * dx } < max_power, for some sampled inputs i. This is the problem, now here's how I attempted to implement it:

model = Sequential([
        Dense(300, input_dim=1, activation='relu'),
        Dense(300, activation='relu'),
        Dense(1, activation='linear', name=encoder_output),
        GaussianNoise(nvar, name='noise'),
        Dense(300, activation='relu', name=decoder_input),
        Dense(300, activation='relu'),
        Dense(1, activation='linear', name=decoder_output),
    ])

Mainly, this is supposedly a single neural net, and not really 2 (although there is no difference obviously). The import things to note is the input dim 1, output dim 1 (x and y in the diagram), and the gaussian noise in the middle. The hidden layers are not very interesting right now, I'll optimize them at a later point. In this model, I wish to impose a constraint on the output of a (supposedly) hidden layer named encoder_output. Hope this clarifies things.

Answer 1

You could use a multi input/multi output model with shared weights layers. The model could for example look like this:

from keras.layers import Input, Dense, Add
from keras.models import Model

# Shared weights layers
hidden = Dense(10, activation='relu')
nn_output = Dense(1, activation='relu')

x1 = Input(shape=(1,))
h1 = hidden(x1)
y1 = nn_output(h1)

x2 = Input(shape=(1,))
h2 = hidden(x2)
y2 = nn_output(h2)

# Your constraint
# In case it should be more complicated, you can implement
# a custom keras layer
sum = Add()([y1, y2]) 

model = Model(inputs=[x1, x2], outputs=[y1, y2, sum])
model.compile(optimizer='sgd', loss='mse')

X_train_1 = [3,4]
X_train_2 = [4,3]
y_train_1 = [123,456] # your expected output
y_train_2 = [456,123] # your expected output
s = [10,10] # expected sums

model.fit([X_train_1, X_train_2], [y_train_1, y_train_2, s], epochs=10)

If you have no exact value for your constraint that can be used as an expected output, you can remove it from the outputs and write a simple custom regularizer that would be used on it. There is a simple example for a custom regularizer in the Keras documentation.