Regularization in Keras: How can I control the maximum number of zero weights on the last layer?

Question

I have a neural network and the last layer outputs a vector of size N (N=8). As I am working on a multilabel classification, I have identified that most of the output vector elements are equal to zero, with at most two elements equal to 1. For instance y_pred == [1, 0, 0, 0, 0, 0, 0, 1].

I would like to tell that to my network, i.e. say that at least N-2 output weights are equal to 0.

My current model is as follows:

ResNet18, preprocess_input = Classifiers.get('resnet18')
resnet = ResNet18((im_size, im_size, 3), weights='imagenet', include_top=False)
headModel = keras.layers.pooling.AveragePooling2D(pool_size=(3,3))(resnet.output)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
    
# 'sigmoid' parameter indicating that we’ll be performing multi-label classification.
headModel = Dense(8, activation="sigmoid")(headModel)

I am looking into adding a regularizer my_reg to my last Dense layer such that is resembles to something like

headModel = Dense(8, activation="sigmoid", kernel_regularizer=my_reg)(headModel)

I have no experience with regularizers in Keras and how to manipulate the weights.

Answer 1

You can make a custom function as your activation function. More specifically, set the two smallest probabilities to zero.

def custom_func(x):
    second_smallest = tf.sort(tf.squeeze(x))[1]
    x = tf.where(second_smallest >= x, tf.zeros_like(x), x)
    return x

import numpy as np
import tensorflow as tf

inp = tf.keras.Input(shape=(224, 224, 3))
base = tf.keras.applications.MobileNetV2(include_top=False, 
                                         input_shape=(224, 224, 3))(inp)
gap = tf.keras.layers.GlobalAveragePooling2D()(base)
out = tf.keras.layers.Dense(8, activation='sigmoid')(gap)
custom_function = tf.keras.layers.Lambda(custom_func)(out)

model = tf.keras.Model(inp, custom_function)

model(np.random.rand(1, 224, 224, 3).astype(np.float32))

<tf.Tensor: shape=(1, 8), dtype=float32, numpy=
array([[0.36225533, 0.66996753, 0.9467776 , 0.        , 0.6429986 ,
        0.9498544 , 0.        , 0.6883256 ]], dtype=float32)>

You can also have it accept an argument as such:

import numpy as np
import tensorflow as tf


def custom_func(inputs, n_to_zero):
    second_smallest = tf.sort(tf.squeeze(inputs))[n_to_zero - 1]
    out = tf.where(second_smallest >= inputs, tf.zeros_like(inputs), inputs)
    return out


inp = tf.keras.Input(shape=(224, 224, 3))
base = tf.keras.applications.MobileNetV2(include_top=False, 
                                         input_shape=(224, 224, 3))(inp)
gap = tf.keras.layers.GlobalAveragePooling2D()(base)
out = tf.keras.layers.Dense(8, activation='sigmoid')(gap)
custom_function = tf.keras.layers.Lambda(
    lambda x: custom_func(inputs=x, n_to_zero=4)
                                        )(out)

model = tf.keras.Model(inp, custom_function)

model(np.random.rand(1, 224, 224, 3).astype(np.float32))

<tf.Tensor: shape=(1, 8), dtype=float32, numpy=
array([[0.8537902, 0.       , 0.       , 0.       , 0.7386258, 0.       ,
        0.0948523, 0.7973974]], dtype=float32)>