Search code examples
kerasfeature-extraction

How to extract feature vector for image when using CNN in Keras


I am doing a binary classification problem, my model architecture is as follow

def CNN_model(height, width, depth):
    input_shape = (height, width, depth)

    model = Sequential()
    # Block 1
    model.add(Conv2D(64, kernel_size=(3, 3), strides=1, activation='relu', input_shape=input_shape, padding='VALID'))
    model.add(Conv2D(64, kernel_size=(3, 3), strides=1, activation='relu', padding='VALID'))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    # Block 2
    model.add(Conv2D(128, kernel_size=(3, 3), strides=1, activation='relu', padding='VALID'))
    model.add(Conv2D(128, kernel_size=(3, 3), strides=1, activation='relu', padding='VALID'))
    model.add(AveragePooling2D(pool_size=(19, 19)))

    # set of FC => RELU layers
    model.add(Flatten())
    model.add(Dense(128))
    model.add(Activation('relu'))
    model.add(BatchNormalization())
    model.add(Dense(num_classes, activation='softmax'))
    model.compile(loss=keras.losses.binary_crossentropy,
                  optimizer=keras.optimizers.Adadelta(),
                  metrics=['accuracy'])
    return model

I need for each image on a test set, I get a 128-D feature vector collected from FC layer use for SVM classification. More detail, from model.add(Dense(128)). Can you please show me how to solve this problem? Thank you!


Solution

  • Here the simplest way is to remove the Dense layer.

    I will answer with a counter example with similar layers but different input_shape:

    from keras.layers import *
    from keras.preprocessing import image
    from keras.applications.vgg16 import VGG16
    from keras.applications.vgg16 import preprocess_input
    import numpy as np
    from scipy.misc import imsave
    import  numpy  as  np
    from keras.layers import *
    from keras.applications.vgg16 import VGG16
    from keras.applications.vgg16 import preprocess_input
    from keras.layers import Dropout, Flatten, Dense
    from keras.applications import ResNet50
    from keras.models import Model, Sequential
    from keras.layers import Dense, GlobalAveragePooling2D
    from keras import backend as K
    import matplotlib.pyplot as plt
    from keras.applications.resnet50 import preprocess_input
    
    model = Sequential()
    model.add(Conv2D(64, kernel_size=(3, 3), input_shape=(530, 700, 3), padding='VALID'))
    model.add(Conv2D(64, kernel_size=(3, 3), padding='VALID'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    
    # Block 2
    model.add(Conv2D(128, kernel_size=(3, 3), strides=1, activation='relu', padding='VALID'))
    model.add(Conv2D(128, kernel_size=(3, 3), strides=1, activation='relu', padding='VALID'))
    model.add(AveragePooling2D(pool_size=(19, 19)))
    
    # set of FC => RELU layers
    model.add(Flatten())
    
    #getting the summary of the model (architecture)
    model.summary()
    
    img_path = '/home/sb0709/Desktop/dqn/DQN/data/data/2016_11_01-2017_11_01.png'
    img = image.load_img(img_path, target_size=(530, 700))
    img_data = image.img_to_array(img)
    img_data = np.expand_dims(img_data, axis=0)
    img_data = preprocess_input(img_data)
    
    vgg_feature = model.predict(img_data)
    #print the shape of the output (so from your architecture is clear will be (1, 128))
    #print shape
    print(vgg_feature.shape)
    
    #print the numpy array output flatten layer
    print(vgg_feature.shape)
    

    Here is the output model architecture with all layers: model summary

    Also here is listed the feature vector: feature vector size of (1,128) (numpy array)

    Image used in the example:

    Image for the example

    Second method is for when using Functional Api instead of Sequencial() to use How can I obtain the output of an intermediate layer?

    from keras import backend as K
    # with a Sequential model
    get_6rd_layer_output = K.function([model.layers[0].input],
                                      [model.layers[6].output])
    layer_output = get_6rd_layer_output([x])[0]
    
    #print shape
    print(layer_output.shape)
    
    #print the numpy array output flatten layer
    print(layer_output.shape)
    

    One more useful step is the visualization of the features, I bet a lot of people want to see what see the computer and will illustrate only the "Flatten" layer output(better said the network):

    def visualize_stock(img_data):
        plt.figure(1, figsize=(25, 25))
        stock = np.squeeze(img_data, axis=0)
        print(stock.shape)
        plt.imshow(stock)
    

    and the magic:

    visualize_stock(img_data)
    

    feature map Note: changed from input_shape=(530, 700, 3) from input_shape=(84, 84, 3) for better visualization for the public.

    P.S: Decided to post so anyone who has this type of question to benefit (struggled with same type of questions recently).