I am trying to write a lambda layer which converts an input tensor into a numpy array and performs a set of affine transforms on slices of said array. To get the underlying numpy array of the tensor I am calling K.eval(). Once I have done all of the processing on the numpy array, I need to convert it back into a keras tensor so it can be returned. Is there an operation in the keras backend which I can use to do this? Or should I be updating the original input tensor using a different backend function?
def apply_affine(x, y):
# Get dimensions of main tensor
dimens = K.int_shape(x)
# Get numpy array behind main tensor
filter_arr = K.eval(x)
if dimens[0] is not None:
# Go through batch...
for i in range(0, dimens[0]):
# Get the correpsonding affine transformation in the form of a numpy array
affine = K.eval(y)[i, :, :]
# Create an skimage affine transform from the numpy array
transform = AffineTransform(matrix=affine)
# Loop through each filter output from the previous layer of the CNN
for j in range(0, dims[1]):
# Warp each filter output according to the corresponding affine transform
warp(filter_arr[i, j, :, :], transform)
# Need to convert filter array back to a keras tensor HERE before return
return None
transformed_twin = Lambda(function=lambda x: apply_affine(x[0], x[1]))([twin1, transformInput])
EDIT: Added some context...
AffineTransform: https://github.com/scikit-image/scikit-image/blob/master/skimage/transform/_geometric.py#L715
warp: https://github.com/scikit-image/scikit-image/blob/master/skimage/transform/_warps.py#L601
I am trying to re-implement the CNN in "Unsupervised learning of object landmarks by factorized spatial embeddings". filter_arr is the output from a convolutional layer containing 10 filters. I want to apply the same affine transform to all of the filter outputs. There is an affine transform associated with each data input. The affine transforms for each data input are passed to the neural net as a tensor and are passed to the lambda layer as the second input transformInput. I have left the structure of my current network below.
twin = Sequential()
twin.add(Conv2D(20, (3, 3), activation=None, input_shape=(28, 28, 1)))
# print(twin.output_shape)
# twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
twin.add(Activation('relu'))
twin.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))
# print(twin.output_shape)
twin.add(Conv2D(48, (3, 3), activation=None))
# print(twin.output_shape)
twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
twin.add(Activation('relu'))
twin.add(Conv2D(64, (3, 3), activation=None))
twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
twin.add(Activation('relu'))
# print(twin.output_shape)
twin.add(Conv2D(80, (3, 3), activation=None))
twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
twin.add(Activation('relu'))
# print(twin.output_shape)
twin.add(Conv2D(256, (3, 3), activation=None))
twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
twin.add(Activation('relu'))
# print(twin.output_shape)
twin.add(Conv2D(no_filters, (3, 3), activation=None))
twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
twin.add(Activation('relu'))
# print(twin.output_shape)
# Reshape the image outputs to a 1D list so softmax can be used on them
finalDims = twin.layers[-1].output_shape
twin.add(Reshape((finalDims[1], finalDims[2]*finalDims[3])))
twin.add(Activation('softmax'))
twin.add(Reshape(finalDims[1:]))
originalInput = Input(shape=(28, 28, 1))
warpedInput = Input(shape=(28, 28, 1))
transformInput = Input(shape=(3, 3))
twin1 = twin(originalInput)
def apply_affine(x, y):
# Get dimensions of main tensor
dimens = K.int_shape(x)
# Get numpy array behind main tensor
filter_arr = K.eval(x)
if dimens[0] is not None:
# Go through batch...
for i in range(0, dimens[0]):
# Get the correpsonding affine transformation in the form of a numpy array
affine = K.eval(y)[i, :, :]
# Create an skimage affine transform from the numpy array
transform = AffineTransform(matrix=affine)
# Loop through each filter output from the previous layer of the CNN
for j in range(0, dims[1]):
# Warp each filter output according to the corresponding affine transform
warp(filter_arr[i, j, :, :], transform)
# Need to convert filter array back to a keras tensor
return None
transformed_twin = Lambda(function=lambda x: apply_affine(x[0], x[1]))([twin1, transformInput])
twin2 = twin(warpedInput)
siamese = Model([originalInput, warpedInput, transformInput], [transformed_twin, twin2])
EDIT: Traceback when using K.variable()
Traceback (most recent call last):
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
return fn(*args)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
status, run_metadata)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
[[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
return fn(*args)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
status, run_metadata)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
[[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/nickb/PycharmProjects/testing/MNIST_implementation.py", line 96, in <module>
transformed_twin = Lambda(function=lambda x: apply_affine(x[0], x[1]))([twin1, transformInput])
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\engine\topology.py", line 585, in __call__
output = self.call(inputs, **kwargs)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\layers\core.py", line 659, in call
return self.function(inputs, **arguments)
File "C:/Users/nickb/PycharmProjects/testing/MNIST_implementation.py", line 96, in <lambda>
transformed_twin = Lambda(function=lambda x: apply_affine(x[0], x[1]))([twin1, transformInput])
File "C:/Users/nickb/PycharmProjects/testing/MNIST_implementation.py", line 81, in apply_affine
filter_arr = K.eval(x)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\backend\tensorflow_backend.py", line 533, in eval
return to_dense(x).eval(session=get_session())
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 569, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 3741, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
run_metadata_ptr)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
[[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op 'batch_normalization_1/keras_learning_phase', defined at:
File "C:/Users/nickb/PycharmProjects/testing/MNIST_implementation.py", line 36, in <module>
twin.add(BatchNormalization(axis=1, momentum=0.99, epsilon=0.001, center=True))
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\models.py", line 466, in add
output_tensor = layer(self.outputs[0])
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\engine\topology.py", line 585, in __call__
output = self.call(inputs, **kwargs)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\layers\normalization.py", line 190, in call
training=training)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\backend\tensorflow_backend.py", line 2559, in in_train_phase
training = learning_phase()
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\keras\backend\tensorflow_backend.py", line 112, in learning_phase
name='keras_learning_phase')
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1507, in placeholder
name=name)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1997, in _placeholder
name=name)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
[[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x0000023AB66D9C88>>
Traceback (most recent call last):
File "C:\Users\nickb\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\client\session.py", line 587, in __del__
AttributeError: 'NoneType' object has no attribute 'TF_NewStatus'
Process finished with exit code 1
As stated in the comments above it is best to implement lambda layer functions using the Keras backend. Since there are currently no functions in the Keras backend that perform affine transformations, I decided to use a tensorflow function in my Lambda layer instead of implementing an affine transform function from scratch using existing Keras backend functions:
def apply_affine(x):
import tensorflow as tf
return tf.contrib.image.transform(x[0], x[1])
def apply_affine_output_shape(input_shapes):
return input_shapes[0]
The downside to this approach is that my lambda layer will only work when using Tensorflow as the backend (as opposed to Theano or CNTK). If you wanted an implementation that is compatible with any backend you could check the current backend being used by Keras and then perform the transformation function from the backend currently in use.