I've been using keras functional API to build me a nice net. However, i don't understand how spatial connectivity in tf.keras.layers.Dense works.
If I flatten a 7x7x1024 volume i get 50,176 parameters. I expect total number of parameters between two layers to be
50,176 * 4096 + 4096 = 205,524,992
And it is .
And suprisingly, when i remove layer Flatten(), I don't get any dimension incompatibility errors. Output shape is 7x7x4096, and the number of parameters is:
1024*4096 + 4096 = 4,198,400
If this is correct, why does tf.keras.layers.Dense only have dense connections between last dimensions of layers and why is the output a 7x7x4096 volume ?
(last layer is 7 x 7 x 1024 volume)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Dense(4096)(x)
When you pass a tensor with dim>2 Dense create connection with the last dimension as a default behaviour [1] (line 889, input_dim = input_shape[-1]), that's why you don't get any error. And as a result you also get the number of parameters you've already calculated. So if you're using 3D inputs, you need to flatten it before to pass it to a Dense layer.
[1] https://github.com/keras-team/keras/blob/master/keras/layers/core.py#L796