Tensorflow feature_column expecting a different shape than input data

Question

I'm trying to implement a tensorflow Estimator, and getting a shape mismatch error I don't know how to debug. I think I may be misunderstanding how to specify the tf.feature_column's shape. My intention is to create a model with 6010 inputs. Any suggestions would be appreciated.

def train_input_fn():
    with np.load(TRAIN_NN_FEATURES) as train:
        train_features = train['features']                                                                   
        train_labels = train['labels']                                                       
    train_dataset = tf.data.Dataset.from_tensor_slices(
            ({'all_features': train_features}, train_labels))     
    train_iterator = train_dataset.make_one_shot_iterator()                                                  
    return train_iterator.get_next()     

all_features = tf.feature_column.numeric_column(
    'all_features', 
    shape=(6010,), 
    dtype=tf.float64
) 

estimator = tf.estimator.DNNClassifier( 
    feature_columns=[all_features],
    hidden_units=[1024, 512, 256]
)

estimator.train(input_fn=train_input_fn)

When I run this, I get the following error:

InvalidArgumentError (see above for traceback): Input to reshape 
is a tensor with 6010 values, but the requested shape has 36120100

[[Node: dnn/input_from_feature_columns/input_layer/all_features/Reshape = 
Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]
(dnn/input_from_feature_columns/input_layer/all_features/ToFloat, 
dnn/input_from_feature_columns/input_layer/all_features/Reshape/shape)]]

The shape of the data is as I expect, but the feature_column seems to be expecting its square.

>>> train_features.shape
(10737, 6010)
>>>train_labels.shape
(10737, 1)
>>> 36120100./6010
6010.0

My understanding is that Dataset.from_tensor_slices takes slices along axis 0 of the given tensor, which corresponds with the error message "Input to reshape is a tensor with 6010 values." But why is a shape with 36120100 values being requested?

Answer 1

I'd still like to know why the above wasn't working, or how to debug though.

The problem is with the tensor size produced from the train_iterator.get_next(). If the batch size is not specified, the iterator returns:

({'all_features': <tf.Tensor 'IteratorGetNext:0' shape=(6010,) dtype=float64>}, 
 <tf.Tensor 'IteratorGetNext:1' shape=(1,) dtype=float64>)

... tuple. As you can see the features tensor shape is (6010,), which DNNClassifier interprets as the batch_size=6010 (by convention the first dimension is the batch size), and it still expects 6010 features. Hence the error: it can't reshape (6010,) to (6010, 6010).

In order to make it working you have to either reshape this tensor manually, or simply set the batch size by calling:

train_dataset = train_dataset.batch(16)

Even batch size 1 will do fine, as it will force the get_next tensor to be:

({'all_features': <tf.Tensor 'IteratorGetNext:0' shape=(?, 6010) dtype=float64>}, 
 <tf.Tensor 'IteratorGetNext:1' shape=(?, 1) dtype=float64>)

... but you clearly want to set it bigger for efficiency.