tf.contrib.learn.LinearRegressor builds unexpectedly bad model for a data with one feature

Question

I am building a simple linear regressor for the data from the csv. Data includes weight and height values of some people. Overall learning process is very simple:

MAX_STEPS = 2000
# ...
features = [tf.contrib.layers.real_valued_column(feature_name) for feature_name in FEATURES_COL]
# ...
linear_regressor = tf.contrib.learn.LinearRegressor(feature_columns=features)
linear_regressor.fit(input_fn=prepare_input, max_steps=MAX_STEPS)

However, the model that is built by the regressor is, unexpectedly, bad. Result could be illustrated with the next picture:

Visualization code(just in case):

plt.plot(height_and_weight_df_filtered[WEIGHT_COL], 
         linear_regressor.predict(input_fn=prepare_full_input), 
         color='blue',
         linewidth=3)

Here is the same data been given to the LinearRegression class from the scikit-learn:

lr_updated = linear_model.LinearRegression()
lr_updated.fit(weight_filtered_reshaped, height_filtered)

And the visualization:

Increasing amount of steps has no effect. I would assume I'm using regressor from the TensorFlow in a wrong way.

iPython notebook with the code.

Answer 1

It looks like your TF model does indeed work and will get there with enough steps. You need to jack it right up though - 200K showed significant improvement, almost as good as the sklearn default.

I think there are two issues:

1. sklearn looks like it simply solves the equation using ordinary least squares. TF's LinearRegressor uses the FtrlOptimizer. The paper indicates it is a better choice for very large datasets.
2. The input_fn to the model is injecting the whole training set at once, for every step. This is just a hunch, but I suspect that the FtrlOptimizer may do better if it sees batches at a time.

Instead of just changing the number of steps up a couple orders of magnitude, you can also jack the learning rate up on the optimizer (the default is 0.2) and get similarly good results from only 4k steps:

linear_regressor = tf.contrib.learn.LinearRegressor(
    feature_columns=features, 
    optimizer=tf.train.FtrlOptimizer(learning_rate=5.0))