Tensorflow evaluation frequency

Question

I am using the train_and_evaluate function in tensorflow and want to make the eval step happen more frequently (either by the global step or time elapsed). This is my code (model function is not shown).

def get_classifier(batch_size):
    config = tf.estimator.RunConfig(
        model_dir="models/shape_model_cnn_3",
        save_checkpoints_secs=300,
        save_summary_steps=100)

    params = tf.contrib.training.HParams(
        batch_size=batch_size,
        num_conv=[48,64,96], # Sizes of each convolutional layer
        conv_len=[2,3,4], # Kernel size of each convolutional layer
        num_nodes=128, # Number of LSTM nodes for each LSTM layer
        num_layers=3, # Number of LSTM layers
        num_classes=7, # Number of classes in final layer
        learning_rate=0.0001,
        gradient_clipping_norm=9.0,
        dropout=0.3)

    classifier = tf.estimator.Estimator(
        model_fn=my_model,
        config=config,
        params=params
    )

    return classifier

classifier = get_classifier(8)

train_spec = tf.estimator.TrainSpec(
    input_fn=lambda:input.batch_dataset("dataset/shape-train-???.tfrecords", tf.estimator.ModeKeys.TRAIN, 8),
    max_steps=100000
)

eval_spec = tf.estimator.EvalSpec(
    input_fn=lambda:input.batch_dataset("dataset/shape-eval-???.tfrecords", tf.estimator.ModeKeys.EVAL, 8)
)

tf.estimator.train_and_evaluate(classifier, train_spec, eval_spec)

I have tried using the start_delay_secs parameter in my EvalSpec, im not sure if this is what it is for but it doesn't seem to have any effect anyway

Answer 1

I have found that there is a parameter in EvalSpec, `throttle_secs' which starts the evaluation stage after a number of seconds. Alternatively if you want to evaluate based on a number of steps, you can use a for loop and incrementally increase the max_steps as suggested by @Kathy Wu.