R: Tensorflow warning when building model inside a loop "please define your @tf.function outside of the loop"

Question

I am using LSTM with keras and tensorflow for R on a time series problem. Therefore I use a rolling window to validate and test the model:

Input; Output

[1:10] -> 11

[2:11] -> 12

Where my model is trained on observations 1:10 to predict 11, than trained on 2:11 to predict 12 and so on. I do this with a loop. In each iteration a new model is created.

But doing so leads to the following warning message:

"WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_predict_function..predict_function at 0x0000023BC04C6680> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details."

I know for sure that (1) is responsible for the warning message as I am indeed using a loop. My question is: Can I ignore this warning, (not sure what it even means) and/or can I work around it?

The suggested solution does not make sense to me, because I need the model inside the loop. If I build it outside, it would not be a "unique" model for each point in time.

The following is NOT my exact code but maybe it helps to have some idea.

Epochs = 100

# THIS is the "problematic" loop
for(i in 10:1){

# irrelevant: some data transformation that uses i to ensure a rolling window
x_train <- data[1+10-i,] # and y_train, x_test similarly defined


##############################################################
# Warning comes from inside this part because its inside the loop (as it should) #

model <- keras_model_sequential() %>% 
    layer_lstm(units = units,
               input_shape = c(lag, features)) %>%
    layer_dense(units = 1)
 
  model %>% compile(loss = "mean_squared_error",
                    optimizer = "adam",
                    metrics = "accuracy")
  
  for(j in 1:Epochs){
    model %>% fit(
      x_train,
      y_train,
      epochs = 1,
      batch_size = batch,
      shuffle = F
    )
    model %>% reset_states()
  }
#########################################################################

# irrelevant: some sort of saving results
result[i] <- model %>% predict(x_test, batch_size = batch)

}

I came across this question regarding my issue. The first comment suggests:

"...I had placed the load command in a loop together with the model.predict command, model = load_model("name"); ... Y = model.predict(X); After I moved the loading of the model out of the loop the warning dissapeared."

But as I said prior, it does not make sense to me, to build the model outside the loop. So can I ignore the warning?

Answer 1

The warning is safe to ignore in this instance, since you are intentionally creating a new model from scratch each loop iteration. The warning exists to help users avoid inadvertently retracing a function many times, when they mean to trace it just once and then call the traced (meaning, compiled and optimized) function repeatedly in a loop.