I have a time-series based ML pipeline. The algorithm used in it is Structural Time Series(STS) from Tensorflow. We retrain it every day from start data to end data with new data gathered from one day before. (Volume of data is now around 2 years data) Rather than doing that workload everyday, is there any way to add only new data into the latest STS model trained from the previous day?
I read related STS sources in Tensorflow official webpages below:
https://www.tensorflow.org/probability/api_docs/python/tfp/sts/StructuralTimeSeries
https://blog.tensorflow.org/2019/03/structural-time-series-modeling-in.html
However, none of these pages could help me. Training the model everyday causes a long time process for STS on my system. I am trying to find a way to decrease that time with a possible approach of retraining the pretrained STS model with only new data.
When you say that you wish to re-use the previous model training to incorporate new data and only train on this new data - what you are referring to specifically is known as transfer learning, whereby a model trained to generate one set of predictions is re-used to generate another.
TensorFlow has the ability to import a MetaGraph file using import_meta_graph(), which allows for the importing of a model and the continued training of that model without having to reconstruct the model outright. In your case, this would allow for importing of the parameters for the existing STS model to be trained on new data on an iterative basis.
I would recommend referring to the following guide for further details: Exporting and Importing a MetaGraph