Saving and restoring neural network weights in production

Question

I am currently building a backend that uses a neural network. I need to save the weights (which may be 2 dimensional, 3 dimensional or 4 dimensional), and restore them. I am currently using heroku, and thus need to save them either to PostgreSQL, or to a S3 bucket and retrieve it everytime the system boots.

What is the go-to solution for storing and restoring weights for ML applications in production, where the weights may be multiple hundred-thousand entries and the matrix can be well over 100mb?

Answer 1

We do store neural network weights over S3 and it works pretty good.

Also we update those weights as we train with new data and update to s3. The live system checks for modified s3 object and update the weights / neural network based on the stored data.

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html

aws s3api head-object --bucket my-bucket --key object.h5

You can also enable accelerated endpoint for speed. This is needed if you are downloading the object over the internet.

https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html

Hope it helps.