Search code examples
message-queuenats.ionats-jetstream

How to backup all streams in JetStream server?


Problem:

I have to backup all the streams of my JetStream server. I don't know what streams are there. I want to backup all the streams into a single snapshot. Currently, we can backup only a single stream using the following command.

$ nats stream backup <stream-name> /backup/dir/<stream-name>.tgz

What I have tried so far:

I have tried providing a wildcard instead of <stream-name>. It does not work.

$ nats stream backup * /backup/dir/backup.tgz
nats: error: "*" is not a valid stream name, try --help

Possible Workaround:

At first, I can list all the streams using nats str ls command. Then, I can loop through all the streams and backup them individualy.

However, this does not satisfy my requirement as I want to backup all the streams into a single snapshot. My snapshot should represent a complete state of the JetStream server just not a single stream.


Solution

  • [Edited as there is now a better way to do this through new functionality]

    Updated answer:

    You can backup all of the streams for an account using nats account backup [<flags>] <directory to create the backup in>

    Original answer:

    Yes creating a script to capture the list of all currently defined streams using nats stream ls and then backing them up in a loop using nats stream backup is the best you can do.

    It is not possible to create a single 'frozen at a point in time' complete backup of all the streams at the same time because:

    • NATS and JetStream continue to operate while the backup is happening
    • NATS and JetStream are distributed systems: servers and streams are independent from each other

    The only way to create such a backup would be to synchronously suspend all writes (publications or consumptions of messages) on all streams and their source subjects, perform the backup (which could take a fair amount of time), and then allow operations on the streams and subject to resume. This interruption in service would not be acceptable to many (most?) nats users.