How do I Backup / Move LXC containers?

I want to take lxc container backup. We have server with 12.04 LTS ubuntu server and I have installed LXC - 1.0.0.alpha2 in it. I wanted to update our ubuntu server to 14.04 LTS. So what I want to do is have LXC containers backed up -> upgrade OS to 14.04 -> restore LXC containers. With earlier version(0.7.5 I guess) there was lxc-backup and lxc-restore but with 1.0.0.alpha2 we don't have backup and restore operations. How can I have lxc containers backup. I spent more than 3 hours with copy container folder(/var/lib/lxc/my_container/) into pendrive and paste it in another 12.04 LTS server but it is not working the error am getting is,

#sudo lxc-start -n my_container
lxc-start: invalid sequence number 1, expected 4.
lxc-start: failed to spwan "my_container"

then how can I expect that it will work in upgraded 14.04 server OS.

Any Idea to have lxc-container backup?

Solution

EDIT: March 29, 2016

In case you stumbled upon this post, my answer is really about moving the LXC containers between systems, since that seemed to be the question being asked.

If you want to backup your LXC containers, see @Stuart's answer for some great options.

Moving LXC containers between host systems

This is how I migrate LXC containers between systems. I've successfully moved ubuntu based 12.04 containers to a 14.04 host, and they work great.

Shutdown the container
```
# lxc-stop -n $NAME
```
Archive container rootfs & config
```
# cd /var/lib/lxc/$NAME/
# tar --numeric-owner -czvf container_fs.tar.gz ./*
```
The --numeric-owner flag is very important! Without it, the container may not boot because the uid/gids get mangled in the extracted filesystem. When tar creates an archive, it preserves user / group ownership information. By default, when extracting, tar tries to resolve the archive user/group ownership names with the ids on the system running tar. This is intended to ensure that user ownership is resolved on the new system, in case the UID numeric values differ between systems.

This is bad for an LXC filesystem because the numeric uid/gid ownership is intended to be preserved for the whole filesystem. If it gets resolved to a different value, bad things happen.

Copy the file to your new server

# rsync -avh container_fs.tar.gz user@newserver:/var/lib/lxc/

Extract rootfs

# mkdir /var/lib/lxc/$NAME/
# cd /var/lib/lxc/$NAME/
# tar --numeric-owner -xzvf container_fs.tar.gz .

If you're using an overlay backed container, you'll also need to migrate the container this new one is based off of. Lastly, you might see a few warnings about skipped socket files:

tar: /var/lib/lxc/$NAME/rootfs/dev/log: socket ignored

I've ignored this error, and haven't had any issues with any of the containers I manage. If you have further issues, add your error messages to the original post and I'll elaborate.