Just wondering if this performance is normal. I'm rsyncing a dir (about 20 gb of images) into a gluster volume (3 nodes, 3 replicas, each node in the same digitalocean data center).
rsync --info=progress2 -r /var/www/app/_appdata/ /mnt
251,670,295 62% 716.84kB/s 0:05:42 (xfr#5708, ir-chk=1257/7830)
When I copy just a big file, I get around 30 MB/s.
When I copy a dir full of many files, I get about ~700 kB/s
Any idea why it's so much slower for many files?
Rsync is a particularly tough workload for GlusterFS because, with its defaults, it exercises some of the worst case operations for GlusterFS. So, getting the best performance from rsync requires some tuning/tweaking on both sides.
Next is described the pain points of the increase in performance of the Rsync and GlusterFS, when they work together.
The first
The main issue with rsync and GlusterFS is rsync uses the “write new then rename” idiom when creating files. The means that for every file created, GlusterFS is forced to rename the file, which is by far the most expensive file operation (FOP). The Gluster developers added a couple tunables to help this workload:
For example, rsync and similar tools often use a "write new then rename" idiom in which a file "xxx" is actually written as ".xxx.1234" and then moved into place only after its contents have been fully written. To make this process more efficient, DHT uses a regular expression to separate the permanent part of a file’s name (in this case "xxx") from what is likely to be a temporary part (the leading "." and trailing ".1234"). That way, after the file is renamed it will be in its correct hashed location – which it wouldn’t be otherwise if "xxx" and ".xxx.1234" hash differently – and no linkfiles or broadcast lookups will be necessary.
In fact, there are two regular expressions available for this purpose – cluster.rsync-hash-regex and cluster.extra-hash-regex. As its name implies, rsync-hash-regex defaults to the pattern that regex uses, while extra-hash-regex can be set by the user to support a second tool using the same temporary-file idiom.
The second
Rsync defaults to a pretty small request size, and this also is a weak point on GlusterFS. GlusterFS tends to perform best with request sizes over 64KB; 1MB tends to provide the best performance. With request sizes that are less than 4KB, things really start to degrade. Rsync does have a tunable to change this behavior. It’s called block-size. Rsync’s default for block size is 2KB, which really hurts performance when rsyncing to/from GlusterFS. For example:
# rsync -vrah /gluster-mount/ /home/bturner/Downloads/ --progress -B=131072
You can also look at the following option (see the rsync man page):
-W, --whole-file
This option disables rsync’s delta-transfer algorithm, which causes all
transferred files to be sent whole. The transfer may be faster if this
option is used when the bandwidth between the source and destination
machines is higher than the bandwidth to disk (especially when the “disk”
is actually a networked filesystem). This is the default when both
the source and destination are specified as local paths, but only if
no batch-writing option is in effect.
The third
--inplace option. This option behaves similarly to the rsync regex option described above except it is implemented on the rsync side instead of the GlusterFS side. The following information is from the man page:
–inplace update destination files in-place
This option changes how rsync transfers a file when its data needs to be
updated: instead of the default method of creating a new copy of the file
and moving it into place when it is complete, rsync instead writes
the updated data directly to the destination file.
See more here.