I am using rsync version 3.1.1 protocol version 31 with cygwin version 2.3.1(0.291/5/3).
I believe that I need to chown all my Win7 files, with unknown uid/gid's, which rsync would copy, BEFORE the first rsync is run on these files.
On unknown uid/gid's rsync generates 'uid/gid impossible to set' messages and does copy the files but they can NOT be then hard linked in future incremental backups.
I have tried a lot of rsync options to solve this with no success.
How do I do this?
The reason I believe that is because when I have rsync create hard links on my incremental backups (with the appropriate --link-dest=destination), only files with a certain user and group have their hard links created. I have one hard drive with 600,000 files that won't allow hard links to be created after they are transferred by rsync. All my other drives have numerous files that behave this way too. This makes incremental backups with hard links useless from that drive and 'not quite right' with the other drives.
Here is an example of various user and group status on that Win7 drive:
ls -al
total 828
drwxrwx---+ 1 Unknown+User Unknown+Group 0 Dec 7 21:33 .
dr-xrwxr-x+ 1 Unknown+User Unknown+Group 0 Dec 7 20:53 ..
-rwxrwx---+ 1 Administrators None 773985 Jan 1 2012 1.txt
-rwxrwx---+ 1 Unknown+User Unknown+Group 27936 Mar 12 2009 DATtoKML.kml
The 1.txt file was put there through a recent drag and drop in windows file explorer. The DATtoKML.kml file is an example of the 600,000 files that are a problem. The 1.txt file has no problem being rsynced and hard linked after that.
Here is what the files look like on the destination after being rsynced (with --link-dest set to a previous rsynced location) there:
ls -al
total 792
drwxrwxrwx 2 root root 4096 Dec 7 21:33 .
drwxrwxrwx 4 root root 4096 Dec 7 21:34 ..
-rwxrwxrwx 2 544 197121 773985 Jan 1 2012 1.txt
-rwxrwxrwx 1 root root 27936 Mar 12 2009 DATtoKML.kml
As you can see, DATtoKML.kml did NOT hard link and will forever get transferred in full on every incremental rsync.
Also, these non-hardlinkable files get messages from rsync when they are worked on by rsync. The messages are that the 'uid/gid's are impossible to set'. These are not fatal errors because the files copy, but don't seem to have their users and groups set so they are hardlinkable.
Here are the two rsync's that resulted in the above listing:
Dale@Uva1 ~
$ rsync -av --chmod=o=rwx /cygdrive/e/DATtoKML/ [email protected]:/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST
sending incremental file list
created directory /DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST
./
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/."
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/."
1.txt
DATtoKML.kml
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/.DATtoKML.kml.eR2hUv"
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/.DATtoKML.kml.eR2hUv"
sent 802,347 bytes received 628 bytes 1,605,950.00 bytes/sec
total size is 801,921 speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
Dale@Uva1 ~
$ rsync -av --chmod=o=rwx --link-dest=/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST /cygdrive/e/DATtoKML/ [email protected]:/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07
sending incremental file list
created directory /DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07
./
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/."
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/."
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/.DATtoKML.kml.1lp4od"
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/.DATtoKML.kml.1lp4od"
DATtoKML.kml
sent 155 bytes received 559 bytes 476.00 bytes/sec
total size is 801,921 speedup is 1,123.14
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
Dale@Uva1 ~
$
So far the only thing that seems to work is to chown and chgrp DATtoKML.kml and the files parent directory on the Win7 drive before I ever rsync them. This does not seem practical to me, not so hard to do on the drive where almost all of them are bad, but the other drive has these type of error scattered across thousands of directories.
I think rsync needs an option like this to handle 'uid/gid impossible to set' cases:
--uid-error=544 --gid-error=197121
Or, it should just use appropriate user and group (which it should know from the user that is rsyncing) for future hardlinkability whenever 'uid/gid impossible to set' arises.
Anyone want to try compiling an rsync version with that option?
EDIT: It looks like all the files and directories at issue have a uid and/or gid of 4294967295. As a temporary solution I am going to write a bash script which does find on each win7 drive (but only include files that will be transferred considering my rsync exclude file) and change any bad uids to 544 and bad gids to 197121. The 544 and 197121 are used because they are the user and group of the user that runs the rsync command.
That way, when I start seeing these uid/gid issues surfacing in my rsync logs I can run the script again.
Hopefully there will be a real answer to this question in the future.
Use the --usermap and --groupmap rsync options (this requires that both source and dest environments use version 3.1.0 or higher). In my case this won't work since my destination (WD MyCloud) uses 3.0.9.
OR
Change the rsync source code with gcc running under cygwin (really not that hard with some C knowledge).