I am using gsutil rsync to backup a directory. Hidden files, like .vimrc and .gitignore, do not upload when the -r option is omitted. Other files upload as expected. when the -r option is added to the command, .vimrc and .gitignore files upload as expected.
Here it is without option -D:
[wolfv@localhost test_hidden]$ ls -A
file1 .vimrc
[wolfv@localhost test_hidden]$ gsutil ls gs://wolfv/
[wolfv@localhost test_hidden]$ gsutil rsync -n . gs://wolfv/
Building synchronization state...
Starting synchronization
Would copy file://./file1 to gs://wolfv/file1
[wolfv@localhost test_hidden]$ gsutil rsync -n -r . gs://wolfv/
Building synchronization state...
Starting synchronization
Would copy file://./.vimrc to gs://wolfv/.vimrc
Would copy file://./file1 to gs://wolfv/file1
[wolfv@localhost test_hidden]$ ls -A
file1 .vimrc
[wolfv@localhost test_hidden]$ gsutil -D ls gs://wolfv/
Same commands as above but with option -D:
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 4.13
checksum: 141a3e09b42e1b0b6033108aa24c2286 (OK)
boto version: 2.38.0
python version: 2.7.10 (default, May 27 2015, 18:11:38) [GCC 5.1.1 20150422 (Red Hat 5.1.1-1)]
OS: Linux 4.0.7-300.fc22.x86_64
multiprocessing available: True
using cloud sdk: False
config path: /home/wolfv/.boto
gsutil path: /home/wolfv/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /home/wolfv/gsutil/gsutil -D ls gs://wolfv/
config_file_list: ['/home/wolfv/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', 'ringed-tempo-100819')]
DEBUG 0718 22:33:54.942377 oauth2_client.py] GetAccessToken: checking cache for key ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6
DEBUG 0718 22:33:54.942723 oauth2_client.py] FileSystemTokenCache.GetToken: key=ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6 present (cache_file=/tmp/oauth2_client-tokencache.1000.ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6)
DEBUG 0718 22:33:54.942815 oauth2_client.py] GetAccessToken: token from cache: AccessToken(token=ya29.tQG4ErIFUzLf0ZoD6MdhERdJ4IhQHQF9-9ZcKoBUg1e5rOISXJAIAJD-XG52T12wpZvLMw, expiry=2015-07-19 04:29:00.268766Z)
DEBUG 0718 22:33:54.942959 oauth2_client.py] GetAccessToken: fetching fresh access token...
INFO 0718 22:33:54.943222 client.py] Refreshing access_token
connect: (accounts.google.com, 443)
send: 'POST /o/oauth2/token HTTP/1.1\r\nHost: accounts.google.com\r\nContent-Length: 177\r\ncontent-type: application/x-www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.7 (gzip)\r\n\r\nclient_secret=p3RlpR10xMFh9ZXBS%2FZNLYUu&grant_type=refresh_token&refresh_token=1%2FJvrq_d-mE-fbwwFS4JLxagrhVBW07jfz21ekrnfaPsU&client_id=909320924072.apps.googleusercontent.com'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=utf-8
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Fri, 01 Jan 1990 00:00:00 GMT
header: Date: Sun, 19 Jul 2015 04:33:55 GMT
header: Content-Disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt
header: Content-Encoding: gzip
header: P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Set-Cookie: NID=69=lZBJSKuCt9Fp7sd4TYUVI3n2qiQMMmzZsWJ-rkeBlnC1Bq0HVKNVR87DiGcK5w_aqdmqtWjXBUvy2wR1gmtpJxrI5xiy7MZK0fMB3N6G1Krr9PX53lp5AoG1wmWZaIZF;Domain=.google.com;Path=/;Expires=Mon, 18-Jan-2016 04:33:55 GMT;HttpOnly
header: Alternate-Protocol: 443:quic,p=1
header: Transfer-Encoding: chunked
DEBUG 0718 22:33:55.357366 oauth2_client.py] GetAccessToken: fresh access token: AccessToken(token=ya29.tQEgWx31iAmwz37g4HUo_Ke1MilZengvYlARB4qtCALZLjWXvEfITKBn1nk4RIeN3k1epQ, expiry=2015-07-19 05:33:55.356008Z)
DEBUG 0718 22:33:55.357599 oauth2_client.py] FileSystemTokenCache.PutToken: key=ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6, cache_file=/tmp/oauth2_client-tokencache.1000.ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6
INFO 0718 22:33:55.362168 base_api.py] Calling method storage.objects.list with StorageObjectsListRequest: <StorageObjectsListRequest
bucket: u'wolfv'
delimiter: '/'
maxResults: 1000
projection: ProjectionValueValuesEnum(full, 0)
versions: False>
INFO 0718 22:33:55.363248 base_api.py] Making http GET to https://www.googleapis.com/storage/v1/b/wolfv/o?delimiter=%2F&projection=full&versions=False&fields=prefixes%2CnextPageToken%2Citems%2Fname&alt=json&maxResults=1000
INFO 0718 22:33:55.363693 base_api.py] Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools gsutil/4.13 (linux2)'}
INFO 0718 22:33:55.363809 base_api.py] Body: (none)
connect: (www.googleapis.com, 443)
send: 'GET /storage/v1/b/wolfv/o?delimiter=%2F&projection=full&versions=False&fields=prefixes%2CnextPageToken%2Citems%2Fname&alt=json&maxResults=1000 HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Sun, 19 Jul 2015 04:33:55 GMT
header: Date: Sun, 19 Jul 2015 04:33:55 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 3
header: Server: GSE
header: Alternate-Protocol: 443:quic,p=1
INFO 0718 22:33:55.831361 base_api.py] Response of type Objects: <Objects
items: []
prefixes: []>
[wolfv@localhost test_hidden]$ gsutil -D rsync -n . gs://wolfv/
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 4.13
checksum: 141a3e09b42e1b0b6033108aa24c2286 (OK)
boto version: 2.38.0
python version: 2.7.10 (default, May 27 2015, 18:11:38) [GCC 5.1.1 20150422 (Red Hat 5.1.1-1)]
OS: Linux 4.0.7-300.fc22.x86_64
multiprocessing available: True
using cloud sdk: False
config path: /home/wolfv/.boto
gsutil path: /home/wolfv/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /home/wolfv/gsutil/gsutil -D rsync -n . gs://wolfv/
config_file_list: ['/home/wolfv/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', 'ringed-tempo-100819')]
Building synchronization state...
process count: 4
thread count: 10
DEBUG: wildcard=*, prefix=None, delimiter=/, prefix_wildcard=*, suffix_wildcard=
DEBUG 0718 22:34:30.723059 oauth2_client.py] GetAccessToken: checking cache for key ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6
DEBUG 0718 22:34:30.724428 oauth2_client.py] FileSystemTokenCache.GetToken: key=ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6 present (cache_file=/tmp/oauth2_client-tokencache.1000.ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6)
DEBUG 0718 22:34:30.724675 oauth2_client.py] GetAccessToken: token from cache: AccessToken(token=ya29.tQEgWx31iAmwz37g4HUo_Ke1MilZengvYlARB4qtCALZLjWXvEfITKBn1nk4RIeN3k1epQ, expiry=2015-07-19 05:33:55.356008Z)
INFO 0718 22:34:30.728948 base_api.py] Calling method storage.objects.list with StorageObjectsListRequest: <StorageObjectsListRequest
bucket: u'wolfv'
delimiter: '/'
maxResults: 1000
projection: ProjectionValueValuesEnum(full, 0)
versions: False>
INFO 0718 22:34:30.730205 base_api.py] Making http GET to https://www.googleapis.com/storage/v1/b/wolfv/o?delimiter=%2F&projection=full&versions=False&fields=nextPageToken%2Citems%2Fcrc32c%2Cprefixes%2Citems%2Fname%2Citems%2Fmd5Hash%2Citems%2Fsize&alt=json&maxResults=1000
INFO 0718 22:34:30.730626 base_api.py] Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools gsutil/4.13 (linux2)'}
INFO 0718 22:34:30.730757 base_api.py] Body: (none)
connect: (www.googleapis.com, 443)
send: 'GET /storage/v1/b/wolfv/o?delimiter=%2F&projection=full&versions=False&fields=nextPageToken%2Citems%2Fcrc32c%2Cprefixes%2Citems%2Fname%2Citems%2Fmd5Hash%2Citems%2Fsize&alt=json&maxResults=1000 HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Sun, 19 Jul 2015 04:34:31 GMT
header: Date: Sun, 19 Jul 2015 04:34:31 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 3
header: Server: GSE
header: Alternate-Protocol: 443:quic,p=1
INFO 0718 22:34:31.169427 base_api.py] Response of type Objects: <Objects
items: []
prefixes: []>
Starting synchronization
process count: 1
thread count: 1
Would copy file://./file1 to gs://wolfv/file1
[wolfv@localhost test_hidden]$ gsutil -D rsync -n -r . gs://wolfv/
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 4.13
checksum: 141a3e09b42e1b0b6033108aa24c2286 (OK)
boto version: 2.38.0
python version: 2.7.10 (default, May 27 2015, 18:11:38) [GCC 5.1.1 20150422 (Red Hat 5.1.1-1)]
OS: Linux 4.0.7-300.fc22.x86_64
multiprocessing available: True
using cloud sdk: False
config path: /home/wolfv/.boto
gsutil path: /home/wolfv/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /home/wolfv/gsutil/gsutil -D rsync -n -r . gs://wolfv/
config_file_list: ['/home/wolfv/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', 'ringed-tempo-100819')]
Building synchronization state...
process count: 4
thread count: 10
DEBUG: wildcard=**, prefix=None, delimiter=None, prefix_wildcard=**, suffix_wildcard=
DEBUG 0718 22:34:50.613022 oauth2_client.py] GetAccessToken: checking cache for key ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6
DEBUG 0718 22:34:50.613509 oauth2_client.py] FileSystemTokenCache.GetToken: key=ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6 present (cache_file=/tmp/oauth2_client-tokencache.1000.ec5a17a3e80585aa19a1b83b618765c8d0d5b3c6)
DEBUG 0718 22:34:50.613626 oauth2_client.py] GetAccessToken: token from cache: AccessToken(token=ya29.tQEgWx31iAmwz37g4HUo_Ke1MilZengvYlARB4qtCALZLjWXvEfITKBn1nk4RIeN3k1epQ, expiry=2015-07-19 05:33:55.356008Z)
INFO 0718 22:34:50.617128 base_api.py] Calling method storage.objects.list with StorageObjectsListRequest: <StorageObjectsListRequest
bucket: u'wolfv'
maxResults: 1000
projection: ProjectionValueValuesEnum(full, 0)
versions: False>
INFO 0718 22:34:50.618298 base_api.py] Making http GET to https://www.googleapis.com/storage/v1/b/wolfv/o?fields=nextPageToken%2Citems%2Fcrc32c%2Cprefixes%2Citems%2Fname%2Citems%2Fmd5Hash%2Citems%2Fsize&alt=json&projection=full&maxResults=1000&versions=False
INFO 0718 22:34:50.618670 base_api.py] Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools gsutil/4.13 (linux2)'}
INFO 0718 22:34:50.618785 base_api.py] Body: (none)
connect: (www.googleapis.com, 443)
send: 'GET /storage/v1/b/wolfv/o?fields=nextPageToken%2Citems%2Fcrc32c%2Cprefixes%2Citems%2Fname%2Citems%2Fmd5Hash%2Citems%2Fsize&alt=json&projection=full&maxResults=1000&versions=False HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Sun, 19 Jul 2015 04:34:51 GMT
header: Date: Sun, 19 Jul 2015 04:34:51 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 3
header: Server: GSE
header: Alternate-Protocol: 443:quic,p=1
INFO 0718 22:34:51.126102 base_api.py] Response of type Objects: <Objects
items: []
prefixes: []>
Starting synchronization
process count: 1
thread count: 1
Would copy file://./.vimrc to gs://wolfv/.vimrc
Would copy file://./file1 to gs://wolfv/file1
You've uncovered a gsutil bug. The fix will be in the gsutil 4.14 release.
Thanks for reporting this.