I've mounted a GCS bucket using gcsfuse, After updating my startup script to include --implicit-dirs, I can no longer read from the mounted directory /gcs/my-bucket. I receive an "Input/output error" when attempting to list files (ls) or access the directory. However, while I cannot access the /gcs/my-bucket directory directly, I am able to access files within its subdirectories. The permissions seem correct, and I have full access setup for the storage.
I added the --implicit-dirs flag to resolve issues related to accessing files that were uploaded using the GCP SDK. Sometimes vm can't access those, while files uploaded via gcsfuse are reachable.
Cloud API access scopes (VM PERMISSIONS)
Allow custom access
BigQuery Disabled
Bigtable Admin Disabled
Bigtable Data Disabled
Cloud Datastore Disabled
Cloud Debugger Disabled
Cloud Platform Disabled
Cloud Pub/Sub Disabled
Cloud Source Repositories Full
Cloud SQL Disabled
Compute Engine Disabled
Service Control Enabled
Service Management Read Only
Stackdriver Logging API Write Only
Stackdriver Monitoring API Write Only
Stackdriver Trace Write Only
Storage Full
Task queue Disabled
User info Disabled
Service Account Permissions: Firebase Test Lab Viewer Logs Writer Storage Admin Storage Object Admin Storage Object Creator
Startup Script of vm:
#! /bin/bash
gcsfuse --implicit-dirs --foreground --debug_gcs -o rw,allow_other my-bucket /gcs/my-bucket
chmod 777 /gcs/my-bucket
gcsfuse logs:
"startup-script: {"timestamp":{"seconds":1718604762,"nanos":135066298},"severity":"INFO","message":"Start gcsfuse/2.1.0 (Go version go1.22.3) for app \"\" using mount point: /gcs/my-bucket\n"}"
startup-script: {"timestamp":{"seconds":1718604762,"nanos":135769488},"severity":"INFO","message":"GCSFuse mount command flags: {\"AppName\":\"\",\"Foreground\":true,\"ConfigFile\":\"\",\"MountOptions\":{\"allow_other\":\"\",\"ro\":\"\"},\"DirMode\":493,\"FileMode\":420,\"Uid\":-1,\"Gid\":-1,\"ImplicitDirs\":true,\"OnlyDir\":\"\",\"RenameDirLimit\":0,\"IgnoreInterrupts\":false,\"CustomEndpoint\":null,\"BillingProject\":\"\",\"KeyFile\":\"\",\"TokenUrl\":\"\",\"ReuseTokenFromUrl\":true,\"EgressBandwidthLimitBytesPerSecond\":-1,\"OpRateLimitHz\":-1,\"SequentialReadSizeMb\":200,\"AnonymousAccess\":false,\"MaxRetrySleep\":30000000000,\"StatCacheCapacity\":20460,\"StatCacheTTL\":60000000000,\"TypeCacheTTL\":60000000000,\"HttpClientTimeout\":0,\"MaxRetryDuration\":-1000000000,\"RetryMultiplier\":2,\"LocalFileCache\":false,\"TempDir\":\"\",\"ClientProtocol\":\"http1\",\"MaxConnsPerHost\":0,\"MaxIdleConnsPerHost\":100,\"EnableNonexistentTypeCache\":false,\"StackdriverExportInterval\":0,\"OtelCollectorAddress\":\"\",\"LogFile\":\"\",\"LogFormat\":\"json\",\"ExperimentalEnableJsonRead\":false,\"DebugFuseErrors\":true,\"DebugFuse\":false,\"DebugFS\":false,\"DebugGCS\":true,\"DebugHTTP\":false,\"DebugInvariants\":false,\"DebugMutex\":false}"}
startup-script: {"timestamp":{"seconds":1718604762,"nanos":135881882},"severity":"INFO","message":"GCSFuse mount config flags: {\"CreateEmptyFile\":false,\"Severity\":\"TRACE\",\"Format\":\"json\",\"FilePath\":\"\",\"LogRotateConfig\":{\"MaxFileSizeMB\":512,\"BackupFileCount\":10,\"Compress\":true},\"MaxSizeMB\":-1,\"CacheFileForRangeRead\":false,\"CacheDir\":\"\",\"TtlInSeconds\":-9223372036854775808,\"TypeCacheMaxSizeMB\":4,\"StatCacheMaxSizeMB\":-9223372036854775808,\"EnableEmptyManagedFolders\":false,\"ConnPoolSize\":1,\"AnonymousAccess\":false,\"EnableHNS\":false,\"IgnoreInterrupts\":false}"}
$ mount
my-bucket on /gcs/my-bucket type fuse.gcsfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
$ df -h
Filesystem Size Used Avail Use% Mounted on
my-bucket 1.0P 0 1.0P 0% /gcs/my-bucket
/gcs$ ls -al my-bucket/
ls: reading directory 'my-bucket/': Input/output error
total 0
/gcs$ $ ls -al my-bucket/prod
total 0
drwxr-xr-x 1 root root 0 Jun 17 07:25 folder1
drwxr-xr-x 1 root root 0 Jun 17 07:25 folder2
Steps Already Tried:
- Checking permissions on the Google Cloud Console.
- Remounting the bucket without the --implicit-dirs option (which resolves the reading issue but doesn't work for my use case).
- Verifying the service account has adequate permissions.
I'm looking for any advice on what might be causing this and how to resolve it so that I can use --implicit-dirs without losing read access. Thanks in advance for your help!
Is there any directory with empty name on the bucket (can be seen as "/" on cloud console)? GCSFuse does not support directories with empty name in its path (i.e. objects with double-slash). For eg:
gsutil cp <file-name> gs://<bucket-name>//<file-name>
will create an empty directory on bucket and it will cause this type input/output error while doing ls on gcsfuse mounted dir.
Removing this empty directory from the bucket should resolve the issue.