Recently I have needed to add web sockets to my backend application currently hosted on Google App Engine (GAE) standard environment. Because web sockets are a feature only available in GAE's flexible environment, I have been attempting a redeployment but with little success.
To make the change to a flexible environment I have updated the app.yaml file from
runtime: nodejs10
env: standard
to
runtime: nodejs
env: flex
While previously working in the standard environment, now with env: flex when I run the command gcloud app deploy --app-yaml=app-staging.yaml --verbosity=debug I get the following stack trace:
Do you want to continue (Y/n)? Y
DEBUG: No bucket specified, retrieving default bucket.
DEBUG: Using bucket [gs://staging.finnsalud.appspot.com].
DEBUG: Service [appengineflex.googleapis.com] is already enabled for project [finnsalud]
Beginning deployment of service [finnsalud-staging]...
INFO: Using ignore file at [~/checkouts/twilio/backend/.gcloudignore].
DEBUG: not expecting type '<class 'NoneType'>'
Traceback (most recent call last):
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
resources = command_instance.Run(args)
File "/google-cloud-sdk/lib/surface/app/deploy.py", line 115, in Run
return deploy_util.RunDeploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 669, in RunDeploy
deployer.Deploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 428, in Deploy
source_files = source_files_util.GetSourceFiles(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/source_files_util.py", line 184, in GetSourceFiles
return list(it)
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/gcloudignore.py", line 233, in GetIncludedFiles
six.ensure_str(upload_directory), followlinks=True):
File "//google-cloud-sdk/lib/third_party/six/__init__.py", line 884, in ensure_str
raise TypeError("not expecting type '%s'" % type(s))
TypeError: not expecting type '<class 'NoneType'>'
ERROR: gcloud crashed (TypeError): not expecting type '<class 'NoneType'>'
In this stack trace, it mentions an error in google-cloud-sdk/lib/googlecloudsdk/command_lib/util/gcloudignore.py so I had also reviewed my .gcloudignore file but was unable to find anything out of place:
.gcloudignore
.git
.gitignore
node_modules/
In an attempt to work around this bug I tried removing my .gcloudignore file which resulted in a different error, but still failed nevertheless:
Do you want to continue (Y/n)? Y
DEBUG: No bucket specified, retrieving default bucket.
DEBUG: Using bucket [gs://staging.finnsalud.appspot.com].
DEBUG: Service [appengineflex.googleapis.com] is already enabled for project [finnsalud]
Beginning deployment of service [finnsalud-staging]...
DEBUG: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
resources = command_instance.Run(args)
File "/google-cloud-sdk/lib/surface/app/deploy.py", line 115, in Run
return deploy_util.RunDeploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 669, in RunDeploy
deployer.Deploy(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 428, in Deploy
source_files = source_files_util.GetSourceFiles(
File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/source_files_util.py", line 184, in GetSourceFiles
return list(it)
File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/util.py", line 165, in FileIterator
entries = set(os.listdir(os.path.join(base, current_dir)))
File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/posixpath.py", line 76, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: gcloud crashed (TypeError): expected str, bytes or os.PathLike object, not NoneType
Thinking maybe this was an error relating to the version of my CLI I have also run the following commands to try and update:
gcloud app update
gcloud components update
Unfortunately, this had no change on the output.
I have noticed that when I run this command with the app.yaml env value set to flexible, there are no updates to the logging section on google cloud and no changes to the files uploaded to the project's storage bucket. To me, this indicates that the crash is occurring in the CLI before any communication to the google cloud services is made. If this is correct, then it seems unlikely that the cause of the error would be related to a bad configuration on google cloud and must be related to something (software or configuration) on my local machine.
I have also tried using the 'Hello World' app.yaml configuration on the flexible environments 'Getting Started' page to rule out a configuration error my own application's app.yaml but this also had no change on the output.
Finally, if at any point I change env: flex back to env: standard then the issue does disappear. Unfortunately, as stated above, this won't work for deploying my web sockets feature.
This has gotten me thinking that possibly the error is due to a bug with the gcloud cli application. However, if this were the case, I would have expected to see many more bug reports for this issue by others whom are also using the GAE's flexible environment.
Regardless, given this stack trace points to code within the gcloud cli, I have opened a bug ticket with google which can be found here: https://issuetracker.google.com/issues/176839574
I have also seen this similar SO post, but it is not the exact error I am experiencing and remains unresolved: gcloud app deploy fails with flexible environment
If anyone has any ideas on other steps to try or methods to overcome this issue, I would be immensely grateful if you drop a note on this post. Thanks!
Surprisingly, this issue does seem to be related to a bug in the gcloud cli. However, there does seem to be a workaround.
When a --appyaml flag is specified for a deployment to the flex environment, then the CLI crashes with the messages outlines in my question above. However, if you copy your .yaml file renaming to app.yaml (the default) and delete this --appyaml flag when deploying then the build will proceed without errors.
If you have also experienced this error, please follow the google issue as I am working with the google engineers to be sure they reproduce and eventually fix this bug.