I have cloud run services hosting GO OSB application implementing gRpc but exposing the http REST api's via grpc-gateway which uses cloud sql (mysql) as a DB. All the CRUD API's are responding fine except the PATCH one.
It's throwing the below error with http response code 503:
{
"textPayload": "The request failed because either the HTTP response was malformed or connection to the instance had an error.",
"insertId": "6141e984000c63529e7b7afd",
"httpRequest": {
"requestMethod": "PATCH",
"requestUrl": "https://********-********-mr336-qv7hk7cx3a-uc.a.run.app/v2/service_instances/237e80fd-b22e-4df0-b9ed-23c91a4d7f51",
"requestSize": "1102",
"status": 503,
"responseSize": "976",
"userAgent": "PostmanRuntime/7.28.4",
"remoteIp": "********",
"serverIp": "********",
"latency": "0.410343680s",
"protocol": "HTTP/1.1"
},
"resource": {
"type": "cloud_run_revision",
"labels": {
"location": "us-central1",
"revision_name": "********-********-mr336-00001-hop",
"project_id": "********-********-l-app-us-01",
"configuration_name": "********-********-mr336",
"service_name": "********-********-mr336"
}
},
"timestamp": "2021-09-15T12:39:32.811858Z",
"severity": "ERROR",
"labels": {
"instanceId": "00bf4bf02dff6d5f53cff1f1828cafbca265606a996eddff5cb44e3fff674efb77ca51eca7087fb8b8e7acba227b2a3e3e913bdfcc0a487640a2e028"
},
"logName": "projects/********/logs/run.googleapis.com%2Frequests",
"trace": "projects/********/traces/e29e5add9452d171e9eebd26817bb667",
"receiveTimestamp": "2021-09-15T12:39:32.817171397Z"
}
Points to Note :
After every patch request I can see the instance start-up logs, i.e. after the above mentioned logs every time I can see the container entrypoint (server) startup logs (like cold-start).
As soon as server startup is complete, it again throws the same above error in logs.
Important point to note is that I also can't see any logs from my application which suggests PATCH api request is not reaching the container instances running behind the cloud run services.
Also my active instances after cold start goes to ideal and then scales down to 0 with in a 1 min. after the last request hit, but it doesn't seems to create issue for other APIs. This is how it is supposed to work but can't find any lead on what's the issue with PATCH.
This is fixed now !
Facing the due to handling of multiple protocols on the same port and one of the protocol matchers was causing the issue with PATCH API by returning Empty reply from server. So, done changes to the matchers and it worked.
RCA : Cmux HTTP1Fast Matcher only matches the methods in the HTTP request. This matcher is very optimistic: if it returns true, it does not mean that the request is a valid HTTP response. A correct but slower HTTP1 matcher, used "HTTP1" instead which scan the whole request up-to 4096 bytes but its bit slow.