GCP - Failed to update metadata for Cloud SQL instance

Question

I'm receiving the below response from the App which is running on GKE. Please advise. This was working fine until yesterday.

========================================================================== Cloud SQL SSL Setting - Unsecured connections are allowed to connect to this instance.

    spring.datasource.url=jdbc:mysql://google/<DB-NAME>?cloudSqlInstance=<INSTANCE-CONN- 
    URL>&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=<USER>&password=<PASS>

        <dependency>
            <groupId>com.google.cloud.sql</groupId>
            <artifactId>mysql-socket-factory</artifactId>
            <version>1.0.15</version>
        </dependency>


Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Throwables.throwIfUnchecked(Ljava/lang/Throwable;)V

I 2019-09-22T08:01:28.370430889Z    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)

I 2019-09-22T08:01:28.370424094Z }

I 2019-09-22T08:01:28.370417350Z   "status" : "PERMISSION_DENIED"

I 2019-09-22T08:01:28.370409799Z   "message" : "Request had insufficient authentication scopes.",

I 2019-09-22T08:01:28.370402379Z   } ],

I 2019-09-22T08:01:28.370378956Z     "reason" : "forbidden"

I 2019-09-22T08:01:28.370371540Z     "message" : "Request had insufficient authentication scopes.",

I 2019-09-22T08:01:28.370364751Z     "domain" : "global",

I 2019-09-22T08:01:28.370358271Z   "errors" : [ {

I 2019-09-22T08:01:28.370351106Z   "code" : 403,

I 2019-09-22T08:01:28.370344017Z {

I 2019-09-22T08:01:28.370207921Z java.lang.RuntimeException: [test:production] **Failed to update metadata for Cloud SQL instance.**
    at com.google.cloud.sql.core.CloudSqlInstance.addExceptionContext(CloudSqlInstance.java:474)
    at com.google.cloud.sql.core.CloudSqlInstance.fetchMetadata(CloudSqlInstance.java:315)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden

Answer 1

It looks like you are running into an authentication problem:

"message" : "Request had insufficient authentication scopes."

Verify that the service account being used to the the Cloud SQL instance has the following IAM roles:

• Cloud SQL Client (preferred)
• Cloud SQL Editor
• Cloud SQL Admin

OR at least the following IAM permissions:

• cloudsql.instances.connect
• cloudsql.instances.get

(These two permissions are also in the Cloud Client role)