Search code examples
kuberneteskeycloakliquibase

KeyCloak/liquibase - container fails to start in Kubernetes

I had a workflow for Kubernetes that worked. I made some changes to it, that I then reverted back. However, now the build fails with a liquibase checksum error. Any suggestions?

Part of what I tried was to update KeyCloak. This can be seen in the databasechangelog table below. By reverting, I also revert back to an older version. Question: can I perhaps solve the error I get by manually removing the last 5 records from this table, which are the 5 records which were added because of the update I tried.

databasechangelog last records

Pod logs:

2023-10-26 07:56:29,553 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: true
2023-10-26 07:56:35,845 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2023-10-26 07:56:40,085 INFO  [org.infinispan.SERVER] (keycloak-cache-init) ISPN005054: Native IOUring transport not available, using NIO instead: io.netty.incubator.channel.uring.IOUring
2023-10-26 07:56:40,977 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-10-26 07:56:41,166 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-10-26 07:56:41,380 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-10-26 07:56:42,751 WARN  [io.quarkus.vertx.http.runtime.VertxHttpRecorder] (main) The X-Forwarded-* and Forwarded headers will be considered when determining the proxy address. This configuration can cause a security issue as clients can forge requests and send a forwarded header that is not overwritten by the proxy. Please consider use one of these headers just to forward the proxy address in requests.
2023-10-26 07:56:43,394 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2023-10-26 07:56:43,803 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-10-26 07:56:43,828 INFO  [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 2ef9b0d1-bd23-4c12-ad81-346408645302, name: ***-keycloak-cbbd985f7-nzcz9-49201
2023-10-26 07:56:43,862 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2023-10-26 07:56:43,863 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
2023-10-26 07:56:43,865 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2023-10-26 07:56:43,868 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
2023-10-26 07:56:43,903 INFO  [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.17519
2023-10-26 07:56:45,927 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) ***-keycloak-cbbd985f7-nzcz9-49201: no members discovered after 2008 ms: creating cluster as coordinator
2023-10-26 07:56:45,954 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [***-keycloak-cbbd985f7-nzcz9-49201|0] (1) [***-keycloak-cbbd985f7-nzcz9-49201]
2023-10-26 07:56:45,985 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `***-keycloak-cbbd985f7-nzcz9-49201`, physical addresses are `[10.244.0.98:33055]`
2023-10-26 07:56:48,130 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: ***-keycloak-cbbd985f7-nzcz9-49201, Site name: null
2023-10-26 07:56:48,146 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-10-26 07:56:53,545 INFO  [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
2023-10-26 07:56:53,711 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2023-10-26 07:56:53,715 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: liquibase.exception.ValidationFailedException: Validation Failed:
     110 changesets check sum
          META-INF/jpa-changelog-1.0.0.Final.xml::1.0.0.Final-KEYCLOAK-5461::sthorger@redhat.com was: 9:6f1016664e21e16d26517a4418f5e3df but is now: 8:bda77d94bf90182a1e30c24f1c155ec7
          META-INF/db2-jpa-changelog-1.0.0.Final.xml::1.0.0.Final-KEYCLOAK-5461::sthorger@redhat.com was: 9:828775b1596a07d1200ba1d49e5e3941 but is now: 8:1ecb330f30986693d1cba9ab579fa219
          META-INF/jpa-changelog-1.1.0.Beta1.xml::1.1.0.Beta1::sthorger@redhat.com was: 9:5f090e44a7d595883c1fb61f4b41fd38 but is now: 8:cb7ace19bc6d959f305605d255d4c843
          META-INF/jpa-changelog-1.1.0.Final.xml::1.1.0.Final::sthorger@redhat.com was: 9:c07e577387a3d2c04d1adc9aaad8730e but is now: 8:80230013e961310e6872e871be424a63
          META-INF/jpa-changelog-1.2.0.Beta1.xml::1.2.0.Beta1::psilva@redhat.com was: 9:b68ce996c655922dbcd2fe6b6ae72686 but is now: 8:67f4c20929126adc0c8e9bf48279d244
          META-INF/db2-jpa-changelog-1.2.0.Beta1.xml::1.2.0.Beta1::psilva@redhat.com was: 9:543b5c9989f024fe35c6f6c5a97de88e but is now: 8:7311018b0b8179ce14628ab412bb6783
          META-INF/jpa-changelog-1.2.0.CR1.xml::1.2.0.RC1::bburke@redhat.com was: 9:765afebbe21cf5bbca048e632df38336 but is now: 8:037ba1216c3640f8785ee6b8e7c8e3c1
          META-INF/db2-jpa-changelog-1.2.0.CR1.xml::1.2.0.RC1::bburke@redhat.com was: 9:db4a145ba11a6fdaefb397f6dbf829a1 but is now: 8:7fe6ffe4af4df289b3157de32c624263
          META-INF/jpa-changelog-1.2.0.Final.xml::1.2.0.Final::keycloak was: 9:9d05c7be10cdb873f8bcb41bc3a8ab23 but is now: 8:9c136bc3187083a98745c7d03bc8a303
          META-INF/jpa-changelog-1.3.0.xml::1.3.0::bburke@redhat.com was: 9:18593702353128d53111f9b1ff0b82b8 but is now: 8:b5f09474dca81fb56a97cf5b6553d331
          META-INF/jpa-changelog-1.4.0.xml::1.4.0::bburke@redhat.com was: 9:6122efe5f090e41a85c0f1c9e52cbb62 but is now: 8:ca924f31bd2a3b219fdcfe78c82dacf4
          META-INF/db2-jpa-changelog-1.4.0.xml::1.4.0::bburke@redhat.com was: 9:e1ff28bf7568451453f844c5d54bb0b5 but is now: 8:8acad7483e106416bcfa6f3b824a16cd
          META-INF/jpa-changelog-1.5.0.xml::1.5.0::bburke@redhat.com was: 9:7af32cd8957fbc069f796b61217483fd but is now: 8:9b1266d17f4f87c78226f5055408fd5e
          META-INF/jpa-changelog-1.6.1.xml::1.6.1_from15::mposolda@redhat.com was: 9:6005e15e84714cd83226bf7879f54190 but is now: 8:d80ec4ab6dbfe573550ff72396c7e910
          META-INF/jpa-changelog-1.6.1.xml::1.6.1_from16-pre::mposolda@redhat.com was: 9:bf656f5a2b055d07f314431cae76f06c but is now: 8:d86eb172171e7c20b9c849b584d147b2
          META-INF/jpa-changelog-1.6.1.xml::1.6.1_from16::mposolda@redhat.com was: 9:f8dadc9284440469dcf71e25ca6ab99b but is now: 8:5735f46f0fa60689deb0ecdc2a0dea22
          META-INF/jpa-changelog-1.6.1.xml::1.6.1::mposolda@redhat.com was: 9:d41d8cd98f00b204e9800998ecf8427e but is now: 8:d41d8cd98f00b204e9800998ecf8427e
          META-INF/jpa-changelog-1.7.0.xml::1.7.0::bburke@redhat.com was: 9:3368ff0be4c2855ee2dd9ca813b38d8e but is now: 8:5c1a8fd2014ac7fc43b90a700f117b23
          META-INF/jpa-changelog-1.8.0.xml::1.8.0::mposolda@redhat.com was: 9:8ac2fb5dd030b24c0570a763ed75ed20 but is now: 8:1f6c2c2dfc362aff4ed75b3f0ef6b331
          META-INF/jpa-changelog-1.8.0.xml::1.8.0-2::keycloak was: 9:f91ddca9b19743db60e3057679810e6c but is now: 8:dee9246280915712591f83a127665107
          META-INF/db2-jpa-changelog-1.8.0.xml::1.8.0::mposolda@redhat.com was: 9:831e82914316dc8a57dc09d755f23c51 but is now: 8:9eb2ee1fa8ad1c5e426421a6f8fdfa6a
          META-INF/db2-jpa-changelog-1.8.0.xml::1.8.0-2::keycloak was: 9:f91ddca9b19743db60e3057679810e6c but is now: 8:dee9246280915712591f83a127665107
          META-INF/jpa-changelog-1.9.0.xml::1.9.0::mposolda@redhat.com was: 9:bc3d0f9e823a69dc21e23e94c7a94bb1 but is now: 8:d9fa18ffa355320395b86270680dd4fe
          META-INF/jpa-changelog-1.9.1.xml::1.9.1::keycloak was: 9:c9999da42f543575ab790e76439a2679 but is now: 8:90cff506fedb06141ffc1c71c4a1214c
          META-INF/db2-jpa-changelog-1.9.1.xml::1.9.1::keycloak was: 9:0d6c65c6f58732d81569e77b10ba301d but is now: 8:11a788aed4961d6d29c427c063af828c
          META-INF/jpa-changelog-1.9.2.xml::1.9.2::keycloak was: 9:fc576660fc016ae53d2d4778d84d86d0 but is now: 8:a4218e51e1faf380518cce2af5d39b43
2023-10-26 07:56:53,717 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Validation Failed:
     110 changesets check sum
          META-INF/jpa-changelog-1.0.0.Final.xml::1.0.0.Final-KEYCLOAK-5461::sthorger@redhat.com was: 9:6f1016664e21e16d26517a4418f5e3df but is now: 8:bda77d94bf90182a1e30c24f1c155ec7
          META-INF/db2-jpa-changelog-1.0.0.Final.xml::1.0.0.Final-KEYCLOAK-5461::sthorger@redhat.com was: 9:828775b1596a07d1200ba1d49e5e3941 but is now: 8:1ecb330f30986693d1cba9ab579fa219
          META-INF/jpa-changelog-1.1.0.Beta1.xml::1.1.0.Beta1::sthorger@redhat.com was: 9:5f090e44a7d595883c1fb61f4b41fd38 but is now: 8:cb7ace19bc6d959f305605d255d4c843
          META-INF/jpa-changelog-1.1.0.Final.xml::1.1.0.Final::sthorger@redhat.com was: 9:c07e577387a3d2c04d1adc9aaad8730e but is now: 8:80230013e961310e6872e871be424a63
          META-INF/jpa-changelog-1.2.0.Beta1.xml::1.2.0.Beta1::psilva@redhat.com was: 9:b68ce996c655922dbcd2fe6b6ae72686 but is now: 8:67f4c20929126adc0c8e9bf48279d244
          META-INF/db2-jpa-changelog-1.2.0.Beta1.xml::1.2.0.Beta1::psilva@redhat.com was: 9:543b5c9989f024fe35c6f6c5a97de88e but is now: 8:7311018b0b8179ce14628ab412bb6783
          META-INF/jpa-changelog-1.2.0.CR1.xml::1.2.0.RC1::bburke@redhat.com was: 9:765afebbe21cf5bbca048e632df38336 but is now: 8:037ba1216c3640f8785ee6b8e7c8e3c1
          META-INF/db2-jpa-changelog-1.2.0.CR1.xml::1.2.0.RC1::bburke@redhat.com was: 9:db4a145ba11a6fdaefb397f6dbf829a1 but is now: 8:7fe6ffe4af4df289b3157de32c624263
          META-INF/jpa-changelog-1.2.0.Final.xml::1.2.0.Final::keycloak was: 9:9d05c7be10cdb873f8bcb41bc3a8ab23 but is now: 8:9c136bc3187083a98745c7d03bc8a303
          META-INF/jpa-changelog-1.3.0.xml::1.3.0::bburke@redhat.com was: 9:18593702353128d53111f9b1ff0b82b8 but is now: 8:b5f09474dca81fb56a97cf5b6553d331
          META-INF/jpa-changelog-1.4.0.xml::1.4.0::bburke@redhat.com was: 9:6122efe5f090e41a85c0f1c9e52cbb62 but is now: 8:ca924f31bd2a3b219fdcfe78c82dacf4
          META-INF/db2-jpa-changelog-1.4.0.xml::1.4.0::bburke@redhat.com was: 9:e1ff28bf7568451453f844c5d54bb0b5 but is now: 8:8acad7483e106416bcfa6f3b824a16cd
          META-INF/jpa-changelog-1.5.0.xml::1.5.0::bburke@redhat.com was: 9:7af32cd8957fbc069f796b61217483fd but is now: 8:9b1266d17f4f87c78226f5055408fd5e
          META-INF/jpa-changelog-1.6.1.xml::1.6.1_from15::mposolda@redhat.com was: 9:6005e15e84714cd83226bf7879f54190 but is now: 8:d80ec4ab6dbfe573550ff72396c7e910
          META-INF/jpa-changelog-1.6.1.xml::1.6.1_from16-pre::mposolda@redhat.com was: 9:bf656f5a2b055d07f314431cae76f06c but is now: 8:d86eb172171e7c20b9c849b584d147b2
          META-INF/jpa-changelog-1.6.1.xml::1.6.1_from16::mposolda@redhat.com was: 9:f8dadc9284440469dcf71e25ca6ab99b but is now: 8:5735f46f0fa60689deb0ecdc2a0dea22
          META-INF/jpa-changelog-1.6.1.xml::1.6.1::mposolda@redhat.com was: 9:d41d8cd98f00b204e9800998ecf8427e but is now: 8:d41d8cd98f00b204e9800998ecf8427e
          META-INF/jpa-changelog-1.7.0.xml::1.7.0::bburke@redhat.com was: 9:3368ff0be4c2855ee2dd9ca813b38d8e but is now: 8:5c1a8fd2014ac7fc43b90a700f117b23
          META-INF/jpa-changelog-1.8.0.xml::1.8.0::mposolda@redhat.com was: 9:8ac2fb5dd030b24c0570a763ed75ed20 but is now: 8:1f6c2c2dfc362aff4ed75b3f0ef6b331
          META-INF/jpa-changelog-1.8.0.xml::1.8.0-2::keycloak was: 9:f91ddca9b19743db60e3057679810e6c but is now: 8:dee9246280915712591f83a127665107
          META-INF/db2-jpa-changelog-1.8.0.xml::1.8.0::mposolda@redhat.com was: 9:831e82914316dc8a57dc09d755f23c51 but is now: 8:9eb2ee1fa8ad1c5e426421a6f8fdfa6a
          META-INF/db2-jpa-changelog-1.8.0.xml::1.8.0-2::keycloak was: 9:f91ddca9b19743db60e3057679810e6c but is now: 8:dee9246280915712591f83a127665107
          META-INF/jpa-changelog-1.9.0.xml::1.9.0::mposolda@redhat.com was: 9:bc3d0f9e823a69dc21e23e94c7a94bb1 but is now: 8:d9fa18ffa355320395b86270680dd4fe
          META-INF/jpa-changelog-1.9.1.xml::1.9.1::keycloak was: 9:c9999da42f543575ab790e76439a2679 but is now: 8:90cff506fedb06141ffc1c71c4a1214c
          META-INF/db2-jpa-changelog-1.9.1.xml::1.9.1::keycloak was: 9:0d6c65c6f58732d81569e77b10ba301d but is now: 8:11a788aed4961d6d29c427c063af828c
          META-INF/jpa-changelog-1.9.2.xml::1.9.2::keycloak was: 9:fc576660fc016ae53d2d4778d84d86d0 but is now: 8:a4218e51e1faf380518cce2af5d39b43

My workflow:

name: Build and deploy Keycloak

on:
  workflow_dispatch:
    inputs:
      logLevel:
        description: 'Log level'
        required: true
        default: 'warning'

jobs:
  build_and_release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2
      - name: Install doctl
        uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
      - name: Build container image
        run: docker build -t registry.digitalocean.com/***/***-keycloak:$(echo $GITHUB_SHA | head -c7) -f docker/Keycloak.Dockerfile docker
      - name: Log in to DigitalOcean Container Registry with short-lived credentials
        run: doctl registry login --expiry-seconds 1200
      - name: Push image to DigitalOcean Container Registry
        run: docker push registry.digitalocean.com/***/***-keycloak:$(echo $GITHUB_SHA | head -c7)

      - name: Update deployment file
        run: TAG=$(echo $GITHUB_SHA | head -c7) && sed -i 's|<TAG>|'${TAG}'|' $GITHUB_WORKSPACE/helm/***-keycloak/values.yaml
      - name: Save DigitalOcean kubeconfig with short-lived credentials
        run: doctl kubernetes cluster kubeconfig save --expiry-seconds 600 k8s-***-production
      - name: Deploy to DigitalOcean Kubernetes
        run: sed -i "s,VAL_KC_DB_PASSWORD,${{ secrets.INFRA_KC_DB_PASSWORD }}," $GITHUB_WORKSPACE/helm/***-keycloak/values.yaml && helm upgrade ***-keycloak $GITHUB_WORKSPACE/helm/***-keycloak/ --values $GITHUB_WORKSPACE/helm/***-keycloak/values.yaml --install --namespace *** --create-namespace
      - name: Verify deployment
        run: kubectl rollout status deployment/***-keycloak --namespace *** --timeout=30s
Solution

  • The cause turned out to be that in our docker file I updated KeyCloak and then downgraded again. But once the update was done, the databasechangelog table expected the newer version of KeyCloak (so that did not downgrade when I downgraded in the docker file).

    By specifying in the docker file the latest version that databasechangelog expected "solved" it.