Search code examples
jenkins-pipelineflyway

Running a Flyway migration via Jenkins shell does not run all version files available on the locations given

I have a Jenkins pipeline that has a shell to run a Flyway migration. The shell line that runs the flyway migration looks like:

$(${FLYWAY_PATH} migrate -url=jdbc:postgresql://${RDS_HOST}/${RDS_DB} -user=deployer -password=${DEPLOYER_INFO} -table=${FLYWAY_TABLE} -locations=filesystem:./${FLYWAY_SOURCE} -createSchemas=true -baselineOnMigrate=true -schemas=${FLYWAY_SCHEMA} -placeholders.db_schema=${FLYWAY_SCHEMA} -placeholders.db_user_pw=${FLYWAY_PLACEHOLDER} -placeholders.db_create_named_user=${FLYWAY_CREATE_USERS} -placeholders.deployer_pwd=${DEPLOYER_INFO} -validateMigrationNaming=true)

which translate to something similar to:

/usr/local/bin/flyway migrate -url=jdbc:postgresql://vbs.us-east-1.rds.amazonaws.com/finalp -user=deployer -password=Tqj -table=flyway_schema_history -locations=filesystem:./files/schema -createSchemas=true -baselineOnMigrate=true -schemas=_194_jenkins_beta -placeholders.db_schema=_194_jenkins_beta -placeholders.db_user_pw=3HVB -placeholders.db_create_named_user=true -placeholders.deployer_pwd=vTqj

If I run the above line directly on the server it works great, all files are used:

Migrating schema "_194_jenkins_beta" to version "001.01.00 - transfer ownership"
WARNING: DB: role "role_dba" already exists, skipping (SQL State: 42710 - Error Code: 0)
WARNING: DB: role "deployer" already exists, skipping (SQL State: 42710 - Error Code: 0)
WARNING: DB: no privileges were granted for "finalp" (SQL State: 01007 - Error Code: 0)
DB: role "deployer" is already a member of role "role_dba"
Migrating schema "_194_jenkins_beta" to version "001.01.01 - records"
Migrating schema "_194_jenkins_beta" to version "001.01.02 - etl"
Migrating schema "_194_jenkins_beta" to version "001.01.03 - data"
Migrating schema "_194_jenkins_beta" to version "001.02.00 - alert author day etl"
Migrating schema "_194_jenkins_beta" to version "001.02.01 - alert dismissreason etl"
Migrating schema "_194_jenkins_beta" with repeatable migration "50 author group rule"
Migrating schema "_194_jenkins_beta" with repeatable migration "50 group rules"

But when I run my pipeline in Jenkins, it doesn't run all the version files:

Migrating schema "_194_jenkins_beta" to version "001.01.00 - transfer ownership"
WARNING: DB: role "role_dba" already exists, skipping (SQL State: 42710 - Error Code: 0)
WARNING: DB: role "deployer" already exists, skipping (SQL State: 42710 - Error Code: 0)
WARNING: DB: no privileges were granted for "finalp" (SQL State: 01007 - Error Code: 0)
DB: role "deployer" is already a member of role "role_dba"
Migrating schema "_194_jenkins_beta" with repeatable migration "50 author group rules"
Caused by: Migration R__50_group_rules.sql failed
15:54:29  ---------------------------------------------------------------
15:54:29  SQL State  : 42703
15:54:29  Error Code : 0
15:54:29  Message    : ERROR: column f.last_alert_groups does not exist

Which is correct as that column is added on version file "001.02.00", so basically it is omitting the following files:

  • Migrating schema "_194_jenkins_beta" to version "001.01.01 - records"
  • Migrating schema "_194_jenkins_beta" to version "001.01.02 - etl"
  • Migrating schema "_194_jenkins_beta" to version "001.01.03 - data"
  • Migrating schema "_194_jenkins_beta" to version "001.02.00 - alert author day etl"
  • Migrating schema "_194_jenkins_beta" to version "001.02.01 - alert dismissreason etl"

Notes:

  1. I am printing the command that is executed in Jenkins and using exactly that string on the Jenkins Node to run it manually. Manually works, from Jenkins has the above files missing.
  2. I am manually running the command exactly from the same location as Jenkins does, same files are available, so there is no room to have "missing" files as both techniques are using the same source.
  3. The only interesting thing is that I have all the versioned files in ./files/schema/tables EXCEPT for one "001.01.00", which is located on ./files/schema/scripts, but everything looks fine when running manually, I do not think it may affect Jenkins shell execution.
  4. I am aware there is a Flyway plug-in, but I am not allow to install it and the administrators are hesitant to do it as they have too many things and adding a plug-in means a lot of tests on their side.

Any idea of why this may be happening and possible work around?

I am expecting to have all version (V) files executed from Jenkins as they are being done when tunning same command manually.

Thanks you.

Solution

  • Looks like if I add -target works in Jenkins -- not an ideal solution, but I can move on for now. Either way, very interesting to see the different behaviors....