I implemented a short lived docker container that runs a Liquibase migration against a mysql database. The container has been working great during development via docker-compose. However, I'm attempting to implement this container within a DCOS cluster in preparation for production.
When Liquibase runs, it successfully acquires a lock and then just dies. Here is the output of the run:
DEBUG 4/14/17 8:06 PM: liquibase: Connected to root@10.0.1.113@jdbc:mysql://test-db.marathon.l4lb.thisdcos.directory/myapp
DEBUG 4/14/17 8:06 PM: liquibase: Setting auto commit to false from true
DEBUG 4/14/17 8:06 PM: liquibase: Executing QUERY database command: select count(*) from myapp.DATABASECHANGELOGLOCK
DEBUG 4/14/17 8:06 PM: liquibase: Executing QUERY database command: select count(*) from myapp.DATABASECHANGELOGLOCK
INFO 4/14/17 8:06 PM: liquibase: Successfully acquired change log lock
/usr/local/bin/liquibase: line 56: 295 Killed java -cp "$CP" $JAVA_OPTS liquibase.integration.commandline.Main ${1+"$@"}
The DATABASECHANGELOGLOCK table is successfully created so I know it is both accessing the database and creating tables, however it just dies.
After the run, the lock table looks like:
MySQL [myapp]> select * from DATABASECHANGELOGLOCK;
+----+--------+-------------+----------+
| ID | LOCKED | LOCKGRANTED | LOCKEDBY |
+----+--------+-------------+----------+
| 1 | | NULL | NULL |
+----+--------+-------------+----------+
Here is the command that is being run:
liquibase --changeLogFile=base_change_log.xml --driver=com.mysql.jdbc.Driver --url=jdbc:mysql://test-db.marathon.l4lb.thisdcos.directory/myapp --contexts=production --username=root --password=****** --classpath=/opt/jdbc_drivers/mysql-connector-java-5.1.40-bin.jar --logLevel=debug update
This is the command the liquibase bash file runs:
java -cp .:/opt/liquibase/liquibase.jar:/opt/liquibase/lib/snakeyaml-1.17.jar liquibase.integration.commandline.Main --changeLogFile=base_change_log.xml --driver=com.mysql.jdbc.Driver --url=jdbc:mysql://test-db.marathon.l4lb.thisdcos.directory/myapp --contexts=production --username=root --password=****** --classpath=/opt/jdbc_drivers/mysql-connector-java-5.1.40-bin.jar --logLevel=debug update
As I said, this container works fine locally with docker-compose, it also works if I install all deps on one of the dcos slaves and run liquibase locally.
Are there any better ways to debug this? Has anyone successfully run a liquibase migration inside a DCOS cluster?
Please check the Mesos master logs. I suspect you are encountering a OOM kill of the task.