Please note: my question mentions MySQL, but it is a Docker/Docker Compose volume management question at heart, and as such, should be answerable by anyone with decent experience in that area, regardless of their familiarity with MySQL.
My understanding is that Dockerized MySQL containers, when defined from inside a Docker Compose file like below, will be ephemeral, meaning they store all data on the container itself (no bind mounts, etc.) and so when the container dies, the data is gone as well:
version: "3.7"
services:
my-service-db:
image: mysql:8
container_name: $MY_SERVICE_DB_HOST
command: --default-authentication-plugin=mysql_native_password
restart: always
ports:
- $MY_SERVICE_DB_PORT:$MY_SERVICE_DB_PORT
environment:
MYSQL_ROOT_PASSWORD: $MY_SERVICE_DB_ROOT_PASSWORD
MYSQL_DATABASE: my_service_db_$MY_ENV
MYSQL_USER: $MY_SERVICE_DB_APP_USER
MYSQL_PASSWORD: $MY_SERVICE_DB_APP_PASSWORD
other-service-definitions-omitted-for-brevity:
- etc.
To begin with, if that understanding is incorrect, please begin by correcting me! Assuming its more or less correct...
Lets call this Ephemeral Mode.
But by providing a bind mount volume to that service definition, we can specify an external location for where data should be stored, and so the data will persist across service runs (compose ups/downs):
version: "3.7"
services:
my-service-db:
image: mysql:8
container_name: $MY_SERVICE_DB_HOST
command: --default-authentication-plugin=mysql_native_password
restart: always
ports:
- $MY_SERVICE_DB_PORT:$MY_SERVICE_DB_PORT
environment:
MYSQL_ROOT_PASSWORD: $MY_SERVICE_DB_ROOT_PASSWORD
MYSQL_DATABASE: my_service_db_$MY_ENV
MYSQL_USER: $MY_SERVICE_DB_APP_USER
MYSQL_PASSWORD: $MY_SERVICE_DB_APP_PASSWORD
volumes:
- ./my-service-db-data:/var/lib/mysql
other-service-definitions-omitted-for-brevity:
- etc.
Lets call this Persistent Mode.
There are times when I will want to run my Docker Compose file in Ephemeral Mode, and other times, run it in Persistent Mode.
Is it possible to make the volumes definition (inside the Docker Compose file) conditonal somehow? So that sometimes I can run docker-compose up -d <SPECIFY_EPHEMERAL_MODE_SOMEHOW>, and other times I can run docker-compose up -d <SPECIFY_PERSISTENT_MODE_SOMEHOW>?
You can have multiple Compose files that work together, where you have some base file and then other files that extend the definitions in the base file.
Without extra setup, Compose looks for docker-compose.override.yml alongside the main docker-compose.yml. Since the only difference between the "ephemeral" and "persistent" mode is the volumes: declaration, you can have an override file that only contains that:
# docker-compose.override.yml
version: '3.8'
services:
my-service-db: # matches main docker-compose.yml
volumes: # added to base definition
- ./my-service-db-data:/var/lib/mysql
You could also use this technique to move the actual database credentials and port publishing out of the main file into deploy-specific configuration. It's also somewhat common to use it for setups that need to run a known Docker image in production but build it in development, and for setups that overwrite the container's contents with a host directory.
If you want the file to be named something else, you can, but you need to consistently provide a docker-compose -f option or set the COMPOSE_FILE environment variable every time you run Compose.
docker-compose -f docker-compose.yml -f docker-compose.persistence.yml up -d
docker-compose -f docker-compose.yml -f docker-compose.persistence.yml ps
docker-compose -f docker-compose.yml -f docker-compose.persistence.yml logs app
# Slightly easier (Linux syntax):
export COMPOSE_FILE=docker-compose.yml:docker-compose.persistence.yml
docker-compose up -d
Philosophically, your application's data needs to be persisted somewhere. For application containers, a good practice is for them to be totally stateless (they do not mount volumes:) and push all of their data into a database. That means the database needs to persist data, or else it will get lost when the database restarts.
IME it's a little bit unusual to actively want the database to lose data. This would be more interesting if it were straightforward to create a database image with seeded data, but the standard images are built in a way that makes this difficult. In a test environment, still, I could see wanting it.
It's actually possible, and reasonable, to build an application that runs in Docker but uses an external database. Perhaps you're running in a cloud environment, and your cloud provider has a slightly pricey managed database service that provides automatic snapshots and failover, for example; you could configure your production application to use this managed database and keep no data in containers at all.