Docker + Django - Make SQLite3 database really persistent

Question

I want to make my SQLite3 database persistent even if I would run commands like docker compose down stop start up and build.

This is my setup:

Dockerfile_DEV:

# Base image
FROM python:3.10-slim-buster

# Set working directory
WORKDIR /app

# Copy the entire project directory
COPY . /app

# Install project dependencies
RUN pip install -r requirements_dev.txt

# Expose port
EXPOSE 8000

# Run the development server
CMD ["python", "doc_manager/manage.py", "runserver", "0.0.0.0:8000"]

docker-compose-DEV.yml:

version: '3'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile_DEV
    ports:
      - "8000:8000"
    volumes:
      - ./:/app/doc-manager-web
      - data:/app/doc-manager-web/data
    
volumes:
  data:

Directory tree:

doc-manager-web  # Main project's folder (in here there's also Dockerfile_DEV and docker-compose-DEV.yml)
├───data  # Folder containing my SQLite3 database file
├───documentation
└───doc_manager  # Django project
    ├───core
    │   ├───migrations
    │   │   └───__pycache__
    │   ├───templates
    │   │   └───core
    │   └───__pycache__
    ├───doc_manager
    │   └───__pycache__
    └───media
        └───documents

Even though I have set up the volume in my docker-compose-DEV.yml file, it does not seem to work. The SQLite3 database will always get rolled back to it's initial state whenever I try to run docker compose down and then docker compose up.

What should I do to make it really persistent so that I can re-build and re-deploy my whole Django project using Docker whenever I make some code changes but without any changes done to the SQLite3 database file?

Answer 1

Your image's WORKDIR is set to /app. If your SQLite database is in a data subdirectory, it will be in /app/data. You're actually mounting the data volume to a different directory; so in reality the SQLite database is being stored in the container temporary filesystem, and that's why you're seeing data be lost on container restarts.

You need to fix the volumes: so that the data volume is mounted on the correct directory. Do not try to bind-mount the code into the container: it is already in the image, and you can get surprising results if the filesystem layout is substantially different between what's in the image and what's in the host system.

volumes:
  - data:/app/data
  #      ^^^^^^^^^ not in a subdirectory that happens to match the host directory name
  # no bind mount of all of ./

You may need to make sure your code takes steps required to initialize the database if it does not exist, for example running SQL CREATE TABLE instructions. Docker named volumes have special initialization behavior that no other volume type has. If you try to run this in Kubernetes, or bind-mount a host directory, you will start off with an empty directory. If your application framework has a migration system, that can be a good approach.