Search code examples
postgresqldockerubuntuamazon-ec2

Postgres container failed to start with initdb error. popen failure: Cannot allocate memory


I'm using postgres:12 Docker image on AWS instance under Ubuntu 20.04.

  postgres-tests:
    image: "postgres:12"
    restart: always
    command: postgres -c 'max_connections=200'
    environment:
      POSTGRES_DB: "${POSTGRES_DATABASE}"
      POSTGRES_USER: "${POSTGRES_USER}"
      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
    ports:
      - "8396:5432"

When running this container with docker-compose up -d it fails to start with the following error:

postgres-tests_1  | popen failure: Cannot allocate memory
postgres-tests_1  | initdb: error: The program "postgres" is needed by initdb but was not found in the
postgres-tests_1  | same directory as "/usr/lib/postgresql/12/bin/initdb".
postgres-tests_1  | Check your installation.

The error appeared suddenly after most-resent project deploy. The important thing is that the error happens only with this particular container. There is one more postgresql:12 container on the machine for another project, which works fine.

HERE IS WHAT I TRIED:

  1. I've found several suggestions related to increasing the shmall/shmmax params controlling shared memory on the machine.

But these system params are already set to high value:

ubuntu@ip-172-31-10-246:/var/www$ cat /proc/sys/kernel/shmall
18446744073692774399
ubuntu@ip-172-31-10-246:/var/www$ cat /proc/sys/kernel/shmmax
18446744073692774399
ubuntu@ip-172-31-10-246:/var/www$ 

  1. The second suggested option was to try a newer postgres image. Tested with postgres 13.0, 14.0 with no effect. UPDATE Tried with postgres:11 image and it runs OK, but I can not roll down postgres version in production, so it's not a solution in my case.

  2. I tried to stop/start and reboot the instance, also cleaning up docker cache with docker system prune and docker volumes prune.

Software:
Ubuntu 20.04.2
Docker version 20.10.8, build 3967b7d

Instance hardware:
Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz 
RAM:  4/8GB Used by system and other services
Swap: 4.5/20GB Used by system and other services

Solution

  • Maintainers of the postgres docker images have updated the underlying OS image layer:

    Previous: Debian 11 (bullseye)
    New: Debian 12 (bookworm)

    This was pushed up to hub.docker.com on 15 Jun 2023. The move to this container is causing a lot of builds to throw errors and exceptions like the one noted in the question:

    • initdb: error: The program "postgres" is needed by initdb but was not found in the same directory as "/usr/lib/postgresql/12/bin/initdb".

    The mitigation route is to use the following conatiner:

    • postgres:12-bullseye

    The maintainers will keep providing bullseye variants until the Postgres version is EOL or Debian moves to their latest Trixie

    Issues can be found here:

    1. Postgres:12 container is not starting anymore #1103
    2. Postgres container failing with initdb error: program “postgres” is needed by initdb but was not found