My Postgresql database stopped working, which has shut down all of my Django applications that use it. I tried restarting with 'service postgresql restart', but it gives me this response
* Starting PostgreSQL 9.3 database server
* The PostgreSQL server failed to start. Please check the log output.
I checked the logs at /var/log/postgresql, but the most recent logs file has nothing in it. I checked the one before that, and I don't think anything in particular besides a lot of errors that say there is no disc space.
/var/log/postgresql/postgresql-9.3-main.log.1
2018-03-15 08:46:09 CDT LOG: could not write temporary statistics file "pg_stat_tmp/db_12061.tmp": No space left on device
2018-03-15 08:46:09 CDT LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2018-03-15 08:46:09 CDT LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2018-03-15 08:46:10 CDT LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2018-03-15 08:46:10 CDT LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2018-03-15 08:46:15 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-15 08:46:15 CDT HINT: Check free disk space.
2018-03-15 08:46:15 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1606764, 'https://content.homenetiol.com/2001722/2120878/640x480/43d0a25930554cdbbbd0834c9803e5ad.jpg', 1) RET$
2018-03-15 08:46:19 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-15 08:46:20 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-16 08:45:49 CDT LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2018-03-16 08:45:49 CDT LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2018-03-16 08:45:53 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-16 08:45:53 CDT HINT: Check free disk space.
2018-03-16 08:45:53 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1587463, 'https://content.homenetiol.com/2001722/2120878/640x480/c80fb80f8dc04c2581872ad2cf221cbc.jpg', 1) RET$
2018-03-16 08:45:59 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-17 08:45:48 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-17 08:45:48 CDT HINT: Check free disk space.
2018-03-17 08:45:48 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1144699, 'https://content.homenetiol.com/2001722/2120878/640x480/4e7711dd0d5148eeaadd1e8a7f00b0eb.jpg', 1) RET$
2018-03-18 08:45:44 CDT LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2018-03-18 08:45:44 CDT LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2018-03-18 08:45:47 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-18 08:45:47 CDT HINT: Check free disk space.
2018-03-18 08:45:47 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1144694, 'https://content.homenetiol.com/2001722/2120878/640x480/3dee837dd31942c5a26bc73775f07dba.jpg', 0) RET$
2018-03-18 08:45:54 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-18 10:21:40 CDT FATAL: password authentication failed for user "socialauto"
2018-03-18 10:21:40 CDT DETAIL: Connection matched pg_hba.conf line 17: "local all all md5"
2018-03-15 08:46:15 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-15 08:46:15 CDT HINT: Check free disk space.
2018-03-15 08:46:15 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1606764, 'https://content.homenetiol.com/2001722/2120878/640x480/43d0a25930554cdbbbd0834c9803e5ad.jpg', 1) RET$
2018-03-15 08:46:19 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-15 08:46:20 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-16 08:45:49 CDT LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2018-03-16 08:45:49 CDT LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2018-03-16 08:45:53 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-16 08:45:53 CDT HINT: Check free disk space.
2018-03-16 08:45:53 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1587463, 'https://content.homenetiol.com/2001722/2120878/640x480/c80fb80f8dc04c2581872ad2cf221cbc.jpg', 1) RET$
2018-03-16 08:45:59 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-17 08:45:48 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-17 08:45:48 CDT HINT: Check free disk space.
2018-03-17 08:45:48 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1144699, 'https://content.homenetiol.com/2001722/2120878/640x480/4e7711dd0d5148eeaadd1e8a7f00b0eb.jpg', 1) RET$
2018-03-18 08:45:44 CDT LOG: could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No space left on device
2018-03-18 08:45:44 CDT LOG: could not close temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2018-03-18 08:45:47 CDT ERROR: could not extend file "base/16385/616778.1": No space left on device
2018-03-18 08:45:47 CDT HINT: Check free disk space.
2018-03-18 08:45:47 CDT STATEMENT: INSERT INTO "vehicle_vehicleimage" ("vehicle_id", "image_url", "display_sequence") VALUES (1144694, 'https://content.homenetiol.com/2001722/2120878/640x480/3dee837dd31942c5a26bc73775f07dba.jpg', 0) RET$
2018-03-18 08:45:54 CDT LOG: using stale statistics instead of current ones because stats collector is not responding
2018-03-18 10:21:40 CDT FATAL: password authentication failed for user "socialauto"
I checked the disc space on the virtual machine hosting the database with the 'df' command, and it does indeed look like all the space has been used up
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 49300032 49283648 0 100% /
devtmpfs 2014088 4 2014084 1% /dev
none 4 0 4 0% /sys/fs/cgroup
none 403388 340 403048 1% /run
none 5120 0 5120 0% /run/lock
none 2016924 0 2016924 0% /run/shm
none 102400 0 102400 0% /run/user
I also get this weird message whenever I try to autocomplete a path with tab
-bash: cannot create temp file for here-document: No space left on device
Is the only way to get Postgres up and running to increase the disc space on my machine. Is there any way to use 'service postgresql restart' with some debug flag to see the error its facing without having to go into a log file ?
You can strace the server process ...
But seriously — if you are out of space, that is the reason. If PostgreSQL cannot allocate a new transaction log, it will refuse to start.