In a Django project, I have a model Question in the lucy_web app, but the corresponding lucy_web_question table does not exist, as seen from a \dt command in the database shell:
(lucy-web-CVxkrCFK) bash-3.2$ python manage.py dbshell
psql (10.4)
Type "help" for help.
lucy=> \dt
List of relations
Schema | Name | Type | Owner
--------+------------------------------+-------+---------
public | auditlog_logentry | table | lucyapp
public | auth_group | table | lucyapp
public | auth_group_permissions | table | lucyapp
public | auth_permission | table | lucyapp
public | auth_user | table | lucyapp
public | auth_user_groups | table | lucyapp
public | auth_user_user_permissions | table | lucyapp
public | defender_accessattempt | table | lucyapp
public | django_admin_log | table | lucyapp
public | django_content_type | table | lucyapp
public | django_migrations | table | lucyapp
public | django_session | table | lucyapp
public | lucy_web_checkin | table | lucyapp
public | lucy_web_checkintype | table | lucyapp
public | lucy_web_company | table | lucyapp
public | lucy_web_expert | table | lucyapp
public | lucy_web_expertsessiontype | table | lucyapp
public | lucy_web_family | table | lucyapp
public | lucy_web_lucyguide | table | lucyapp
public | lucy_web_notification | table | lucyapp
public | lucy_web_package | table | lucyapp
public | lucy_web_packagesessiontype | table | lucyapp
public | lucy_web_preactivationfamily | table | lucyapp
public | lucy_web_profile | table | lucyapp
public | lucy_web_questionanswer | table | lucyapp
public | lucy_web_questioncategory | table | lucyapp
public | lucy_web_session | table | lucyapp
public | lucy_web_sessioncategory | table | lucyapp
public | lucy_web_sessiontype | table | lucyapp
public | lucy_web_userapn | table | lucyapp
public | oauth2_provider_accesstoken | table | lucyapp
public | oauth2_provider_application | table | lucyapp
public | oauth2_provider_grant | table | lucyapp
public | oauth2_provider_refreshtoken | table | lucyapp
public | otp_static_staticdevice | table | lucyapp
public | otp_static_statictoken | table | lucyapp
public | otp_totp_totpdevice | table | lucyapp
public | two_factor_phonedevice | table | lucyapp
(38 rows)
We also have a staging environment, deployed on Aptible, which does appear to have these tables. Using the Aptible CLI to create a database tunnel, if I psql <connection_url> and dt I do see the lucy_web_question table:
db=# \dt
List of relations
Schema | Name | Type | Owner
--------+------------------------------+-------+---------
public | auditlog_logentry | table | aptible
public | auth_group | table | aptible
public | auth_group_permissions | table | aptible
public | auth_permission | table | aptible
public | auth_user | table | aptible
public | auth_user_groups | table | aptible
public | auth_user_user_permissions | table | aptible
public | defender_accessattempt | table | aptible
public | django_admin_log | table | aptible
public | django_content_type | table | aptible
public | django_migrations | table | aptible
public | django_session | table | aptible
public | lucy_web_checkin | table | aptible
public | lucy_web_checkintype | table | aptible
public | lucy_web_company | table | aptible
public | lucy_web_expert | table | aptible
public | lucy_web_expertsessiontype | table | aptible
public | lucy_web_family | table | aptible
public | lucy_web_lucyguide | table | aptible
public | lucy_web_notification | table | aptible
public | lucy_web_package | table | aptible
public | lucy_web_packagesessiontype | table | aptible
public | lucy_web_preactivationfamily | table | aptible
public | lucy_web_profile | table | aptible
public | lucy_web_question | table | aptible
public | lucy_web_questionanswer | table | aptible
public | lucy_web_questioncategory | table | aptible
public | lucy_web_questionprompt | table | aptible
public | lucy_web_session | table | aptible
public | lucy_web_sessioncategory | table | aptible
public | lucy_web_sessiontype | table | aptible
public | lucy_web_userapn | table | aptible
public | oauth2_provider_accesstoken | table | aptible
public | oauth2_provider_application | table | aptible
public | oauth2_provider_grant | table | aptible
public | oauth2_provider_refreshtoken | table | aptible
public | otp_static_staticdevice | table | aptible
public | otp_static_statictoken | table | aptible
public | otp_totp_totpdevice | table | aptible
public | two_factor_phonedevice | table | aptible
(40 rows)
Because the data on these test environments is not important, I'd like to pg_dump the Aptible database and pg_restore it on my local machine.
My local DATABASE_URL is postgres://lucyapp:<my_password>@localhost/lucy, so firstly, I did a pg_dump with --format=custom and specifying a --file as follows:
Kurts-MacBook-Pro-2:lucy2 kurtpeek$ touch staging_db_12_July.dump
Kurts-MacBook-Pro-2:lucy2 kurtpeek$ pg_dump postgresql://aptible:<some_aptible_hash>@localhost.aptible.in:62288/db --format=custom --file=staging_db_12_July.dump
Kurts-MacBook-Pro-2:lucy2 kurtpeek$ ls -lhtr | tail -1
-rw-r--r-- 1 kurtpeek staff 1.5M Jul 12 18:09 staging_db_12_July.dump
This results in a 1.5Mb .dump file, which I tried to restore from using pg_restore with the --no-owner option and --role=lucyapp (in order change the owner from aptible to lucyapp). However, this gives rise to a large number of 'already exists' errors, of which one is shown below:
Kurts-MacBook-Pro-2:lucy2 kurtpeek$ pg_restore staging_db_12_July.dump --dbname=lucy --no-owner --role=lucyapp
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3522; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension plpgsql
Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
pg_restore: [archiver (db)] Error from TOC entry 2; 3079 16392 EXTENSION hstore
pg_restore: [archiver (db)] could not execute query: ERROR: permission denied to create extension "hstore"
HINT: Must be superuser to create this extension.
Command was: CREATE EXTENSION IF NOT EXISTS hstore WITH SCHEMA public;
pg_restore: [archiver (db)] Error from TOC entry 3523; 0 0 COMMENT EXTENSION hstore
pg_restore: [archiver (db)] could not execute query: ERROR: extension "hstore" does not exist
Command was: COMMENT ON EXTENSION hstore IS 'data type for storing sets of (key, value) pairs';
pg_restore: [archiver (db)] Error from TOC entry 197; 1259 16515 TABLE auditlog_logentry aptible
pg_restore: [archiver (db)] could not execute query: ERROR: relation "auditlog_logentry" already exists
Command was: CREATE TABLE public.auditlog_logentry (
id integer NOT NULL,
object_pk character varying(255) NOT NULL,
object_id bigint,
object_repr text NOT NULL,
action smallint NOT NULL,
changes text NOT NULL,
"timestamp" timestamp with time zone NOT NULL,
actor_id integer,
content_type_id integer NOT NULL,
remote_addr inet,
additional_data jsonb,
CONSTRAINT auditlog_logentry_action_check CHECK ((action >= 0))
);
WARNING: errors ignored on restore: 294
The problem is that if I \dt again in the python manage.py dbshell, I still don't see the lucy_web_question table.
I've come across this solution, Django : Table doesn't exist, for my situation, but in my case the Question model is imported and used as a foreign key in so many places that I thought it would be easier just to restore a database. Why is it not restoring the lucy_web_question table, though?
It seems the problem was that the lucyapp user did not have sufficient privileges to create the table. I basically had to ensure that the \dn+ command produced this result:
lucy=# \dn+
List of schemas
Name | Owner | Access privileges | Description
--------+----------+----------------------+------------------------
public | postgres | postgres=UC/postgres+| standard public schema
| | =UC/postgres +|
| | lucyapp=UC/postgres |
(1 row)
where lucyapp has both USAGE (U) and CREATE (C) privileges. Following https://www.postgresql.org/docs/9.0/static/sql-grant.html, this can be achieved with the commands
GRANT USAGE ON SCHEMA public TO lucyapp;
GRANT CREATE ON SCHEMA public TO lucyapp;
I also made lucyapp a superuser prior to running these commands, although that is not recommended for production.