Search code examples
apache-superset

Apache superset db upgrade command failing with sqlalchemy.exc.OperationalError


I am trying to install Apache superset on Centos7 machine with python3.8. On executing superset db upgrade command it throws below error.

INFO  [alembic.runtime.migration] Running upgrade 06e1e70058c7 -> a39867932713, query_context_to_mediumtext
INFO  [alembic.runtime.migration] Running upgrade a39867932713 -> 409c7b420ab0, add created_by_fk as owner
INFO  [alembic.runtime.migration] Running upgrade 409c7b420ab0 -> ffa79af61a56, rename report_schedule.extra to extra_json
Traceback (most recent call last):
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: near "extra": syntax error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/masayyed/superset/venv/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset==2.1.0', 'console_scripts', 'superset')())
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/flask/cli.py", line 567, in main
    return super().main(*args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/flask/cli.py", line 407, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/command.py", line 378, in upgrade
    script.run_env()
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/script/base.py", line 576, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/superset/extensions/../migrations/env.py", line 126, in <module>
    run_migrations_online()
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/superset/extensions/../migrations/env.py", line 118, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/runtime/environment.py", line 868, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/runtime/migration.py", line 622, in run_migrations
    step.migration_fn(**kw)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/superset/migrations/versions/2022-07-11_11-26_ffa79af61a56_rename_report_schedule_extra_to_extra_.py", line 36, in upgrade
    op.alter_column(
  File "<string>", line 8, in alter_column
  File "<string>", line 3, in alter_column
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/operations/ops.py", line 1879, in alter_column
    return operations.invoke(alt)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/operations/base.py", line 401, in invoke
    return fn(self, operation)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/operations/toimpl.py", line 50, in alter_column
    operations.impl.alter_column(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/ddl/impl.py", line 304, in alter_column
    self._exec(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/alembic/ddl/impl.py", line 193, in _exec
    return conn.execute(  # type: ignore[call-overload]
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1472, in _execute_ddl
    ret = self._execute_context(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/masayyed/superset/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "extra": syntax error
[SQL: ALTER TABLE report_schedule RENAME extra TO extra_json]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Below mentioned commands I am using for installation.

sudo yum upgrade python-setuptools
sudo yum install gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel
python3.8 -m pip install virtualenv
python3.8 -m venv venv
. venv/bin/activate
python3.8 -m pip install --upgrade setuptools pip
python3.8 -m pip install apache-superset
superset db upgrade

Below list of packages with version

(venv) [masayyed@vm1 superset]$ python3.8 -m pip list
Package                Version
---------------------- ---------
alembic                1.10.3
amqp                   5.1.1
apache-superset        2.1.0
apispec                5.2.2
async-generator        1.10
async-timeout          4.0.2
attrs                  22.2.0
Babel                  2.12.1
backoff                2.2.1
bcrypt                 4.0.1
billiard               3.6.4.0
bleach                 3.3.1
Brotli                 1.0.9
cachelib               0.4.1
celery                 5.2.7
certifi                2022.12.7
cffi                   1.15.1
click                  8.1.3
click-didyoumean       0.3.0
click-plugins          1.1.1
click-repl             0.2.0
colorama               0.4.6
convertdate            2.4.0
cron-descriptor        1.2.35
croniter               1.3.10
cryptography           39.0.2
Deprecated             1.2.13
deprecation            2.1.0
dnspython              2.3.0
email-validator        1.3.1
exceptiongroup         1.1.1
Flask                  2.1.3
Flask-AppBuilder       4.3.1
Flask-Babel            2.0.0
Flask-Caching          1.10.1
Flask-Compress         1.13
Flask-JWT-Extended     4.4.4
Flask-Limiter          3.3.0
Flask-Login            0.6.0
Flask-Migrate          3.1.0
Flask-SQLAlchemy       2.5.1
flask-talisman         1.0.0
Flask-WTF              1.0.1
func-timeout           4.3.5
geographiclib          2.0
geopy                  2.3.0
graphlib-backport      1.0.3
greenlet               2.0.2
gunicorn               20.1.0
h11                    0.14.0
hashids                1.3.1
hijri-converter        2.2.4
holidays               0.17.2
humanize               4.6.0
idna                   3.4
importlib-metadata     6.2.0
importlib-resources    5.12.0
isodate                0.6.1
itsdangerous           2.1.2
Jinja2                 3.1.2
jsonschema             4.17.3
kombu                  5.2.4
korean-lunar-calendar  0.3.1
limits                 3.3.1
Mako                   1.2.4
Markdown               3.4.3
markdown-it-py         2.2.0
MarkupSafe             2.1.2
marshmallow            3.19.0
marshmallow-enum       1.5.1
marshmallow-sqlalchemy 0.26.1
mdurl                  0.1.2
msgpack                1.0.5
numpy                  1.23.5
ordered-set            4.1.0
outcome                1.2.0
packaging              23.0
pandas                 1.5.3
paramiko               3.1.0
parsedatetime          2.6
pgsanity               0.2.9
pip                    23.0.1
pkgutil_resolve_name   1.3.10
polyline               2.0.0
prison                 0.2.1
prompt-toolkit         3.0.38
pyarrow                10.0.1
pycparser              2.21
Pygments               2.14.0
PyJWT                  2.6.0
PyMeeus                0.5.12
PyNaCl                 1.5.0
pyparsing              3.0.9
pyrsistent             0.19.3
PySocks                1.7.1
python-dateutil        2.8.2
python-dotenv          1.0.0
python-geohash         0.8.5
pytz                   2023.3
PyYAML                 6.0
redis                  4.5.4
rich                   13.3.3
selenium               4.8.3
setuptools             67.6.1
simplejson             3.19.1
six                    1.16.0
slack-sdk              3.21.0
sniffio                1.3.0
sortedcontainers       2.4.0
SQLAlchemy             1.4.47
SQLAlchemy-Utils       0.38.3
sqlparse               0.4.3
sshtunnel              0.4.0
tabulate               0.8.10
trio                   0.22.0
trio-websocket         0.10.2
typing_extensions      4.5.0
urllib3                1.26.15
vine                   5.0.0
wcwidth                0.2.6
webencodings           0.5.1
Werkzeug               2.2.3
wrapt                  1.15.0
wsproto                1.2.0
WTForms                2.3.3
WTForms-JSON           0.3.5
XlsxWriter             3.0.9
zipp                   3.15.0

Solution

  • I found this issue was with Sqlite only. I have replaced Sqlite with postgres database.

    SQLALCHEMY_DATABASE_URI = 'postgresql://postgres:postgres@localhost/postgres'
    

    It worked fine.