I tried to build container to use prefect server. So I use docker compose and "prefecthq/prefect" image. (version : prefecthq/prefect:2.10-python3.9)
But, This error occur. (main error sentence)
sqlite3.OperationalError: table worker_pool already exists
I don't understand that I'm not generate table "worker_pool" in docker environment. Is Something wrong in my docker-compose.yml?
Here's my codes
version: "3.9"
services:
orion:
image: prefecthq/prefect:2.10-python3.9
restart: always
volumes:
- prefect:/root/.prefect
entrypoint: ["prefect", "server", "start"]
environment:
- PREFECT_SERVER_API_HOST=0.0.0.0
- PREFECT_UI_API_URL=http://localhost:4200/api
ports:
- 4200:4200
And Here's entire logs
Attaching to test_bed-orion-1
test_bed-orion-1 |
test_bed-orion-1 | ___ ___ ___ ___ ___ ___ _____
test_bed-orion-1 | | _ \ _ \ __| __| __/ __|_ _|
test_bed-orion-1 | | _/ / _|| _|| _| (__ | |
test_bed-orion-1 | |_| |_|_\___|_| |___\___| |_|
test_bed-orion-1 |
test_bed-orion-1 | Configure Prefect to communicate with the server with:
test_bed-orion-1 |
test_bed-orion-1 | prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
test_bed-orion-1 |
test_bed-orion-1 | View the API reference documentation at http://0.0.0.0:4200/docs
test_bed-orion-1 |
test_bed-orion-1 | Check out the dashboard at http://0.0.0.0:4200
test_bed-orion-1 |
test_bed-orion-1 |
test_bed-orion-1 |
test_bed-orion-1 | Traceback (most recent call last):
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
test_bed-orion-1 | self.dialect.do_execute(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
test_bed-orion-1 | cursor.execute(statement, parameters)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 108, in execute
test_bed-orion-1 | self._adapt_connection._handle_exception(error)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 236, in _handle_exception
test_bed-orion-1 | raise error
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 90, in execute
test_bed-orion-1 | self.await_(_cursor.execute(operation, parameters))
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
test_bed-orion-1 | return current.driver.switch(awaitable)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
test_bed-orion-1 | value = await result
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/cursor.py", line 37, in execute
test_bed-orion-1 | await self._execute(self._cursor.execute, sql, parameters)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/cursor.py", line 31, in _execute
test_bed-orion-1 | return await self._conn._execute(fn, *args, **kwargs)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/core.py", line 137, in _execute
test_bed-orion-1 | return await future
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/core.py", line 110, in run
test_bed-orion-1 | result = function()
test_bed-orion-1 | sqlite3.OperationalError: table worker_pool already exists
test_bed-orion-1 |
test_bed-orion-1 | The above exception was the direct cause of the following exception:
test_bed-orion-1 |
test_bed-orion-1 | Traceback (most recent call last):
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/api/server.py", line 481, in lifespan
test_bed-orion-1 | await run_migrations()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/api/server.py", line 404, in run_migrations
test_bed-orion-1 | await db.create_db()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/interface.py", line 55, in create_db
test_bed-orion-1 | await self.run_migrations_upgrade()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/interface.py", line 63, in run_migrations_upgrade
test_bed-orion-1 | await run_sync_in_worker_thread(alembic_upgrade)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
test_bed-orion-1 | return await anyio.to_thread.run_sync(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
test_bed-orion-1 | return await get_asynclib().run_sync_in_worker_thread(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
test_bed-orion-1 | return await future
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
test_bed-orion-1 | result = context.run(func, *args)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/alembic_commands.py", line 24, in wrapper
test_bed-orion-1 | return fn(*args, **kwargs)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/alembic_commands.py", line 53, in alembic_upgrade
test_bed-orion-1 | alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 378, in upgrade
test_bed-orion-1 | script.run_env()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 576, in run_env
test_bed-orion-1 | util.load_python_file(self.dir, "env.py")
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
test_bed-orion-1 | module = load_module_py(module_id, path)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
test_bed-orion-1 | spec.loader.exec_module(module) # type: ignore
test_bed-orion-1 | File "<frozen importlib._bootstrap_external>", line 850, in exec_module
test_bed-orion-1 | File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/migrations/env.py", line 147, in <module>
test_bed-orion-1 | apply_migrations()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 248, in coroutine_wrapper
test_bed-orion-1 | return run_async_from_worker_thread(async_fn, *args, **kwargs)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
test_bed-orion-1 | return anyio.from_thread.run(call)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/anyio/from_thread.py", line 49, in run
test_bed-orion-1 | return asynclib.run_async_from_thread(func, *args)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
test_bed-orion-1 | return f.result()
test_bed-orion-1 | File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 446, in result
test_bed-orion-1 | return self.__get_result()
test_bed-orion-1 | File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
test_bed-orion-1 | raise self._exception
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/migrations/env.py", line 141, in apply_migrations
test_bed-orion-1 | await connection.run_sync(do_run_migrations)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/engine.py", line 548, in run_sync
test_bed-orion-1 | return await greenlet_spawn(fn, conn, *arg, **kw)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 128, in greenlet_spawn
test_bed-orion-1 | result = context.switch(value)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/migrations/env.py", line 129, in do_run_migrations
test_bed-orion-1 | context.run_migrations()
test_bed-orion-1 | File "<string>", line 8, in run_migrations
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/runtime/environment.py", line 868, in run_migrations
test_bed-orion-1 | self.get_context().run_migrations(**kw)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/runtime/migration.py", line 622, in run_migrations
test_bed-orion-1 | step.migration_fn(**kw)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/database/migrations/versions/sqlite/2022_11_24_143302_fe77ad0dda06_add_worker_tables.py", line 25, in upgrade
test_bed-orion-1 | op.create_table(
test_bed-orion-1 | File "<string>", line 8, in create_table
test_bed-orion-1 | File "<string>", line 3, in create_table
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/operations/ops.py", line 1255, in create_table
test_bed-orion-1 | return operations.invoke(op)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/operations/base.py", line 401, in invoke
test_bed-orion-1 | return fn(self, operation)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/operations/toimpl.py", line 114, in create_table
test_bed-orion-1 | operations.impl.create_table(table)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/ddl/impl.py", line 354, in create_table
test_bed-orion-1 | self._exec(schema.CreateTable(table))
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/alembic/ddl/impl.py", line 193, in _exec
test_bed-orion-1 | return conn.execute( # type: ignore[call-overload]
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/future/engine.py", line 280, in execute
test_bed-orion-1 | return self._execute_20(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
test_bed-orion-1 | return meth(self, args_10style, kwargs_10style, execution_options)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
test_bed-orion-1 | return connection._execute_ddl(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1472, in _execute_ddl
test_bed-orion-1 | ret = self._execute_context(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
test_bed-orion-1 | self._handle_dbapi_exception(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
test_bed-orion-1 | util.raise_(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
test_bed-orion-1 | raise exception
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
test_bed-orion-1 | self.dialect.do_execute(
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
test_bed-orion-1 | cursor.execute(statement, parameters)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 108, in execute
test_bed-orion-1 | self._adapt_connection._handle_exception(error)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 236, in _handle_exception
test_bed-orion-1 | raise error
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 90, in execute
test_bed-orion-1 | self.await_(_cursor.execute(operation, parameters))
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
test_bed-orion-1 | return current.driver.switch(awaitable)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
test_bed-orion-1 | value = await result
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/cursor.py", line 37, in execute
test_bed-orion-1 | await self._execute(self._cursor.execute, sql, parameters)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/cursor.py", line 31, in _execute
test_bed-orion-1 | return await self._conn._execute(fn, *args, **kwargs)
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/core.py", line 137, in _execute
test_bed-orion-1 | return await future
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/aiosqlite/core.py", line 110, in run
test_bed-orion-1 | result = function()
test_bed-orion-1 | sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table worker_pool already exists
test_bed-orion-1 | [SQL:
test_bed-orion-1 | CREATE TABLE worker_pool (
test_bed-orion-1 | id CHAR(36) DEFAULT (
test_bed-orion-1 | (
test_bed-orion-1 | lower(hex(randomblob(4)))
test_bed-orion-1 | || '-'
test_bed-orion-1 | || lower(hex(randomblob(2)))
test_bed-orion-1 | || '-4'
test_bed-orion-1 | || substr(lower(hex(randomblob(2))),2)
test_bed-orion-1 | || '-'
test_bed-orion-1 | || substr('89ab',abs(random()) % 4 + 1, 1)
test_bed-orion-1 | || substr(lower(hex(randomblob(2))),2)
test_bed-orion-1 | || '-'
test_bed-orion-1 | || lower(hex(randomblob(6)))
test_bed-orion-1 | )
test_bed-orion-1 | ) NOT NULL,
test_bed-orion-1 | created DATETIME DEFAULT (strftime('%Y-%m-%d %H:%M:%f000', 'now')) NOT NULL,
test_bed-orion-1 | updated DATETIME DEFAULT (strftime('%Y-%m-%d %H:%M:%f000', 'now')) NOT NULL,
test_bed-orion-1 | name VARCHAR NOT NULL,
test_bed-orion-1 | description VARCHAR,
test_bed-orion-1 | type VARCHAR,
test_bed-orion-1 | base_job_template JSON DEFAULT '{}' NOT NULL,
test_bed-orion-1 | is_paused BOOLEAN DEFAULT '0' NOT NULL,
test_bed-orion-1 | concurrency_limit INTEGER,
test_bed-orion-1 | default_queue_id CHAR(36),
test_bed-orion-1 | CONSTRAINT pk_worker_pool PRIMARY KEY (id),
test_bed-orion-1 | CONSTRAINT uq_worker_pool__name UNIQUE (name)
test_bed-orion-1 | )
test_bed-orion-1 |
test_bed-orion-1 | ]
test_bed-orion-1 | (Background on this error at: https://sqlalche.me/e/14/e3q8)
test_bed-orion-1 |
test_bed-orion-1 | During handling of the above exception, another exception occurred:
test_bed-orion-1 |
test_bed-orion-1 | Traceback (most recent call last):
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/starlette/datastructures.py", line 702, in __getattr__
test_bed-orion-1 | return self._state[key]
test_bed-orion-1 | KeyError: 'services'
test_bed-orion-1 |
test_bed-orion-1 | During handling of the above exception, another exception occurred:
test_bed-orion-1 |
test_bed-orion-1 | Traceback (most recent call last):
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 677, in lifespan
test_bed-orion-1 | async with self.lifespan_context(app) as maybe_state:
test_bed-orion-1 | File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
test_bed-orion-1 | return await self.gen.__anext__()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/api/server.py", line 486, in lifespan
test_bed-orion-1 | await stop_services()
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/prefect/server/api/server.py", line 468, in stop_services
test_bed-orion-1 | if app.state.services:
test_bed-orion-1 | File "/usr/local/lib/python3.9/site-packages/starlette/datastructures.py", line 705, in __getattr__
test_bed-orion-1 | raise AttributeError(message.format(self.__class__.__name__, key))
test_bed-orion-1 | AttributeError: 'State' object has no attribute 'services'
test_bed-orion-1 |
test_bed-orion-1 | Application startup failed. Exiting.
test_bed-orion-1 | Server stopped!
Please let me know how to remove work_pool or solve this problem..! Thanks to read my question and if you need more information, ask me please.
Prefect is creating sql database at first run inside mounted prefect volume.
At second run database is already created and don't need to be created again. To solve problem you may use two ways:
Remove prefect folder contents every time before starting container and that will allow prefect to recreate db every time
Set PREFECT_API_DATABASE_MIGRATE_ON_START parameter to False. That will stop prefect to create new db on start and allow to re-use current db if it exists