sqalchemy update bindparam primary key

Question

The following code throws "sqlalchemy.exc.CompileError: Unconsumed column names: _id".

User = Table('users', metadata,
 Column('id', Integer, primary_key=True),
 Column('score', Integer)
)

values = [
    {'score': 2, '_id': 1},
    {'score': 3, '_id': 3}
]
query = User.update().where(User.c.id == bindparam('_id')).values(score=bindparam('score'))
await db.execute_many(query, values)

db is an instance of databases.Database. Notice that I have to the name '_id' because SQLalchemy says 'id' is reserved.

Is there any solution other than updating each row individullay?

Answer 1

Database.execute_many() calls Connection.execute_many() which breaks your query up into separate individual queries (one per element in values), here's the method (source):

    async def execute_many(
        self, query: typing.Union[ClauseElement, str], values: list
    ) -> None:
        queries = [self._build_query(query, values_set) for values_set in values]
        async with self._query_lock:
            await self._connection.execute_many(queries)

Note that it calls the _build_query() method (source):

    @staticmethod
    def _build_query(
        query: typing.Union[ClauseElement, str], values: dict = None
    ) -> ClauseElement:
        if isinstance(query, str):
            query = text(query)

            return query.bindparams(**values) if values is not None else query
        elif values:
            return query.values(**values)

        return query

As you aren't passing a str query and you are passing values, control enters the elif values: condition handling where the individual dict of values is unpacked into the .values() method on your query (which is Update.values()). That essentially makes the query it's trying to compile this:

query = (
    User.update()
    .where(User.c.id == bindparam("_id"))
    .values(score=bindparam("score"))
    .values(score=2, _id=1)
)

That second values clause results in a new Update with new bind params that are trying to set values for both score and _id. This causes compilation of the query to fail as there is no _id column on the table.

So the MCVE to reproduce the error is really this:

from sqlalchemy.dialects import postgresql

User.update().values(score=2, _id=1).compile(dialect=postgresql.dialect())

Which raises:

Traceback (most recent call last):
  File ".\main.py", line 31, in <module>
    User.update().values(score=2, _id=1).compile(dialect=postgresql.dialect())
  File "<string>", line 1, in <lambda>
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\elements.py", line 462, in compile
    return self._compiler(dialect, bind=bind, **kw)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\elements.py", line 468, in _compiler
    return dialect.statement_compiler(dialect, self, **kw)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\compiler.py", line 571, in __init__
    Compiled.__init__(self, dialect, statement, **kwargs)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\compiler.py", line 319, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\compiler.py", line 350, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\visitors.py", line 92, in _compiler_dispatch
    return meth(self, **kw)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\compiler.py", line 2569, in visit_update
    self, update_stmt, crud.ISUPDATE, **kw
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\crud.py", line 62, in _setup_crud_params
    return _get_crud_params(compiler, stmt, **kw)
  File "C:\Users\peter\Documents\git\stackoverflow\58668615-sqalchemy-update-bindparam-primary-key\.venv\lib\site-packages\sqlalchemy\sql\crud.py", line 177, in _get_crud_params
    % (", ".join("%s" % c for c in check))
sqlalchemy.exc.CompileError: Unconsumed column names: _id

To summarise the issue, you build a query with bind params passed to both Update.where() and Update.values(). You then pass that query and your values to Database.execute_many() where they unpack the individual elements of your values list into a second call of Update.values() on your query which replaces your query with one that tries to set a value for an _id column which doesn't exist.

Is there any solution other than updating each row individullay?

Well the query works just fine when using sqlalchemy engine as well as query:

# using a sqlalchemy engine
engine.execute(query, values)

Otherwise, what should work is sending the query in as a string to Database.execute_many() as that will mean the query gets handled in the if isinstance(query, str): part of the _build_query() method which will avoid the second .values() call being made on the query:

db.execute_many(str(query), values)