Search code examples
pythonmysqlormsqlalchemy

Dynamically setting __tablename__ for sharding in SQLAlchemy?


In order to handle a growing database table, we are sharding on table name. So we could have database tables that are named like this:

table_md5one
table_md5two
table_md5three

All tables have the exact same schema.

How do we use SQLAlchemy and dynamically specify the tablename for the class that corresponds to this? Looks like the declarative_base() classes need to have tablename pre-specified.

There will eventually be too many tables to manually specify derived classes from a parent/base class. We want to be able to build a class that can have the tablename set up dynamically (maybe passed as a parameter to a function.)


Solution

  • OK, we went with the custom SQLAlchemy declaration rather than the declarative one.

    So we create a dynamic table object like this:

    from sqlalchemy import MetaData, Table, Column
    
    def get_table_object(self, md5hash):
        metadata = MetaData()
        table_name = 'table_' + md5hash
        table_object = Table(table_name, metadata,
            Column('Column1', DATE, nullable=False),
            Column('Column2', DATE, nullable=False)
        )
        clear_mappers()
        mapper(ActualTableObject, table_object)
        return ActualTableObject
    

    Where ActualTableObject is the class mapping to the table.