Search code examples
sqlalchemybulkinsert

SQL Alchemy - INSERT results of query


I'm looking for a way in SQLAlchemy to do a bulk INSERT whose rows are the result of a query. I know the session has the function add which can be used to add an individual object, but I can't seem to find how how it works with a subquery.

I know I could iterate over the results of the subquery and add them individually, but this would seem to be somewhat inefficient. In my case I am dealing with a potentially very large set of data that needs insertion.


Solution

  • I see following options:

    1. using SA Model: create underlying objects with data loaded from the database, add them to session and commit.
      • Pros: if you have any AS Model level validation, you are able to use it; also you can insert into multiple tables if you model objects are mapped to multiple tables (Joined-Table Inheritance); is RDBMS independent
      • Cons: most expensive
    2. using Insert statements: load data from the database into python and execute using Insert Expressions
      • Pros: somewhat faster when compared to 1.
      • Cons: still expensive as python structures are created; cannot directly handle Joined-Table Inheritance
    3. create data using solely RDBMS: bulk insert using RDBMS only bypassing SA and python altogether.
      • Pros: fastest
      • Cons: no business object validation performed; potentially RDBMS-specific implementation required

    I would suggest either option 1) or 3).
    In fact, if you do not have any object validation and you use only one RDBMS, I would stick to option 3).