Search code examples
pythonairflowpostgres-operator

airflow postgresoperator transaction and commits


From the documentation of airflow:

https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/_api/airflow/providers/postgres/operators/postgres/index.html

Says this about autocommit:

autocommit (bool) – if True, each command is automatically committed. (default value: False)

So if I have two commands(statements), as such:

my_operator = PostgresOperator(
    task_id="mytask",
    postgres_conn_id="myconn",
    autocommit=True,
    sql="""

SELECT * FROM FILMS;

DELETE FROM films USING producers
  WHERE producer_id = producers.id AND producers.name = 'foo'""";

Does that mean this will be treated as two transactions with two commits ? Or one transation with two commits ? (guess this is not even possible). the point is if one of the statements fail will everything rollback in an atomic way ?

Thanks.


Solution

  • The postgres provider uses psycopg2 as you can see in the source code:

    from psycopg2.sql import SQL, Identifier
    

    And if we refer to the autocommit documentation:

    It is possible to set the connection in autocommit mode: this way all the commands executed will be immediately committed and no rollback is possible...

    So if you put a SELECT 1/0 in between your first and second action, I believe the first action will still be applied.

    But the best way to make sure is to test it, replace the first action with an INSERT and the modify the delete to cancel it out. Then put a failing operation in between, run it and go see the state of your DB.