Search code examples
pythonpostgresqldataframeescapingspecial-characters

Python Postgres - psycopg2 insert onto a table with columns that includes curly bracket


I have the following function that insert dataframe into a postgres table:

def insert(conn, df, table, return_field_list):
        tuples = [tuple(x) for x in df.to_numpy()]
        cols = ','.join(list(df.columns))
        query  = "INSERT INTO {} ({}) VALUES (%%s)".format(table, cols)
        sub_stmt = ' RETURNING {} '.format(' ,'.join(return_field_list))
        query += sub_stmt
        cursor = conn.cursor()
        try:
            extras.execute_values(cursor, query, tuples)
            conn.commit()
            return cursor.fetchall()
        except (Exception, psycopg2.DatabaseError) as error:
            print("Error: %s" % error)
            conn.rollback()
            cursor.close()
            return None
        finally:
            if cursor is not None:
                cursor.close()

Some of my table columns include special characters, e.g. "Ni[%{wt}]", S[%{wt}] I 'escaped' the dataframe columns with double quotes (except id and name columns) before passing it to the function above:

df.rename(columns = lambda col: f'"{col}"' if col not in ('id', 'name') else col, inplace=True)

However, the function returns the following error:

Error: unsupported format character: '{'

Solution

  • I need to escape the %

    r ={}
    for c in df.columns:
      r = c
      if '%' in r:
        r = r.replace('%', '%%')
        replace[c] = r
    df.rename(columns = replace, inplace=True)