I have been trying to import a large scale data from csv to Postgres after every 10mins. With the help of celery I have scheduled few jobs parallelly.
No. of rows in csv are not matching after import into postgres.
I've been writing CSVs into db with df.to_sql() method and it is not throwing any error but still some rows are missing in DB.
According to my observance the increasing size of table is leading to missing rows.
Is it Postgres limit? or is it the overlapping schedule of jobs which is causing locks or anything?
In one of the columns of my CSV file I had data in this format:
"abc, xyz"
So, using quotechar='"' helped me to fix my issue.
Thank you all for your ideas and suggestions.