pythonmysqlpymysql
Python3 script (using PyMySQL) ultra slow
I'm using Python to read extra large files (.csv) and insert each one on MySQL using PyMySQL.
On Windows, when I run the (same) script on Anaconda Spider, the speed of execution is more than 3000 lines per second.
On production server (a better hardware) running Ubuntu 20.04, the speed downs to 100 lines per second.
I have an homolog server with a poor hardware. On this homolog server the speed is over than 2500 lines per second.
Is there something that I have not seen?
My server lspcu:
Solution
I read about PyMySQL and MySQL drivers on Ubuntu 20.X and proceeded with a downgrade on the server. Ubuntu 18.04. The speed of insertions was reached. About 4K lines/second on my Python scripts.
- Webscraping Roblox
- Remove the mandatory field label 'This field is required.' and fix the bug with 'clean_email'
- How to plot a Probability Density Function in Python?
- How large is a fresh install of Python?
- Appending new elements into an empty list
- Simple way to measure cell execution time in ipython notebook
- PyAudio working, but spits out error messages each time
- Reportlab show page number and page count IF there is more than one page in a document
- How to read SharePoint Online (Office365) Excel files into Python specifically pandas with Work or School Account?
- How to set a column which suffix name is based on a value in another column
- Debugging Python C++ extension from Visual Studio Code on Linux
- How can I get all users on Google admin_sdk?
- csv.Error: iterator should return strings, not bytes
- How to check if an object has an attribute?
- How to use selenium with proxy auth in headless mode?
- Is there a way to exit a pytest test and continue to the next one?
- Returning the lowest index for the first non whitespace character in a string in Python
- Formatting exceptions as Python does
- Prime factorization using list comprehension in Python
- Why does the power spectrum E(k) of my velocity field follow 𝑘 ^(−(n−1)) instead of 𝑘^(−n)?
- How to merge dataframes over multiple columns and split rows?
- How to create a Sympy IndexedBase using a custom subclass of Symbol?
- Removing dynamically an element from a list
- Returning boolean if set is empty
- Can variables be decorated?
- Fast(est) exponentiation of numpy 3D matrix
- Removing an element from a list based on a condition
- Printing elements of dictionary line by line
- Matplotlib does not display the hatch of a patch in a legend
- Python win32com - Class not registered error