Python multithreading in a loop for executing query sql

Question

I search to transform my program in multithreading, but I don't say how to do. Indeed, I thought of 2 solutions, but i think it is not the more optimal : Do a select and stock the result in an array in python and filter after by range for the threads. Or multithread with a clause id for the select to have only one Domain by thread Is there a better way to make this directly in the loop?

#!/usr/bin/python
import mysql.connector as mariadb
from subprocess import Popen, PIPE, STDOUT
import re
import os

mariadb_connection = mariadb.connect(user='root', password='xxxx', database='xxxx');
cursor = mariadb_connection.cursor(buffered=True)
#retrieving information

cursor.execute("SELECT Domain,Id FROM classement where Domain like '%com';")

for Domain,Id in cursor:
          counter=0
          for character in Domain:
                if (character == "."):
                        counter = counter + 1
                        if (counter==1):
                            print(Domain)
                            pingresult = Popen(['ping','-c','3', Domain], stdout=PIPE, stderr=STDOUT).communicate()[0]
                            pingresult=str(pingresult)
                            ping = pingresult.find('min/avg/max/mdev')
                            ping=pingresult[ping+19:-6]
                            print(ping)
                            cursor.execute('update classement set Ping = "%s" where Domain = "%s";' % (ping,Domain))
                            mariadb_connection.commit()
mariadb_connection.close()

Answer 1

I think simplest way is a wrap your loop in function and use multiprocessing.dummy thread pool:

from multiprocessing.dummy import Pool
...
cursor.execute( ... )
...

def process (entry):
    Domain,Id = entry
    ...
    # your code there
    ...
    cursor.execute( ... )
    mariadb_connection.commit()

pool = Pool(os.cpu_count())
# or may be N*os.cpu_count() if ping waits for
# network interaction too long

pool.map(process, cursor)
pool.join()

May be additional locking required for multithreaded access to database.