Building python dict using multithreading

Question

I'm currently building a Tool that generates a PDF report using Python pdfkit and jinja.

This report builds on a lot of static html and around 30 functions that produces data and images (charts) for the report. These functions all access external data through pyodbc or pandas from_sql.

I am now running into performance issues, and the report takes around 5 minutes to build.

I'm hoping to utilize multithreading in order to build a dictionary of data, but have not been able to figure out how to approach the issue.

My current code looks something like this.

def buildReport():

    if checkKvaegCVR(SQL = checkKvaegCVRSQL(cvrNummer = cvrNummer), cursor = OEDBCursor):

        env = Environment(loader=FileSystemLoader('.'))
        template = env.get_template("templates/kvaeg/kvaegBase.html")

        pdfOptions = {
            'page-size': 'A4',
            'margin-top': '0.75in',
            'margin-right': '0.75in',
            'margin-bottom': '0.75in',
            'margin-left': '0.75in',
            'quiet': '',
            'encoding': "UTF-8",
            'footer-right': '[page]'
        }

        css = 'static/css/style.css'

        template_vars = {'kvaegForsideBillede': imageBuilder()['kvaegForsideBillede'],
                        'bagsideBillede': imageBuilder()['bagsideBillede'],
                        'navn' : bedriftAdresse(cvrNummer = cvrNummer, 
                                                cursor = KundeAnalyseDBCursor)[0],
                        'adresse' : bedriftAdresse(cvrNummer = cvrNummer, 
                                                cursor = KundeAnalyseDBCursor)[1],
                        'postnrBy' : str(int(bedriftAdresse(cvrNummer = cvrNummer, cursor = KundeAnalyseDBCursor)[2])) + ' ' + 
                                            bedriftAdresse(cvrNummer = cvrNummer, cursor = KundeAnalyseDBCursor)[3],
                        'fremstillingsprisKorr': imageBuilder()['fremstillingsprisKorr'],
                        'fremstillingsprisForbedring':imageBuilder()['fremstillingsprisForbedring'],
                        'graesoptagelse':kgGraesPrKo(),
                        'indreSaedskifteKort':indreSaedskifteKortPNG(CVRPunkt = CVRPunkt(cvrNummer, KundeAnalyseDBCursor), 
                                                                    CVRBuffer = CVRBuffer(cvrNummer, KundeAnalyseDBCursor),
                                                                    indreSaedskifteKort = indreSaedskifteKort(indreSaedskifteKortSQL = indreSaedskifteKortSQL(cvrNummer = cvrNummer), cursor = KundeAnalyseDBCursor)),
                        'naboKort':naboKortPNG(CVRPunkt = CVRPunkt(cvrNummer = cvrNummer, cursor = KundeAnalyseDBCursor), 
                                                CVRBuffer = CVRBuffer(cvrNummer = cvrNummer, cursor = KundeAnalyseDBCursor),
                                                naboKort = naboKort(naboMarkerSQL = naboMarkerSQL(cvrNummer = cvrNummer), 
                                                                    egneMarkerSQL = egneMarkerSQL(cvrNummer = cvrNummer), 
                                                                    cursor = KundeAnalyseDBCursor))
                        ...
                        ...
                        30 more functions here
                        ...
                        ...}

        pdfkit.from_string(template.render(template_vars), 'KvaegRapport - {}.pdf'.format(cvrNummer), options=pdfOptions, css=css)
        print('Rapporten er klar')
    else:
        print('Kan ikke bygge rapport på dette CVR nummer')

I would like to build the dictionary "Template_vars" using multithreading (probably outside my main function)

Any suggestions?

Answer 1

I could suggest following but with multiprocessing (the following code has not been tested):

from multiprocessing import Process, Queue


def make_smth(func, queue, name, *args, **kwargs):
    queue.put((name, func(*args, **kwargs)))


result_queue = Queue()
processes = list()
processes.append(
    Process(target=make_smth,
            args=(bedriftAdresse, result_queue, "navn"),
            kwargs={cvrNummer: cvrNummer, cursor: KundeAnalyseDBCursor[0]}
    )
)
processes.append(
    Process(target=make_smth,
            args=(kgGraesPrKo, result_queue, "graesoptagelse"),
            kwargs={}
    )
)
#...... You should do it for each of your functions

for p in processes:
    p.start()

template_vars = {}
result = result_queue.get()
while result:
    template_vars[result[0]] = result[1]
    result = result_queue.get()