Apps aren't loaded yet exception occurs when using multi-processing in Django

Question

I'm doing a Django project and try to improve computing speed in backend.

The task is something like a CPU-bound conversion process

Here's my environment

• Python 3.6.1
• Django 1.10
• PostgreSQL 9.6

And I stuck with following errors when I try to parallel a computing API by python multi-processing library.

  File "D:\\project\apps\converter\models\convert_manager.py", line 1, in <module>
    from apps.conversion.models import Conversion
  File "D:\\project\apps\conversion\models.py", line 5, in <module>
    class Conversion(models.Model):
  File "C:\\virtenv\lib\site-packages\django\db\models\base.py", line 105, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\\virtenv\ib\site-packages\django\apps\registry.py", line 237, in get_containing_app_config
    self.check_apps_ready()
  File "C:\\lib\site-packages\django\apps\registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")

look like each process import Conversion model and Conversion model is like

from django.db import models


    Conversion(model.Model):

       conversion_name = models.CharField(max_length=63)
       conversion_user = models.CharField(max_length=31)
       conversion_description = models.TextField(blank=True)
       ...

Below is my sample function which I want to parallel, each iteration is independent but will access or insert data into SQL.

Class ConversionJob():
     ...

    def run(self, p_list):
        list_merge_result = []
        for p in p_list:
            list_result = self.Couputing_api(p)
            list_merge_result.extend(list_result)

and I'm try to do is

from multiprocessing import Pool


 Class ConversionJob():
         ...
        def run(self, p_list):
            list_merge_result = []

            p = Pool(process=4)
            list_result = p.map(self.couputing_api, p_list)
            list_merge_result.extend(list_result)

In computing_api(), it'll try to get current conversion's info which has completed and save into SQL before this api call, but this caused the error.

My question is

• Why import Conversion model will caused Apps aren't loaded yet errors, I had google lots of article but not actually solve my problems.

• I can see each Process SpawnPoolWorker-x generated and try to boot django server again(why?), each worker will stop at same errors.

• computing API will try to access sql , I haven't think about how to deal with this work. (share db connections or create new connection in each process)

Answer 1

This post can solve the problem.

Django upgrading to 1.9 error “AppRegistryNotReady: Apps aren't loaded yet.”

I had found this answer before, but not actually solve my problems at that time.

After I repeated test, I have to add these codes before import another model, otherwise, child-process will booting failed and give the error.

import django
django.setup()
from another.app import models