I've created a flask app that is effectively a news search app. It currently serves up a number of routes, integrates with elasticsearch for document searching and indexing and integrates with a mySQL DB via Flask-SQLAlchemy.
The app should enable logged in users to search for a document/article at which point an elasticsearch is triggered and the results are returned to the user. This all works.
I am now building out functionality to kick off a background async search using RQ and Ajax. Now I need my RQ to queue up a task which will search (calling a few APIs), update my DB and update my ES Index. All of this functionality has already been built within my Flask app (search API calls, config, DB ORM, ES interface) and so I want to ideally reuse all of that.
I've got an RQ set up which calls a module which then tries to create a new application instance, so that I can reuse the existing DB/ES setups but I get an error from RQ when the code gets called:
File "./news_functions.py", line 14, in <module>
from app import create_app, db
ModuleNotFoundError: No module named 'app'
Application structure is as follows:
project
│ project.py
│
└───app
│ │ __init.py__
│ │ config.py
│ │ models.py
│ │ classes.py
│ │ news_functions.py
│ │ .env
│ │
│ └───main
│ │ __init__.py
│ │ forms.py
│ │ views.py
│ │ ...
│ └───static
│ │ css, etc.
│ └───templates
│ │ templates
│
└───migrations
│ ... DB migrations here...
└───logs
│ ... logs stored here
In my views.py file I attempt to queue up the task with the following line:
job = current_app.task_queue.enqueue('news_functions.execute_search', jobKwargs)
This causes my no issues on the flask side but throws the aforementioned error on the RQ worker
File "./news_functions.py", line 14, in <module>
from app import create_app, db
ModuleNotFoundError: No module named 'app'
In the news_functions.py file I attempted to create an instance of the app so that I could reuse my models, etc with the following imports and setup which seems to be throwing the error:
from app import create_app, db
app = create_app()
app.app_context().push()
I think that the issue is that I'm trying to import create_app from within the app folder itself but I don't know what my other options are. If I move up a folder then the RQ job can't queue up the task as it's outside the scope of the app.
I'm not sure if I'm approaching this in the right way but I effectively want to be able to reuse the application config, flask-sqlalchemy, es setup to execute this additional task in the background.
Hopefully makes sesnse and someone can help!
After absolutely pulling my hair out I realised that I had started the RQ worker from within the app directory itself rather than the main directory which meant that it was effectively looking for a module named 'app' within the 'app' folder, i.e. app.app.news_functions which obviously didn't exist.
Definite lesson learned after 2 days of pulling my hair out as to why I couldn't import a module which I knew existed!
Always check your working directories, kids!