How can I structure my flask project such that my models can be aware of if TESTING is enabled (and thereby use a testing database), without having them deal with or have any knowledge of app context.
I'm developing this as an open source project so the source might shed some light on this: https://github.com/nficano/jotonce.com/blob/master/jotonce/messages/models.py#L33
I think you're running into this problem because you don't provide a way for users to specify configuration at run-time. Instead, managers.py grabs whatever settings that are specified in your settings.py file without consulting what settings the end-user might have specified.
Since you do have factory.py, you could potentially import current_app from Flask (assuming that your db functions are called within an app context) and use the settings value there. If that's an option for you, Flask has some good suggestions for configuration handling.
If you're running this outside of your application context, I don't think how factory.py is currently structured is going to work for you. You'll need to handle your own configuration manually.
You can take a look at https://github.com/Robpol86/Flask-Large-Application-Example/blob/master/pypi_portal/application.py for an example of a large flask project that uses the app factory with different configuration values well.
Good luck, and happy holidays!