Send scheduled emails with pyramid_mailer and apscheduler

I've tried getting this to work but there must be a better way, any input is welcome.

I'm trying to send scheduled emails in my python pyramid app using pyramid_mailer (settings stored in .ini file), and apscheduler to set the schedule.

I also use the SQLAlchemyJobStore so jobs can be restarted if the app restarts.

jobstores = {
    'default': SQLAlchemyJobStore(url='mysql://localhost/lgmim')
}
scheduler = BackgroundScheduler(jobstores=jobstores)

@view_config(route_name='start_email_schedule')
def start_email_schedule(request):
    # add the job and start the scheduler
    scheduler.add_job(send_scheduled_email, 'interval', [request], weeks=1)
    scheduler.start()

    return HTTPOk()

def send_scheduled_email(request):

    # compile message and recipients
    # send mail  
    send_mail(request, subject, recipients, message)

def send_mail(request, subject, recipients, body):

    mailer = request.registry['mailer']
    message = Message(subject=subject,
                  recipients=recipients,
                  body=body)

    mailer.send_immediately(message, fail_silently=False)

This is as far as I've gotten, now I'm getting an error, presumably because it can't pickle the request.

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Using pyramid.threadlocal.get_current_registry().settings to get the mailer works the first time, but thereafter I get an error. I'm advised not to use it in any case.

What else can I do?

Solution

Generally, you cannot pickle request object as it contains references to things like open sockets and other liveful objects.

Some useful patterns here are that

You pregenerate email id in the database and then pass id (int, UUID) over scheduler
You generate template context (JSON dict) and then pass that over the scheduler and render the template inside a worker
You do all database fetching and related inside a scheduler and don't pass any arguments

Specifically, the problem how to generate a faux request object inside a scheduler can be solved like this:

from pyramid import scripting
from pyramid.paster import bootstrap

def make_standalone_request():
    bootstrap_env = bootstrap("your-pyramid-config.ini")
    app = bootstrap_env["app"]
    pyramid_env = scripting.prepare(registry=bootstrap_env["registry"])
    request = pyramid_env["request"]

    # Note that request.url will be always dummy,
    # so if your email refers to site URL, you need to 
    # resolve request.route_url() calls before calling the scheduler
    # or read the URLs from settings

    return request

Some more inspiration can be found here (disclaimer: I am the author).