I've created a small python script to forward approximately 11000 mails (as *.eml) to my mail server. The mails are attached with CSV files, containing data from daily energy measurements.
So, my approach is working correctly, however it takes nearly 20min or so to send the mails! Any ideas to speed this up?
#! /usr/bin/env python
import os, sys
import email, smtplib
storage_folder = "data/export"
storage_header = 0
mail_server = 'server-address'
mail_username = 'mail-user'
mail_password = 'mail-password'
mail_to = 'mail-address-to'
mail_from = 'mail-sender'
def load_local_mails(file_folder):
#: scan desired folder for *.eml files
email_body = []
for dirname, dirnames, filenames in os.walk(file_folder):
for filename in filenames:
if filename.endswith('eml'):
f = open(os.path.join(dirname, filename))
email_body.append(f.read())
return email_body
def send_mail(email_body):
#: send mails
smtp = smtplib.SMTP(mail_server, 587)
smtp.starttls()
smtp.login(mail_username, mail_password)
for idx, item in enumerate(email_body):
m = email.message_from_string(item)
m.replace_header("From", mail_from)
m.replace_header("To", mail_to)
m.replace_header("Subject", m['subject'])
smtp.sendmail(mail_from, mail_to, m.as_string())
print "Forwarding mail %d/%d: '%s' from '%s' to %s " % (idx+1, len(email_body), m['subject'], m['date'], mail_to)
smtp.quit()
def main():
if len(sys.argv) > 1:
if sys.argv[1] == '--start':
send_mail(load_local_mails(sys.argv[2]))
else:
exit("Don't know what to do? :(. Goodbye")
if __name__ == '__main__':
main()
Update: I was able to manage a direct (SCP) upload to the specific folder in the qmail environment. I think this is the fastest solution! Anyway, thanks for the suggestions.
Well, you don't provide much data but 20 minutes does not look very unreasonable for such amount of emails with smtp protocol. With smtp you have to exchange 5 messages with server.
When you assume 10ms ping to your destination that gives 11k * 50ms = 550s for only network latency overhead. That is roughly 10 minutes.
Then you need to actually open and read 11k files it also can take a while, maybe not that much. But you did not specify the size of data you send. Maybe it can account for the rest of the time. Especially if you say you send csv data files.
It will be hard to speed it up not changing technology solutions you use. But I would suggest transmitting data as binary in bigger chunks preferably using compression. Compression should be meaningful for csv files.