JavaMail, IMAP, performance with large number of folders

Question

We are developing a java based mail client for IMAP accounts and use latest java mail api (1.5.6). Our customers have mail accounts with more than 400 folders. Users perform check mails on folders and iterating on each folder and getting new messages for example,

folder.getMessagesByUid(lastStoredUID, UIDFolder.LastUID)

or getting count of unread messages takes too much time beacuse huge number of folders. (We have to iterate on 400 folders)

To increase the performance we used paralel working connections in threads and we have a single SESSION instance but each thread(connection) has its own STORE instance. When we have to fetch new messages, we generate 10 threads with their STORE instances on the fly, connect/login them separately, get folder instance on that STORE, open folder, perform folder operations, close the folder and close the SESSION afterwards. But for me its not clear, whether I have to keep the folders intances and also I it is not clear whether the folders should be kept opened or should we explicity close aftwerwards or we may leave all opened (leave the close operation to mailserver).

In my current implementation,

new Thread() {
    public void run() {
        //Get a new store instance
        Store tempStore = MySingleSession.getStore("imap");
        tempStore.connect(..);

        //Get a folder for example inbox
        Folder inbox = tempStore.get("INBOX");

        inbox.open(Folder.READ);

        // Perform get new messages
        inbox.getMessagesByUd(lastUID, UIDFolder.MaxUID);

        inbox.close();

        tempStore.close();
        ..

When checked the performance, I see that opening/closing folders takes a long time especially on folders with more than 100.000 messages. And I see that even if I close the store of a folder, folder remains opened and we can operate on it if we do not explicitly close that folder. What are the disadvantages for a such mechanism if I change my implementation and do not close the folder explicitly and keep them open?

• Keep the folder references in a hashtable keyed by their fullname, for example "INBOX" -> IMAPFolder and perform isOpened check before any operation (getMessages), but when finished, do not close explicitly as in my current implementation. When doing like that we may have 400 folders opened, it may seem bad for mail server and mail server close them, but I will check for each folder opeation first, if it is opened or not. As a result folders will be kept opened as possible (until mailserver closes connection) and if the same folder is needed again we do not have to open it again.

• I do not understand, why I can still operate on a folder which has its store closed?

• Default value for mail.imap.connectionpoolsize is 1, does it help to increase this value in this case?

• And I do not understand what happens with this scenario below?

  Store tempStore = MySingleSession.getStore("imap");
  
  tempStore.connect(..);
  
  Folder f = tempStore.getFolder("INBOX");
  
  f.open(Folder.READ_ONLY);
  
  // close the store!!!
  tempStore.close();
  
  // now folder f becomes closed,
  // but we may open it again even if its Store is disconnected!
  
  f.open(Folder.READ_ONLY);
  
  // we can continue operating on folder f
  

• Which is better approach, keep folder references in an hastable keyed by fullname or perform tempStore.getFolder(foldername) each time? For example we have to handle INBOX, first option use

IMAPFolder f = folderCache.get("INBOX"); // suppose already put in hashtable folderCache

or

IMAPFolder f = tempStore.getFolder("INBOX");

I generate the Store instances on demand and close them afterwards in order not to use huge resources on the mail server. But opening the folder again and not closing manually is not a good approach beacuse it uses still resource? or using that way is acceptable because mailserver may close it when needed and we have just to check whether

• if the folder is already opened continue operating,
• if the folder is closed perform open first, then operate, keep opened for further usage?

Answer 1

Way too many questions for one post. Let me try to answer a few of them...

After you close a Folder, you shouldn't use any Message objects from the Folder, so only close the Folder when you're done using its Messages.

If a single user has 400 folders, you can use a single Store connection since each open Folder will get its own connection.

If you open and close Folders frequently, increasing the connection pool size can help since the connection for a closed Folder will be added to the pool and reused when you open a new Folder, without needing to create a new connection.

Closing the Store should close all the Folders for that Store, but there's an inherent race condition if multiple threads are involved. Closing the Store is a way to clean up all the connections when you're done using it, not a way to prevent other threads from re-opening Folders.

Caching Folder objects for closed Folders is not likely to be a big advantage over calling getFolder each time.