My team and I coded a mail component to be used by other teams in our application. This component is designed to allow them to download emails using custom search conditions. The one that is being used at the moment is a mail search between a range of dates from Gmail accounts using IMAP protocol. It was working properly until April 4th 2016 6:00AM GMT+1, only downloading email within the range. However, since 8:00AM GMT+1, the response from the server suddenly changed and it started returning the whole INBOX folder!
Repeating the initial tests, we discovered that we were receiving the whole INBOX folder instead of those within the range. After some research down to the IMAP request, it seemed that Gmail changed the amount of emails returned under certain conditions:
SearchTerm had a ComparisonTerm.GT term, orSearchTerm had a ComparisonTerm.LE termIn both cases, Gmail's IMAP server returns all the emails in the folder. On the contrary, as an example, Yahoo's IMAP server just returns those emails which match the given condition.
We opened a ticket to them, asking if there was any change in their response, but they answered that they did not change anything. Of course, there is no way to prove it (we still don't have the time machine).
As we had a great impact with this unexpected behaviour, we decided to investigate further, and we created a simple mail client to download mail using different condition combinations from different IMAP servers.
Starting from this link's program, we added the functionality to check with other, but changing the library to meet our javax.mail version 1.5.5. Therefore, we avoid different implementation behaviours.
The code is attached below:
package gmail;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Properties;
import javax.mail.Folder;
import javax.mail.Message;
import javax.mail.Session;
import javax.mail.Store;
import javax.mail.search.AndTerm;
import javax.mail.search.ComparisonTerm;
import javax.mail.search.ReceivedDateTerm;
import javax.mail.search.SearchTerm;
public class JavaMailTest {
public static void main(String[] args) {
JavaMailTest testClient=new JavaMailTest();
//Yahoo
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GE,ComparisonTerm.LT);
//Gmail
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GE,ComparisonTerm.LT);
//Yahoo
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GE,ComparisonTerm.LE);
//Gmail
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GE,ComparisonTerm.LE);
//Yahoo
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GT,ComparisonTerm.LT);
//Gmail
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GT,ComparisonTerm.LT);
//Yahoo
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GT,ComparisonTerm.LE);
//Gmail
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GT,ComparisonTerm.LE);
}
public int performTest(Properties props, int startTermCondition, int endTermCondition)
{
try {
Session session = Session.getDefaultInstance(props, null);
Message[] totalMessages = null;
Store store = session.getStore(props.getProperty("protocol"));
store.connect(props.getProperty("mail.imap.host"), props.getProperty("account"), props.getProperty("pass"));
Folder inbox = store.getFolder("inbox");
inbox.open(Folder.READ_ONLY);
SimpleDateFormat ft = new SimpleDateFormat ("yyyy-MM-dd");
Date startDate = ft.parse("2016-04-03");
Date endDate = ft.parse("2016-04-07");
System.out.println("Testing email account: <"+props.getProperty("account")+"> with startDate: <" + startDate + "> (ComparisonTerm."+
getConditionText(startTermCondition)+") and endDate: <" + endDate+"> (ComparisonTerm."+getConditionText(endTermCondition)+")");
Message[] messages = inbox.getMessages();
if (messages != null && messages.length > 0) {
SearchTerm startDateTearm =
new ReceivedDateTerm(startTermCondition, startDate);
SearchTerm endDateTerm =
new ReceivedDateTerm(endTermCondition, endDate);
SearchTerm andTerm = new AndTerm(startDateTearm, endDateTerm);
totalMessages = inbox.search(andTerm);
if(totalMessages.length > 0){
System.out.println("Found "+totalMessages.length+" matching messages (Total messages in folder: "+messages.length+")");
}
/*for (int i = 0; i < totalMessages.length; i++) {
System.out.println(totalMessages[i].getReceivedDate() + " --> Mail Subject:- " + totalMessages[i].getSubject());
}*/
//First message
int currentMessage=0;
System.out.println("First Message: "+totalMessages[currentMessage].getReceivedDate() + " --> Mail Subject: " + totalMessages[currentMessage].getSubject().substring(0, 5));
//Last message
currentMessage=totalMessages.length-1;
System.out.println("Last Message: "+totalMessages[currentMessage].getReceivedDate() + " --> Mail Subject: " + totalMessages[currentMessage].getSubject().substring(0, 5));
}
inbox.close(true);
store.close();
return totalMessages.length;
} catch (Exception e) {
e.printStackTrace();
return -1;
}
}
public static String getConditionText(int condition)
{
String returnValue="";
switch (condition) {
case ComparisonTerm.EQ:
returnValue="EQ";
break;
case ComparisonTerm.GE:
returnValue="GE";
break;
case ComparisonTerm.GT:
returnValue="GT";
break;
case ComparisonTerm.LE:
returnValue="LE";
break;
case ComparisonTerm.LT:
returnValue="LT";
break;
case ComparisonTerm.NE:
returnValue="NE";
break;
default:
returnValue="Error";
break;
}
return returnValue;
}
public Properties getYahooImapProps()
{
Properties props = new Properties();
//Yahoo
String imapServer="imap.mail.yahoo.com";
String imapPort="993";
String account="********@yahoo.es";
String pass="*********";
props.setProperty("mail.imap.host", imapServer);
props.setProperty("mail.imap.port", imapPort);
props.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
props.setProperty("mail.imap.auth", "true");
props.setProperty("account", account);
props.setProperty("pass", pass);
props.setProperty("protocol", "imaps");
return props;
}
public Properties getGmailImapProps()
{
Properties props = new Properties();
//Gmail
String imapServer="imap.gmail.com";
String imapPort="143";
String account="******@gmail.com";
String pass="********";
props.setProperty("mail.imap.host", imapServer);
props.setProperty("mail.imap.port", imapPort);
/*props.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
props.setProperty("mail.imap.auth", "true");*/
props.setProperty("account", account);
props.setProperty("pass", pass);
props.setProperty("protocol", "imap");
return props;
}
}
To make the program output clearer, I added the request right before the first method output:
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********@yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********@gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 92 matching messages (Total messages in folder: 20138)
First Message: Sun Apr 03 04:08:47 CEST 2016 --> Mail Subject: Tu es
Last Message: Wed Apr 06 22:12:45 CEST 2016 --> Mail Subject: ¿Quie
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********@yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********@gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 20138 matching messages (Total messages in folder: 20138)
First Message: Sat Sep 04 13:23:35 CEST 2004 --> Mail Subject: Gmail
Last Message: Tue Apr 19 10:09:06 CEST 2016 --> Mail Subject: El re
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********@yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********@gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 20138 matching messages (Total messages in folder: 20138)
First Message: Sat Sep 04 13:23:35 CEST 2004 --> Mail Subject: Gmail
Last Message: Tue Apr 19 10:09:06 CEST 2016 --> Mail Subject: El re
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********@yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********@gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 20138 matching messages (Total messages in folder: 20138)
First Message: Sat Sep 04 13:23:35 CEST 2004 --> Mail Subject: Gmail
Last Message: Tue Apr 19 10:09:06 CEST 2016 --> Mail Subject: El re
From this result we can say the following:
We also tried with an Outlook account, with the same result as Gmail.
It sure looks like a bug in Gmail to me. I was able to reproduce it myself. I'm really surprised if Outlook.com has the same bug.
Why do you think the request is ambiguous? The IMAP spec is pretty clear.
A few notes...