I'm troubleshooting a process that (I think) has been working for a year or more. This solution updated a newer version of the solution software, and now it appears that one of our IMAP retrieval processes is not working correctly. (Before the update, I didn't hear any complaints about it - hence it must be working, right? :) But I don't have any data or samples from 'before the update' to know exactly what kind of results it was getting then.)
This solution is coded in FileMaker, using MBS plugin for the cURL functions. This process connects to a GMail account via IMAP protocol, using cURL commands. This process runs every hour.
In general, it works well - no errors. The problem is that it appears to be returning the wrong set of emails. We use the 'SEARCHSINCE 18-Oct-2021' function to restrict the emails found.
If I log into the account in a browser, I see 6 emails received on 10/19 in the Inbox.
When I do a search in the Gmail web interface: After:2021/10/18 I get 24 emails, and these emails include new ones received on 10/19.
• Actual: 11 emails... • BUT - these emails are from 10/18, not >=10/19... these results do NOT include anything from 10/19.
•Actual: 38 emails...
•BUT - some of these emails are from 10/15, not >=10/18; AND these results do NOT include anything from 10/19.
(I checked sample emails from both searches, and the 'Date' and 'ReceivedOn' headers in the source have 10/15 for the Date.)
•Result = No IDs
Here's an actual IMAP exchange:
Connected to imap.gmail.com (74.125.142.109) port 993 (#0)
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
Server certificate:
subject: CN=imap.gmail.com
start date: Sep 13 03:11:41 2021 GMT
expire date: Nov 20 03:11:40 2021 GMT
issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
old SSL session ID is stale, removing
* OK Gimap ready for requests from <server> <id>
A001 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
A001 OK Thats all she wrote! <id>
A002 AUTHENTICATE PLAIN <key>
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
A002 OK <email account> authenticated (Success)
A003 SELECT INBOX
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing \*)] Flags permitted.
* OK [UIDVALIDITY 1] UIDs valid.
* 5101 EXISTS
* 0 RECENT
* OK [UIDNEXT 5114] Predicted next UID.
* OK [HIGHESTMODSEQ 793152]
A003 OK [READ-WRITE] INBOX selected. (Success)
A004 SEARCH SENTSINCE 19-Oct-2021
* SEARCH 5091 5092 5093 5094 5095 5096 5097 5098 5099 5100 5101
A004 OK SEARCH completed (Success)
Connection #0 to host imap.gmail.com left intact
Anybody have ideas on how to get Google to work?
EDIT, 10/20: Have tested a few additional things, but I seem to be getting the same results when using IMAP, which don't match the results I get with the GMail web search interface.
There are 3 emails from today (10/20) in this Inbox.
Found an IMAP Extension for Gmail, and tried using that to do the search. It uses GMail-interface-style search inputs. Instead of 'SEARCH SENTSINCE', with this extension it looks like:
SEARCH X-GM-RAW "after:2021/10/19"
RESULT: not finding messages from 10/20
Slightly different form (this is the Unix epoch # of seconds for 2021/10/19):
SEARCH X-GM-RAW "after:1634626801"
RESULT: no emails from 10/20
Searching directly in the Gmail web interface: 'after:1634626801' = 19 results, including emails from 10/20 (today)
Have also tried:
SEARCH SINCE 2021-10-19
Result: no messages from 10/20 (does return ones from 10/19).
So the IMAP search results are consistent with themselves, at least, across the different methods of searching. But they aren't the results that I need - it's not finding the most recent stuff.
I finally figured it out. There was a bug in my FMP script for this process. It was not MBS, nor IMAP, nor Google (darn it).
Specifically, there were two different IMAP commands being made. The SEARCH (which is what I was showing above, where I thought the bug was), and then the RETRIEVE. The issue was that these two commands were referencing different IDs: IMAP has two IDs - default 'Id' and 'UID'. They are very similar - they are both serial numbers - but they are not the same. UID is more permanent. So my Search was returning message IDs of the 'default' variety, and the Retrieve was referencing the 'UID' value of emails.
Search (does NOT specify UID, so returns default ID):
SEARCH SINCE 10-Oct-2021
Retrieve (uses UID)
imaps://<ServerName>/INBOX?UID=...
New Search (returns UIDs, so that it matches Retrieve):
UID SEARCH SINCE 10-Oct-2021