Skip to main content

gimap efficiency when only reading headers

  4 posts   Feedicon  
Replies: 3 - Last Post: October 11, 2013 03:52
by: Bill Shannon
showing 1 - 4 of 4
Posted: October 09, 2013 19:34 by Benjamin
I'm loading a LOT of messages from my gmail account, and going through the "from" and "to" lists. It seems to be taking a long time.

Even though I added everything available to the FetchProfile,

    FetchProfile fp = new FetchProfile();
    fp.add(GmailFolder.FetchProfileItem.ENVELOPE);
    fp.add(GmailFolder.FetchProfileItem.CONTENT_INFO);
    fp.add(GmailFolder.FetchProfileItem.FLAGS);
    fp.add(GmailFolder.FetchProfileItem.LABELS);
    fp.add(GmailFolder.FetchProfileItem.MSGID);
    fp.add(GmailFolder.FetchProfileItem.THRID);
    allMailFolder.fetch(allMessages, fp);


profiling shows that every call is trying to fetch the message body for some reason, which I'm sure is the reason it is so slow. Any ideas on how to make this fast, if all I want is the to/from fields?

    for (final Message message : allMessages) {
        final Address[] addrs = message.getFrom();
        if (addrs != null) {
          for (final Address addr : addrs) {
            System.out.println(addr.toString());


getAllRecipients - getRecipients - getHeader - peekBody - fetchBody
Posted: October 10, 2013 04:43 by Bill Shannon
What exactly is "every call" doing? Just calling message.getFrom()? All of that data should come from the envelope, which you're prefetching.

What does the Session debug output show?
Posted: October 10, 2013 16:29 by Benjamin
It is actually the message.getAllRecipients that causes the per-message call back to the server. Which is strange, because I can see the recipient (me) in the envelope fetch.


run:
DEBUG: setDebug: JavaMail version 1.5.0
DEBUG: getProvider() returning javax.mail.Provider[STORE,gimaps,com.sun.mail.gimap.GmailSSLStore,Oracle]
DEBUG GIMAPS: mail.imap.fetchsize: 16384
DEBUG GIMAPS: mail.imap.ignorebodystructuresize: false
DEBUG GIMAPS: mail.imap.statuscachetimeout: 1000
DEBUG GIMAPS: mail.imap.appendbuffersize: -1
DEBUG GIMAPS: mail.imap.minidletime: 10
DEBUG GIMAPS: trying to connect to host "imap.gmail.com", port 993, isSSL true
* OK Gimap ready for requests from ...
A0 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
A0 OK Thats all she wrote! 
DEBUG GIMAPS: AUTH: XOAUTH
DEBUG GIMAPS: AUTH: XOAUTH2
DEBUG GIMAPS: AUTH: PLAIN
DEBUG GIMAPS: AUTH: PLAIN-CLIENTTOKEN
DEBUG GIMAPS: connected to Gmail
DEBUG GIMAPS: protocolConnect login, host=imap.gmail.com, user=..., password=<non-null>
DEBUG GIMAPS: AUTHENTICATE PLAIN command trace suppressed
DEBUG GIMAPS: AUTHENTICATE PLAIN command result: A1 OK ... authenticated (Success)
A2 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH
A2 OK Success
DEBUG GIMAPS: connection available -- size: 1
A3 EXAMINE "[Gmail]/All Mail"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Forwarded $NotJunk $Junk)
* OK [PERMANENTFLAGS ()] Flags permitted.
* OK [UIDVALIDITY 671783979] UIDs valid.
* 77363 EXISTS
* 0 RECENT
* OK [UIDNEXT 124006] Predicted next UID.
* OK [HIGHESTMODSEQ 2667826]
A3 OK [READ-ONLY] [Gmail]/All Mail selected. (Success)
Messages:77363
A4 FETCH 1:77363 (ENVELOPE INTERNALDATE RFC822.SIZE)


.. snip ...

* 11 FETCH (RFC822.SIZE 15307 INTERNALDATE "14-Mar-2005 19:56:23 +0000" ENVELOPE ("Mon, 14 Mar 2005 13:56:21 -0600" "Your Car Rental Confirmation" (("Orbitz Traveler Care" NIL "travelercare" "orbitz.com")) (("Orbitz Traveler Care" NIL "travelercare" "orbitz.com")) (("Orbitz Traveler Care" NIL "travelercare" "orbitz.com")) (("Benjamin M. Hill" NIL "benjaminhill" "gmail.com")) NIL NIL NIL "<13131856.1110830181225.JavaMail.orbitz@wmadmwl04pp.wm.orbitz.com>"))


... snip ...

During every getAllRecipients call:

A14 OK Success
A15 FETCH 11 (BODY.PEEK[HEADER.FIELDS (Newsgroups)])
* 11 FETCH (BODY[HEADER.FIELDS (Newsgroups)] {4}
Posted: October 11, 2013 03:52 by Bill Shannon
Ya, most email messages don't have newsgroups as recipients so you probably don't want to use getAllRecipients. Or, you could add the Newsgroups header to the data that you prefetch for each message.
Replies: 3 - Last Post: October 11, 2013 03:52
by: Bill Shannon
 
 
Close
loading
Please Confirm
Close