How bad is IMAP IDLE?

Over on Mail-in-a-Box issue #129, @llchen223 noted that I hadn’t changed Dovecot‘s default setting of imap_idle_notify_interval. This setting controls the duration of IMAP IDLE pauses during which the mail client waits patiently for a new mail notification.

Here’s how it looks with K-9 mail (client) talking to Dovecot (server):

   CLIENT> 9 IDLE
   SERVER> + idling
   SERVER> * OK Still here
   . . . server waits imap_idle_notify_interval minutes . . .
   SERVER> * OK Still here
   . . . server waits imap_idle_notify_interval minutes . . .
   SERVER> * OK Still here

The dance ends when the server reports something other than OK, such as the presence of new mail, or if the client decides to go back into normal IMAP command mode. Multiple of these may be actually happening simultaneously in different connections if the mail client is monitoring for new mail in more than one folder.

This is pretty efficient. The * OK Still here message is just 444 bytes (over SSL!).

But the concern is that with the default 2 minute delay, that’s 720 × the number of monitored folders possible times a day that a phone has to do something. Does waking the phone’s networking capabilities drain the battery?  If multiple connections are open to monitor multiple folders Dovecot seems to smartly group the OKs together so that the phone is woken up just once — so at least there’s that.

None of this appears to have actually been a problem for my phone, and @llchen223 reports that having K-9 monitoring a mailbox has negligible impact on battery usage on his phone.

Peter Kieser suggested (three years ago) increasing imap_idle_notify_interval to be so long that the client checks in first (h/t @jkaberg). The IMAP IDLE standard says the server can hang up after 29 minutes if it hasn’t heard from the client, and so K-9 checks in after at most 24 minutes from the start of the IDLE by ending the IDLE (with DONE) and starting a new one (IDLE again). If imap_idle_notify_interval is more than 24 minutes, * OK Still here will never occur (I think). 

This sounds great, but the longer the connection remains totally silent the higher the risk that some intermediate point on the connection will decide the connection is broken and reset it. In recording these sessions, I ran into socat’s timeout of 15 minutes.

Not all mail clients are as efficient as K-9. Mozilla Thunderbird restarts the IDLE after each server OK:

   CLIENT> 7 IDLE
   SERVER> + idling
   . . . both sides wait imap_idle_notify_interval minutes . . .
   SERVER> * OK Still here
   CLIENT> DONE
   SERVER> 7 OK Idle completed.
   CLIENT> 8 noop
   SERVER> 8 OK NOOP completed.

Immediately after the NOOP at the end the client issues a new IDLE command and the pattern repeats. It only looks like 93 bytes, but inside an SSL connection it takes 1,292 bytes. (That’s both sides of the connection.) With the default Dovecot setting of a 2 minute delay, that’s a little less than 1 MB per day (× the number of monitored folders).

The default settings of Dovecot and K-9 seem to be just fine both in terms of bandwidth and battery usage, and there’s no indication that increasing the interval will benefit phones running K-9. Mozilla Thunderbird is not as good at minimizing bandwidth, but I don’t expect many people are running Mozilla Thunderbird on a mobile broadband connection. Maybe an increase of imap_idle_notify_interval from 2 to 4 minutes would be prudent.

I recorded the session with socat by having it act as an OpenSSL server (on a new port, here 992) to terminate the encryption, log the unencrypted IMAP stream, and proxy the IMAP traffic to a new SSL connection to the Dovecot IMAP server (port 993):

    socat -x 
        OPENSSL-LISTEN:992,fork,reuseaddr,bind=0.0.0.0,cert=ssl_certificate.pem,key=ssl_priv_key.pem,verify=0,rcvtimeo=9999999 
        OPENSSL:localhost:993,verify=0,rcvtimeo=9999999

This outputs the IMAP stream in hex, which happens to be a little cleaner than outputting the ASCII stream. socat’s default socket timeout was about 15 minutes, so I’ve also extended it to be able to handle the 24-minute IDLE length.

To measure the size of the encrypted traffic (including link-level headers and so on), I used tcpdump to monitor port 992:

    tcpdump -q -nn -e -ttttt -U "port 992"

I’m using Dovecot 2.2.9, K-9 4.804, and Mozilla Thunderbird 31.0. The exact sizes of the encrypted IDLE-related messages probably depend on which protocol and ciphers happen to be selected for the connection, so socat will be affecting those measurements.

(While recording these sessions, I noticed that K-9 would also check the Drafts folder every 90 seconds if no Drafts folder exists. As soon as the first draft was saved, causing K-9 to create the folder, this poll stopped. So I’ll have to revise Mail-in-a-Box to create the Drafts folder by default.)