How bad is IMAP IDLE?

Over on Mail-in-a-Box issue #129@llchen223 noted that I hadn’t changed Dovecot‘s default setting of imap_idle_notify_interval. This setting controls the duration of IMAP IDLE pauses during which the mail client waits patiently for a new mail notification.

Here’s how it looks with K-9 mail (client) talking to Dovecot (server):

   CLIENT> 9 IDLE
   SERVER> + idling
   SERVER> * OK Still here
   . . . server waits imap_idle_notify_interval minutes . . .
   SERVER> * OK Still here
   . . . server waits imap_idle_notify_interval minutes . . .
   SERVER> * OK Still here

The dance ends when the server reports something other than OK, such as the presence of new mail, or if the client decides to go back into normal IMAP command mode. Multiple of these may be actually happening simultaneously in different connections if the mail client is monitoring for new mail in more than one folder.

This is pretty efficient. The * OK Still here message is just 444 bytes (over SSL!).

But the concern is that with the default 2 minute delay, that’s 720 × the number of monitored folders possible times a day that a phone has to do something. Does waking the phone’s networking capabilities drain the battery?  If multiple connections are open to monitor multiple folders Dovecot seems to smartly group the OKs together so that the phone is woken up just once — so at least there’s that.

None of this appears to have actually been a problem for my phone, and @llchen223 reports that having K-9 monitoring a mailbox has negligible impact on battery usage on his phone.

Peter Kieser suggested (three years ago) increasing imap_idle_notify_interval to be so long that the client checks in first (h/t @jkaberg). The IMAP IDLE standard says the server can hang up after 29 minutes if it hasn’t heard from the client, and so K-9 checks in after at most 24 minutes from the start of the IDLE by ending the IDLE (with DONE) and starting a new one (IDLE again). If imap_idle_notify_interval is more than 24 minutes, * OK Still here will never occur (I think). 

This sounds great, but the longer the connection remains totally silent the higher the risk that some intermediate point on the connection will decide the connection is broken and reset it. In recording these sessions, I ran into socat’s timeout of 15 minutes.

Not all mail clients are as efficient as K-9. Mozilla Thunderbird restarts the IDLE after each server OK:

   CLIENT> 7 IDLE
   SERVER> + idling
   . . . both sides wait imap_idle_notify_interval minutes . . .
   SERVER> * OK Still here
   CLIENT> DONE
   SERVER> 7 OK Idle completed.
   CLIENT> 8 noop
   SERVER> 8 OK NOOP completed.

Immediately after the NOOP at the end the client issues a new IDLE command and the pattern repeats. It only looks like 93 bytes, but inside an SSL connection it takes 1,292 bytes. (That’s both sides of the connection.) With the default Dovecot setting of a 2 minute delay, that’s a little less than 1 MB per day (× the number of monitored folders).

The default settings of Dovecot and K-9 seem to be just fine both in terms of bandwidth and battery usage, and there’s no indication that increasing the interval will benefit phones running K-9. Mozilla Thunderbird is not as good at minimizing bandwidth, but I don’t expect many people are running Mozilla Thunderbird on a mobile broadband connection. Maybe an increase of imap_idle_notify_interval from 2 to 4 minutes would be prudent.

I recorded the session with socat by having it act as an OpenSSL server (on a new port, here 992) to terminate the encryption, log the unencrypted IMAP stream, and proxy the IMAP traffic to a new SSL connection to the Dovecot IMAP server (port 993):

    socat -x 
        OPENSSL-LISTEN:992,fork,reuseaddr,bind=0.0.0.0,cert=ssl_certificate.pem,key=ssl_priv_key.pem,verify=0,rcvtimeo=9999999 
        OPENSSL:localhost:993,verify=0,rcvtimeo=9999999

This outputs the IMAP stream in hex, which happens to be a little cleaner than outputting the ASCII stream. socat’s default socket timeout was about 15 minutes, so I’ve also extended it to be able to handle the 24-minute IDLE length.

To measure the size of the encrypted traffic (including link-level headers and so on), I used tcpdump to monitor port 992:

    tcpdump -q -nn -e -ttttt -U "port 992"

I’m using Dovecot 2.2.9, K-9 4.804, and Mozilla Thunderbird 31.0. The exact sizes of the encrypted IDLE-related messages probably depend on which protocol and ciphers happen to be selected for the connection, so socat will be affecting those measurements.

(While recording these sessions, I noticed that K-9 would also check the Drafts folder every 90 seconds if no Drafts folder exists. As soon as the first draft was saved, causing K-9 to create the folder, this poll stopped. So I’ll have to revise Mail-in-a-Box to create the Drafts folder by default.)

Sunsets over DC

Last week I noticed that the sunset aligned unusually well with my cross-street, Newton St NW, and it made me wonder if we have any Manhattanhenge-like events in DC. DC can one-up Manhattan — we’ve got a double-henge, if you’ll let me coin a phrase.

The Double-henge

Here in Columbia Heights we have a unique street pattern. Two roads — Park Rd and Monroe St. —  come to an apex on 14th St. They go north both to the east and west of 14th St. On a few days a year — centered on May 15 and July 29 — the roads point east toward sunrise and west toward sunset. Click the links to see on suncalc.net. (The alignment isn’t exact, so the effect spans a few days.)

All the henges

Like Manhattan, DC’s grid lines up with sunrise & sunset. It’s on the equinoxes, so we get a boring double-henge on those days too.

Some of the state avenues are kind of close to the solar azimuths on the solstices, but the peak days are a few days off. In the summer it is on the same days as the Columbia Heights Doublehenge. On those days the avenues parallel to New York Avenue line up with sunrise and the avenues parallel to Pennsylvania Avenue line up with sunset. Around the winter solstice — Nov 5 and Feb 6 — the avenues parallel to Pennsylvania Avenue line up with sunrise and the avenues parallel to New York Avenue line up with sunset.

I wondered for each day of the year, what was the DC road that best aligns with sunrise and sunset. If you’re driving these would also be the roads to avoid (h/t @knowtheory). Here’s a table for the next year. The links will show you where exactly it is:

Date Sunrise Street Sunset Street
2014-06-20 Military Rd NW Ridge Rd SE
2014-06-21 Military Rd NW Ridge Rd SE
2014-06-22 Military Rd NW Ridge Rd SE
2014-06-23 Military Rd NW Ridge Rd SE
2014-06-24 Military Rd NW Ridge Rd SE
2014-06-25 Military Rd NW Ridge Rd SE
2014-06-26 Nebraska Ave NW Ridge Rd SE
2014-06-27 Nebraska Ave NW Ridge Rd SE
2014-06-28 Nebraska Ave NW Ridge Rd SE
2014-06-29 Nebraska Ave NW Mount Olivet Rd NE
2014-06-30 Nebraska Ave NW Mount Olivet Rd NE
2014-07-01 Nebraska Ave NW Pennsylvania Ave SE
2014-07-02 Nebraska Ave NW Pennsylvania Ave SE
2014-07-03 Nebraska Ave NW Pennsylvania Ave SE
2014-07-04 Nebraska Ave NW Pennsylvania Ave SE
2014-07-05 Nebraska Ave NW Pennsylvania Ave SE
2014-07-06 Nebraska Ave NW Reno Rd NW
2014-07-07 Nebraska Ave NW Reno Rd NW
2014-07-08 Anacostia Dr SE Thomas Rd SW
2014-07-09 Anacostia Dr SE Thomas Rd SW
2014-07-10 Anacostia Dr SE Thomas Rd SW
2014-07-11 DC Hwy 295 R St SE
2014-07-12 DC Hwy 295 R St SE
2014-07-13 Minnesota Ave SE R St SE
2014-07-14 Minnesota Ave SE R St SE
2014-07-15 DC Hwy 295 R St SE
2014-07-16 US Hwy 1 Macarthur Blvd NW
2014-07-17 US Hwy 1 Macarthur Blvd NW
2014-07-18 US Hwy 1 Macarthur Blvd NW
2014-07-19 US Hwy 1 Florida Ave NE
2014-07-20 US Hwy 1 Neal St NE
2014-07-21 Legation St NW Neal St NE
2014-07-22 Legation St NW Morse St NE
2014-07-23 Legation St NW Morse St NE
2014-07-24 Mississippi Ave SE Pennsylvania Ave SE
2014-07-25 Mississippi Ave SE Pennsylvania Ave SE
2014-07-26 Mississippi Ave SE Pennsylvania Ave SE
2014-07-27 US Hwy 1 Alt Monroe St NW
2014-07-28 US Hwy 1 Alt Monroe St NW
2014-07-29 US Hwy 1 Alt Park Rd NW
2014-07-30 Potomac Ave SE Park Rd NW
2014-07-31 US Hwy 1 Alt Lamont St NW
2014-08-01 US Hwy 1 Alt Pennsylvania Ave NW
2014-08-02 Firth Sterling Ave SE Massachusetts Ave NW
2014-08-03 Myrtle Ave NE Pennsylvania Ave NW
2014-08-04 Arlington Memorial Brg Missouri Ave NW
2014-08-05 Arlington Memorial Brg Missouri Ave NW
2014-08-06 Arlington Memorial Brg Missouri Ave NW
2014-08-07 US Hwy 50 Missouri Ave NW
2014-08-08 US Hwy 50 Canal Rd NW
2014-08-09 US Hwy 50 Canal Rd NW
2014-08-10 US Hwy 1 Virginia Ave SE
2014-08-11 US Hwy 1 Pennsylvania Ave NW
2014-08-12 US Hwy 1 Pennsylvania Ave NW
2014-08-13 US Hwy 1 Pennsylvania Ave NW
2014-08-14 US Hwy 1 Spring Rd NW
2014-08-15 Alabama Ave SE Howard Rd SE
2014-08-16 Savannah St SE March Ln SW
2014-08-17 S Carolina Ave SE Macdill Blvd SW
2014-08-18 S Carolina Ave SE McChord St SW
2014-08-19 S Carolina Ave SE Macdill Blvd SW
2014-08-20 Valley Ave SE Macdill Blvd SW
2014-08-21 Valley Ave SE Military Rd NW
2014-08-22 Alabama Ave SE Kalmia Rd NW
2014-08-23 Riggs Rd NE Kalmia Rd NW
2014-08-24 Mc Guire Ave SE Good Hope Rd SE
2014-08-25 Mc Guire Ave SE Gales St NE
2014-08-26 Whittier St NW Gales St NE
2014-08-27 Whittier St NW Military Rd NW
2014-08-28 Alabama Ave SE Military Rd NW
2014-08-29 Savannah St SE C St SE
2014-08-30 Kenyon St NW Brooks St NE
2014-08-31 Kenyon St NW Brooks St NE
2014-09-01 Princeton Pl NW Blaine St NE
2014-09-02 Princeton Pl NW Atlantic St SE
2014-09-03 Princeton Pl NW Jonquil St NW
2014-09-04 Roxanna Rd NW H St SE
2014-09-05 Perimeter North Rd SW Watson St NW
2014-09-06 W St NW Woodley Rd NW
2014-09-07 W St NW Calvert St NW
2014-09-08 W St NW Independence Ave SW
2014-09-09 Central Ave NE Independence Ave SW
2014-09-10 Chapin St NW Independence Ave SW
2014-09-11 Jackson St NE Independence Ave SW
2014-09-12 Newton St NE Forrester St SW
2014-09-13 Ingraham St NW H St NE
2014-09-14 Webster St NW L St NW
2014-09-15 Emerson St NW Morrison St NW
2014-09-16 Madison Dr NW Kennedy St NW
2014-09-17 McKinley St NW Emerson St NW
2014-09-18 L St NW Ingraham St NW
2014-09-19 US Hwy 50 Newton St NE
2014-09-20 Forrester St SW Newton St NE
2014-09-21 Independence Ave SW Jackson St NE
2014-09-22 Independence Ave SW Central Ave NE
2014-09-23 Independence Ave SW Central Ave NE
2014-09-24 V St NE W St NW
2014-09-25 Watson St NW W St NW
2014-09-26 Watson St NW Perimeter North Rd SW
2014-09-27 H St SE Perimeter North Rd SW
2014-09-28 Jonquil St NW Princeton Pl NW
2014-09-29 Atlantic St SE Princeton Pl NW
2014-09-30 Atlantic St SE Princeton Pl NW
2014-10-01 Brooks St NE Kenyon St NW
2014-10-02 Brooks St NE Kenyon St NW
2014-10-03 C St SE Alabama Ave SE
2014-10-04 Military Rd NW Alabama Ave SE
2014-10-05 Benning Rd NE Whittier St NW
2014-10-06 Gales St NE Mc Guire Ave SE
2014-10-07 Good Hope Rd SE Mc Guire Ave SE
2014-10-08 Gales St NE Mc Guire Ave SE
2014-10-09 Kalmia Rd NW Riggs Rd NE
2014-10-10 Kalmia Rd NW Valley Ave SE
2014-10-11 Macdill Blvd SW Valley Ave SE
2014-10-12 Macdill Blvd SW S Carolina Ave SE
2014-10-13 McChord St SW S Carolina Ave SE
2014-10-14 Blanchard Dr SW S Carolina Ave SE
2014-10-15 March Ln SW Alabama Ave SE
2014-10-16 Howard Rd SE US Hwy 1
2014-10-17 Pennsylvania Ave NW US Hwy 1
2014-10-18 Pennsylvania Ave NW US Hwy 1
2014-10-19 Pennsylvania Ave NW US Hwy 1
2014-10-20 Pennsylvania Ave NW US Hwy 1
2014-10-21 Canal Rd NW US Hwy 50
2014-10-22 Douglas St NE US Hwy 50
2014-10-23 Missouri Ave NW US Hwy 50
2014-10-24 Missouri Ave NW Arlington Memorial Brg
2014-10-25 Pennsylvania Ave NW Arlington Memorial Brg
2014-10-26 Massachusetts Ave NW Firth Sterling Ave SE
2014-10-27 Pennsylvania Ave NW US Hwy 1 Alt
2014-10-28 Lamont St NW US Hwy 1 Alt
2014-10-29 Park Rd NW US Hwy 1 Alt
2014-10-30 Monroe St NW US Hwy 1 Alt
2014-10-31 Monroe St NW US Hwy 1 Alt
2014-11-01 Pennsylvania Ave SE Mississippi Ave SE
2014-11-02 Pennsylvania Ave SE Mississippi Ave SE
2014-11-03 Morse St NE Legation St NW
2014-11-04 Morse St NE Legation St NW
2014-11-05 Oates St NE US Hwy 1
2014-11-06 Florida Ave NE US Hwy 1
2014-11-07 Macarthur Blvd NW US Hwy 1
2014-11-08 R St SE Minnesota Ave SE
2014-11-09 R St SE Minnesota Ave SE
2014-11-10 R St SE Anacostia Dr SE
2014-11-11 Thomas Rd SW Anacostia Dr SE
2014-11-12 Thomas Rd SW Nebraska Ave NW
2014-11-13 Pennsylvania Ave SE Nebraska Ave NW
2014-11-14 Pennsylvania Ave SE Nebraska Ave NW
2014-11-15 Mount Olivet Rd NE Military Rd NW
2014-11-16 Ridge Rd SE Military Rd NW
2014-11-17 Ridge Rd SE Military Rd NW
2014-11-18 Linnean Ave NW Military Rd NW
2014-11-19 Virginia Ave NW Kalorama Rd NW
2014-11-20 Virginia Ave NW Kalorama Rd NW
2014-11-21 Virginia Ave NW Kalorama Rd NW
2014-11-22 Pope St SE DC Hwy 295
2014-11-23 Pope St SE DC Hwy 295
2014-11-24 Aeration Rd SW Cathedral Ave NW
2014-11-25 Aeration Rd SW Cathedral Ave NW
2014-11-26 Aeration Rd SW Cathedral Ave NW
2014-11-27 Aeration Rd SW Westover Ave SW
2014-11-28 Newcomb St SE Condon Ter SE
2014-11-29 Newcomb St SE Mississippi Ave SE
2014-11-30 Mellon St SE Mississippi Ave SE
2014-12-01 Sumner Rd SE Mississippi Ave SE
2014-12-02 Sumner Rd SE US Hwy 1 Alt
2014-12-03 Sumner Rd SE US Hwy 1 Alt
2014-12-04 Howard Rd SE US Hwy 1 Alt
2014-12-05 Howard Rd SE US Hwy 1 Alt
2014-12-06 13th St NE Vista St NE
2014-12-07 Ainger Pl SE Vista St NE
2014-12-08 Ainger Pl SE Vista St NE
2014-12-09 Ainger Pl SE Vista St NE
2014-12-10 Ainger Pl SE Vista St NE
2014-12-11 S Dakota Ave NE US Hwy 1 Alt
2014-12-12 S Dakota Ave NE US Hwy 1 Alt
2014-12-13 S Dakota Ave NE US Hwy 1 Alt
2014-12-14 S Dakota Ave NE US Hwy 1 Alt
2014-12-15 S Dakota Ave NE US Hwy 1 Alt
2014-12-16 S Dakota Ave NE US Hwy 1 Alt
2014-12-17 S Dakota Ave NE US Hwy 1 Alt
2014-12-18 S Dakota Ave NE US Hwy 1 Alt
2014-12-19 Montana Ave NE US Hwy 1 Alt
2014-12-20 Montana Ave NE US Hwy 1 Alt
2014-12-21 Montana Ave NE US Hwy 1 Alt
2014-12-22 Montana Ave NE US Hwy 1 Alt
2014-12-23 Montana Ave NE US Hwy 1 Alt
2014-12-24 S Dakota Ave NE US Hwy 1 Alt
2014-12-25 S Dakota Ave NE US Hwy 1 Alt
2014-12-26 S Dakota Ave NE US Hwy 1 Alt
2014-12-27 S Dakota Ave NE US Hwy 1 Alt
2014-12-28 S Dakota Ave NE US Hwy 1 Alt
2014-12-29 S Dakota Ave NE US Hwy 1 Alt
2014-12-30 S Dakota Ave NE US Hwy 1 Alt
2014-12-31 S Dakota Ave NE US Hwy 1 Alt
2015-01-01 Ainger Pl SE US Hwy 1 Alt
2015-01-02 Ainger Pl SE Vista St NE
2015-01-03 Ainger Pl SE Vista St NE
2015-01-04 Ainger Pl SE Vista St NE
2015-01-05 13th St NE Vista St NE
2015-01-06 Howard Rd SE US Hwy 1 Alt
2015-01-07 Howard Rd SE Lanier Pl NW
2015-01-08 Sumner Rd SE US Hwy 1 Alt
2015-01-09 Sumner Rd SE US Hwy 1 Alt
2015-01-10 Sumner Rd SE Mississippi Ave SE
2015-01-11 Newcomb St SE Mississippi Ave SE
2015-01-12 Newcomb St SE Mississippi Ave SE
2015-01-13 Newcomb St SE Condon Ter SE
2015-01-14 Newcomb St SE Westover Ave SW
2015-01-15 Aeration Rd SW Cathedral Ave NW
2015-01-16 Aeration Rd SW Cathedral Ave NW
2015-01-17 Aeration Rd SW Cathedral Ave NW
2015-01-18 Aeration Rd SW DC Hwy 295
2015-01-19 Pope St SE DC Hwy 295
2015-01-20 Pope St SE Kalorama Rd NW
2015-01-21 Virginia Ave NW Kalorama Rd NW
2015-01-22 Virginia Ave NW Kalorama Rd NW
2015-01-23 Virginia Ave NW Military Rd NW
2015-01-24 Linnean Ave NW Military Rd NW
2015-01-25 Ridge Rd SE Military Rd NW
2015-01-26 Ridge Rd SE Nebraska Ave NW
2015-01-27 Pennsylvania Ave SE Nebraska Ave NW
2015-01-28 Pennsylvania Ave SE Nebraska Ave NW
2015-01-29 Thomas Rd SW Nebraska Ave NW
2015-01-30 Thomas Rd SW Anacostia Dr SE
2015-01-31 R St SE DC Hwy 295
2015-02-01 R St SE Minnesota Ave SE
2015-02-02 R St SE Minnesota Ave SE
2015-02-03 Macarthur Blvd NW US Hwy 1
2015-02-04 Florida Ave NE US Hwy 1
2015-02-05 Neal St NE US Hwy 1
2015-02-06 Morse St NE Legation St NW
2015-02-07 Morse St NE Legation St NW
2015-02-08 Pennsylvania Ave SE Mississippi Ave SE
2015-02-09 Pennsylvania Ave SE Mississippi Ave SE
2015-02-10 S Capitol St SE US Hwy 1 Alt
2015-02-11 Monroe St NW US Hwy 1 Alt
2015-02-12 Park Rd NW US Hwy 1 Alt
2015-02-13 Lamont St NW US Hwy 1 Alt
2015-02-14 Pennsylvania Ave NW Firth Sterling Ave SE
2015-02-15 Massachusetts Ave NW Myrtle Ave NE
2015-02-16 Pennsylvania Ave NW Arlington Memorial Brg
2015-02-17 Missouri Ave NW Arlington Memorial Brg
2015-02-18 Missouri Ave NW US Hwy 50
2015-02-19 Douglas St NE US Hwy 50
2015-02-20 Canal Rd NW US Hwy 50
2015-02-21 Nash St SE US Hwy 1
2015-02-22 Pennsylvania Ave NW US Hwy 1
2015-02-23 Pennsylvania Ave NW US Hwy 1
2015-02-24 Pennsylvania Ave NW US Hwy 1
2015-02-25 Howard Rd SE Alabama Ave SE
2015-02-26 March Ln SW S Carolina Ave SE
2015-02-27 Blanchard Dr SW S Carolina Ave SE
2015-02-28 McChord St SW S Carolina Ave SE
2015-03-01 Macdill Blvd SW Valley Ave SE
2015-03-02 Macdill Blvd SW Valley Ave SE
2015-03-03 Kalmia Rd NW Alabama Ave SE
2015-03-04 Kalmia Rd NW Riggs Rd NE
2015-03-05 Gales St NE Mc Guire Ave SE
2015-03-06 Good Hope Rd SE Mc Guire Ave SE
2015-03-07 Gales St NE Whittier St NW
2015-03-08 Benning Rd NE Alabama Ave SE
2015-03-09 Military Rd NW Alabama Ave SE
2015-03-10 C St SE Kenyon St NW
2015-03-11 Brooks St NE Kenyon St NW
2015-03-12 Brooks St NE Princeton Pl NW
2015-03-13 Atlantic St SE Princeton Pl NW
2015-03-14 Atlantic St SE Princeton Pl NW
2015-03-15 Woodley Rd NW Roxanna Rd NW
2015-03-16 Watson St NW Perimeter North Rd SW
2015-03-17 H St SE W St NW
2015-03-18 Calvert St NW W St NW
2015-03-19 V St NE W St NW
2015-03-20 Independence Ave SW Central Ave NE
2015-03-21 Independence Ave SW Calvert St NW
2015-03-22 Independence Ave SW Newton St NE
2015-03-23 Forrester St SW Newton St NE
2015-03-24 H St NE Ingraham St NW
2015-03-25 L St NW Emerson St NW
2015-03-26 Morrison St NW Kennedy St NW
2015-03-27 Kennedy St NW Morrison St NW
2015-03-28 Emerson St NW L St NW
2015-03-29 Webster St NW H St NE
2015-03-30 Newton St NE Forrester St SW
2015-03-31 Newton St NE Independence Ave SW
2015-04-01 Jackson St NE Independence Ave SW
2015-04-02 Central Ave NE Independence Ave SW
2015-04-03 Central Ave NE V St NE
2015-04-04 W St NW Calvert St NW
2015-04-05 W St NW Woodley Rd NW
2015-04-06 Bryant St NW Watson St NW
2015-04-07 Perimeter North Rd SW H St SE
2015-04-08 Princeton Pl NW Atlantic St SE
2015-04-09 Princeton Pl NW Atlantic St SE
2015-04-10 Princeton Pl NW Blaine St NE
2015-04-11 Irvington St SW Eads St NE
2015-04-12 Kenyon St NW Brooks St NE
2015-04-13 Savannah St SE C St SE
2015-04-14 Alabama Ave SE Military Rd NW
2015-04-15 I- 295 Benning Rd NE
2015-04-16 Whittier St NW Gales St NE
2015-04-17 Mc Guire Ave SE Good Hope Rd SE
2015-04-18 Mc Guire Ave SE Gales St NE
2015-04-19 Riggs Rd NE Kalmia Rd NW
2015-04-20 Riggs Rd NE Kalmia Rd NW
2015-04-21 Valley Ave SE Military Rd NW
2015-04-22 Valley Ave SE Macdill Blvd SW
2015-04-23 Alabama Ave SE Macdill Blvd SW
2015-04-24 S Carolina Ave SE McChord St SW
2015-04-25 S Carolina Ave SE Blanchard Dr SW
2015-04-26 S Carolina Ave SE March Ln SW
2015-04-27 Alabama Ave SE Howard Rd SE
2015-04-28 US Hwy 1 Pennsylvania Ave NW
2015-04-29 US Hwy 1 Pennsylvania Ave NW
2015-04-30 US Hwy 1 Pennsylvania Ave NW
2015-05-01 US Hwy 1 Virginia Ave SE
2015-05-02 US Hwy 50 Canal Rd NW
2015-05-03 US Hwy 50 Canal Rd NW
2015-05-04 US Hwy 50 Douglas St NE
2015-05-05 US Hwy 50 Missouri Ave NW
2015-05-06 Arlington Memorial Brg Missouri Ave NW
2015-05-07 Arlington Memorial Brg Missouri Ave NW
2015-05-08 Arlington Memorial Brg Pennsylvania Ave NW
2015-05-09 Myrtle Ave NE Massachusetts Ave NW
2015-05-10 Firth Sterling Ave SE Pennsylvania Ave NW
2015-05-11 US Hwy 1 Alt Pennsylvania Ave NW
2015-05-12 US Hwy 1 Alt Monroe St NW
2015-05-13 Potomac Ave SE Park Rd NW
2015-05-14 US Hwy 1 Alt Monroe St NW
2015-05-15 US Hwy 1 Alt Monroe St NW
2015-05-16 US Hwy 1 Alt Pennsylvania Ave SE
2015-05-17 Mississippi Ave SE Pennsylvania Ave SE
2015-05-18 Mississippi Ave SE Pennsylvania Ave SE
2015-05-19 Mississippi Ave SE Pennsylvania Ave SE
2015-05-20 Legation St NW Morse St NE
2015-05-21 Legation St NW Morse St NE
2015-05-22 Legation St NW Neal St NE
2015-05-23 US Hwy 1 Neal St NE
2015-05-24 US Hwy 1 Florida Ave NE
2015-05-25 US Hwy 1 Macarthur Blvd NW
2015-05-26 US Hwy 1 Macarthur Blvd NW
2015-05-27 US Hwy 1 Macarthur Blvd NW
2015-05-28 Minnesota Ave SE R St SE
2015-05-29 Minnesota Ave SE R St SE
2015-05-30 Minnesota Ave SE R St SE
2015-05-31 DC Hwy 295 R St SE
2015-06-01 Anacostia Dr SE R St SE
2015-06-02 Anacostia Dr SE Thomas Rd SW
2015-06-03 Anacostia Dr SE Thomas Rd SW
2015-06-04 Anacostia Dr SE Thomas Rd SW
2015-06-05 Nebraska Ave NW Thomas Rd SW
2015-06-06 Nebraska Ave NW Reno Rd NW
2015-06-07 Nebraska Ave NW Pennsylvania Ave SE
2015-06-08 Nebraska Ave NW Pennsylvania Ave SE
2015-06-09 Nebraska Ave NW Pennsylvania Ave SE
2015-06-10 Nebraska Ave NW Pennsylvania Ave SE
2015-06-11 Nebraska Ave NW Pennsylvania Ave SE
2015-06-12 Nebraska Ave NW Pennsylvania Ave SE
2015-06-13 Nebraska Ave NW Mount Olivet Rd NE
2015-06-14 Nebraska Ave NW Ridge Rd SE
2015-06-15 Nebraska Ave NW Ridge Rd SE
2015-06-16 Military Rd NW Ridge Rd SE
2015-06-17 Military Rd NW Ridge Rd SE
2015-06-18 Military Rd NW Ridge Rd SE
2015-06-19 Military Rd NW Ridge Rd SE
2015-06-20 Military Rd NW Ridge Rd SE

source code

User Experience at “Tunnel Creek”: What we can all learn from The New York Times’s Snow Fall piece

Just a few paragraphs in to The New York Times’s six-part Snow Fall series, I was captivated equally by the story and by the innovative magazine-style in which the story was presented. So I began taking notes about the user experience of reading Snow Fall, knowing there would be a lot to learn for other user interface projects.

Continue reading “User Experience at “Tunnel Creek”: What we can all learn from The New York Times’s Snow Fall piece”

Transparent Tethering for Android & Linux

Update 10/4/2010: New post that does this in a much simpler way using OpenVPN.

I’m a new Android user on my HTC Incredible, and while I’ve read that the next version of Android is going to support tethering, it might take a while for this to be supported on existing HTC devices and I’m not sure I believe Verizon will let HTC ship it. So the hacker that I am, I found a relatively elegant way to tether. The only catch is that you need some other Linux machine with a public facing address on the Internet to serve as your “router”.

The only other good technique I found that works on Linux is to use an app that provides port forwarding or a SOCKS server. This isn’t really tethering as I understand it. SOCKS gets you only so far: it won’t do DNS, for instance, and it only works in applications that support SOCKS. The solution here provides what looks like a full Internet connection to the tethered device.

The gist is:

  • The laptop you want to tether uses the socat tool to create basically a virtual ethernet device (“tun0”) that forwards packets over a TCP connection, rather than a physical cable. The tun0 device is set up as the default gateway for the computer so that its Internet connection goes through the virtual device.
  • This TCP connection goes first to localhost, but an Andr0id SDK tool forwards the TCP connection over the USB cable to your phone.
  • The Connect Bot app on your phone makes an SSH connection to the host computer and sets up port forwarding from the phone to the host. I’m not particularly concerned about encrypting the traffic, but the advantage of using a port-forward over SSH (versus using e.g. the Internet Sharer app by JADS to forward directly) is that on the host side we can make sure that we only provide Internet sharing for ourselves and not the whole world.
  • The host computer is listening for the connection from localhost only and forwards packets from the connection back into a second virtual ethernet device (“tun0” on the hots). Internet sharing (NAT) is enabled for the tun0 device so that the packets get routed to the Internet.

You will need…

  • An Android phone. I’m testing on Android 2.1. No root access is required here.
  • A Linux computer to tether. I’m testing on Ubuntu 9.04. Root access required.
  • A Linux computer that has a permanent connection to the Internet with a public IP address on which you have root access. This is the host computer. I’m testing this one with Ubuntu 10.04.

Before you begin…

On the phone, install the free Connect Bot app from the marketplace. Enable USB debugging on the phone in Settings -> Applications -> Development -> USB debugging, which allows us to forward connections over the USB cable.

On the computer to tether, while you still have an Internet connection, download  the Android SDK for Linux. Extract it and find the adb tool. On both the computer to tether and the host computer, make sure you have socat installed. (In Ubuntu, apt-get install socat.)

On the phone:

Start the Connect Bot app. Create & start a new SSH connection to your host computer. Add a port forward (hit Menu) with the options: Type=Local; Source port=9000; Destination=localhost:9000.
On the forwarding host computer:
You can do this either locally before you leave, or through the SSH connection you just made. Everything here must be run as root:
Forward connections from local port 9000 to a new virtual tunnel device tun0 and assign the computer the IP address 192.168.0.1 on this network:

sudo socat TUN:192.168.0.1/24,up

TCP-LISTEN:9000,bind=127.0.0.1

(TUN goes first so that the device is set up before socat blocks while listening for connections)

Set up the NAT internet connection sharing. If you use Firestarter, a nice firewall configuration tool, you can enable this easily in Edit -> Preferences, choose the local network device as tun0, and turn on enable internet connection sharing. This probably won’t stick across a reboot since the tun0 device won’t be available when Firestarter starts next time.
To set up NAT from a command line, see these instructions, but substitute tun0 for eth1:
echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o tun0 -m state –state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i tun0 -o eth0 -j ACCEPT
On the tethering computer:
It would be a good idea to turn off wireless networking and unplug any ethernet cables at this point. Then connect the computer to the phone with the USB cable.
Start the connection forwarding from the directory containing the adb tool from the Android SDK:
sudo ./adb forward tcp:9000 tcp:9000
sudo socat TUN:192.168.0.2/24,up TCP:localhost:9000
Put socat into the background with CTRL+Z and bg, or start a new terminal. Add routing information for the rest of the network:
sudo route add default tun0

Check that the connections are working:
ping 192.168.0.2 <— pinging itself, should be fast
ping 192.168.0.1 <— pinging host computer, should be slow

Add to /etc/resolv.conf the IP address of the DNS nameserver used by the forwarding host computer, i.e.:
nameserver 192.168.1.1
Or check out what you need by running:
ssh 192.168.0.1 -C “cat /etc/resolv.conf” | grep nameserver

And you should be good to go.

Printable Congressional District Maps: Behind The Scenes

Today I’m releasing print-quality maps of congressional districts, with street-level detail and county border lines. This has been one of the most sought-after resources based on emails I’ve received over the last some four years and I don’t think you can find this anywhere else. (At least not comprehensively for the whole nation. Local state clerk’s offices may have them. NationalAtlas.gov has maps but not with very much detail.)

This was a solid 2-day project with less than 300 lines of code and it’s something that only recently became this easy to do. I used Amazon Web Services (AWS), Census TIGER/Line cartographic data in an AWS pubic data snapshot, OpenStreetMap for the street detail in an AWS snapshot prepared by MapBox.com, Mapnik to render the maps (pre-installed on an AWS machine image prepared by MapBox), and the Python modules osgeo (for OGR) and PIL.

Here’s what  I did. This took a lot of trial and error, but in the end the steps were relatively simple.

Setting up the EC2 instance and the OpenStreetMap (OSM) planet data:

  • Start up a new Amazon EC2 Linux instance using the AWS machine image (AMI) prepared by MapBox linked above.
  • Create Amazon Elastic Block Storage (EBS) volumes for the two data sets (OpenStreetMap and Census TIGER/Line) in the same availability zone as the EC2 instance. If you do it in the AWS console, you’ll just need to search for the snapshots by ID or name (see the links above).
  • Attach the two volumes to the running EC2 instance as /dev/sdf (OSM) and /dev/sdg (TIGER).
  • Log into the EC2 instance with SSH.
  • Mount the two volumes: mkdir /mnt/osm; mount -t ext3 /dev/sdf /mnt/osm; mkdir /mnt/tiger; mount -t ext3 /dev/sdg /mnt/tiger
  • Following the MapBox instructions, attach the OSM data to Postgres, change the Postgres configuration to remove password protection, and restart Postgres.

To set up Mapnik, I followed the OpenStreetMap wiki which shows how to reuse their map styling. Most of the steps can be skipped because the data has already been set up in Postgres by MapBox. That involved:

  • Getting the OSM Mapnik files from their SVN repository.
  • Downloading some extraneous boundary information.
  • Create a new style definition that controls how map features are rendered based on the OSM defaults.
  • Editing the defaults a) so it actually works, and b) so it looks good at high DPI for printing (increasing font sizes, removing some icons). This took a lot of trial and error since I didn’t understand what was going on and regenerating a map takes some time.

The last step was to write a Python script that invokes Mapnik for each congressional district and generates a high-resolution map image.

  • The Census’s TIGER/Line cartographic data has a Shapefile-format file for each state containing the congressional districts in the state. The osgeo/OGR Python module can load the file and tell you the latitude/longitude bounds of the congressional district (among other things).
  • Then the Mapnik Python bindings are used to create a new map with the given size, loading in the OSM street data.
  • Additional layers are added from the TIGER/Line data for place names (CDPs and county subdivisions if you’re familiar with Census data), county names and borders, state borders (and shading of other states), and the boundaries of the congressional district itself and shading of other congressional districts.
  • After rendering the map, which takes ~30 seconds, I used the Python Imaging Library module to add header and footer text with a nice translucent effect.

Generating the maps at three resolutions for all of the congressional districts (except districts at-large) took several hours. I let it run overnight. They’re stored on Amazon S3 (the s3cmd tool is really useful for that).

There’s still room for a lot of improvement. After playing with the style instructions I got too much local road detail that in some places just ruins the whole map at low resolution. And in many places the county names aren’t showing up. Maybe because there’s too much detail. It’ll take some more trial and error to fix.

The source code (which includes all of the preparation steps in detail) is posted here.

Have we forgotten how to have an opinion and still be fair?

Maybe it was never true, but I have this sense that we’ve lost something in American public discourse over the last century. We’ve lost the conception of having an opinion and still being fair. It’s like we just can’t imagine both being true in the same brain. After watching the President’s speech tonight I realized that I feel seriously inhibited in what I say publicly because I want to maintain an impartial image so that people see GovTrack as an impartial source. Am I over concerned? I doubt it. This mistaken concept also underlies “professional journalism”, which is the style of most news operations now, and I think is perhaps the second greatest contributing factor to the downfall of news (after “The Internet”). More on that below.

People often mistake me as a liberal. And others mistake me as a conservative. Here’s a story about someone that did both. I’ve gotten some amusing feedback from people who mistook my GovTrack experiment in collaborative letter writing, for which I delievered an anti-gun-control letter to congressmen, as representing my own views. That couldn’t be further from the truth. If it were up to me, guns would be illegal. I explained this contradiction to someone who wrote me a letter. He said:

Julles: But don’t you see the similarities between what this administration is doing and what was done in Germany in the 30’s?

Then I replied:

Me: I really get personally offended sometimes. To compare a president who is trying to improve health care to a regime that killed however many millions is to belittle the damage and suffering done to anyone that experienced it. Disagree on policy all you want, but don’t belittle one of the world’s greatest tragedies.

And he replied:

Julles: HR3200 is a BAD bill . . . Open your eyes, kid.

(H.R. 3200 is the health care bill.) Expressions about eyes always strike a chord with me. But more to the point, I never told this guy I thought H.R. 3200 was a good bill. And, quite honestly, after the President’s speech tonight, I am not so enamored by where health care reform is going. In particular I wonder about the constitutional authority to require everyone to possess health insurance. I suspect it will be turned into a tax penalty to avoid a straightforward law and side-step constitutional questions.

I don’t have an agenda. But if I have an opinion, I may jeopardize the perception of fairness and accuracy in anything I do in the world of civics. Can I have an opinion and still be trusted to be fair when I put my nonpartisan hat on? I’m not even partisan. I vote Democratic, but so does most everyone else in the places I’ve ever lived. Am I allowed to say that? Have I lost credibility merely for being more open about my views?

And this is what I imagine journalists go through. They vote too, I hope. If they write for the New York Times, they probably live in New York and vote like most New Yorkers. But then they turn off their passion when they put their fingers down to the newsroom keyboard. And we suspend disbelief for a moment as we read their articles that journalists can’t have opinions and be fair at the same time. They make it easy for us to suspend disbelief because they write like they’re dead. No interest in the outcome. They’ve got to write a few words because they need to pay for the electricity that keeps their computers going, but if newspapers paid them to write a summary of the tax law they’d do that too. It doesn’t matter to them, at least as far as we can tell from reading.

This is ridiculous and, worse, counterproductive. I’d be more interested in news if articles pleaded with me that the issue was important, that it isn’t a conceptual exercise but that it even matters to the reporter. This is, apparently, how news used to be 100 to 250 years ago. It’s how the most compelling documentaries and long-form video news segments are today. Of course, it was also not very reliable 100-250 years ago. But I don’t think that dichotomy has to be so today. If we opened ourselves up to the idea that a reporter could have an opinion and still be fair, we wouldn’t need to suspend disbelief. Reporters wouldn’t have to die each time they start writing the next piece.

I don’t want reporters to die. Save the reporters. (Ironic hyperbole.)

Civic Hacking, the Semantic Web, and Visualization

Yesterday I held a session called Semantic Web II: Civic Hacking, the Semantic Web, and Visualization at Transparency Camp. In addition to posting my slides, here’s basically what I said during the talk (or, now on reflection, what I should have said):

Who I Am: I run the site GovTrack.us which collects information on the status of bills in the U.S. Congress. I don’t make use of the semantic web to run the site, but as an experiment I generate a large semantic web database out of the data I collect, and some additional related data that I find interesting.

Data Isolation: What the semantic web addresses is data isolation. For instance, the website MAPLight.org, which looks for correlations between campaign contributions to Members of Congress and how they voted on legislation, is essentially something that is too expensive to do for its own sake. Campaign data from the Federal Election Commission isn’t tied to roll call vote data from the House and Senate. It’s only because separate projects have, for independent reasons, massaged the existing data and made it more easily mashable that MAPLight is possible (that’s my site GovTrack and the site opensecrets.org). The semantic web wants to make this process cheaper by addressing mashability at the core. This is important for civic (i.e. political/government) data: machines help us sort, search, and transform information so we can learn something, which is good for civic education, journalism (government oversight), and research (health and economy). And it’s important for the data to be mashable by the public because uses of the data go beyond the resources, mission, and mandate of government agencies.

Beyond Metadata: We can think of the semantic web as going beyond metadata if we think of metadata as tabular, isolated data sets. The semantic web helps us encode non-tabular, non-hierarchical data. It lets us make a web of knowledge about the real world, connecting entities like bills in congress with members of congress, what districts they represent, etc. We establish relations like sponsorship, represents, voted.

Why I care: Machine processing of knowledge combined with machine processing of language is going to radically and fundamentally transform the way we learn, communicate, and live. But this is far off still. (This explains why I study linguistics…)

Then there are some slides on URIs and RDF.

My Cloud: When the data gets too big, it’s hard to remember the exact relations between the entities represented in the data set, so I start to think of my semantic web data as several clouds. One cloud is the data I generate from GovTrack, which is 13 million triples about legislation and politicians. Another cloud is data I generate about campaign contributions: 18 million triples. A third data set is census data: 1 billion triples. I’ve related the clouds together so we can take interesting slices through it and ask questions: how did politicians vote on bills, what are the census statistics of the districts represented by congressmen, are votes correlated with campaign contributions aggregted by zipcode, are campaign contributions by zipcode correlated with census statistics for the zipcode (ZCTA), etc. Once the semantic web framework is in place, the marginal cost of asking a new question is much lower. We don’t need to go through the work that MAPLight did each time we want a new correlation.

Linked Open Data (LOD): I showed my part of the greater LOD cloud/community.

Implementation: A website ties itself to the LOD or semantic web world by including <link/> elements to RDF URIs for the primary topic of a page. This URI can be plugged into a web browser to retrieve RDF about that resource: it’s self-describing. I showed excerpts from a URI for a bill in congress that I created. It has basic metadata, but goes beyond metadata. The pages are auto-generated from a SPARQL DESCRIBE query as I explained in my Census case study on my site rdfabout.com.

SPARQL: The query language, the SQL, for the semantic web. It is similar to SQL in metaphors and keywords like SELECT, FROM, and WHERE. It differs in every other way. Interestingly, there is a cultural difference: SPARQL servers (“endpoints”) are often made publicly acessible directly, whereas SQL servers are usually private. This might be because SPARQL is read-only.

Example 1: Did a state’s median income predict the votes of Senators on H.R. 1424, the October 2008 stimulus bill? I show the partial RDF graph related to this question and how the graph relates to the SPARQL query. First it is an example SPARQL query. Then the real one. The real one is complicated not because RDF or SPARQL are complicated, but because the data model *I* chose to represent the information is complicated. That is, my data set is very detailed and precise, and it takes a precise query to access it properly. I showed how this data might be plugged into Many Eyes to visualize it.

My visualization dream: Visualization tools like Swivel (ehm: I had real problems getting it to work), Many Eyes, Ggobi, and mapping tools should go from SPARQL query to visualization in one step.

Example 2: Show me the campaign contributions to Rep. Steve Israel (NY-2) by zipcode on a map. I showed the actual SPARQL query I issue on my SPARQL server and a map that I want to generate. In fact, I made a prototype of a form where I can submit any arbitrary SPARQL query and it creates an interactive map showing the information.

Other notes: My SPARQL server uses my own .NET/C# RDF library. That creates a “triple store”, the equivalent of a RDBMS for the semantic web. Underlyingly, though, it stores the triples in a MySQL database with a table whose columns are “subject, predicate, object”, i.e. a table of triples. See also: D2R server for getting existing data online.