Yesterday at a small meeting the Senate announced that it would be making its legislative data available to the public. This has been a long time coming.
The what & why
No legislative branch agency makes available a spreadsheet that lists every bill introduced in Congress. This issue is that simple. We’re finally going to get a list of bills in a useful data format, and, hopefully, a lot more information on top of it, some time next year.
I first asked the Library of Congress for access to its database of legislation in 2001 when I began building GovTrack. They said no, under orders from the House and Senate, and so I began “screen scraping”, or reverse engineering, their public THOMAS.gov website for the same information and making that data freely available to others. The data is what you need to create large-scale visualization, analysis, and tools, such as the ideology and leadership scores, bill prognosis, email updates, legislator report cards, bill text paragraph permalinks, maps of congressional districts, advanced search, and much much more that I built on GovTrack.
And my data on GovTrack, rather than anything Congress produces, quickly became the authoritative source for legislative information. Endless apps have been build on top of the data I made available. Even Congress comes to me for data. Representatives embed the maps on GovTrack on their websites and ask me, from time to time, for their own voting statistics. The House Democrats use GovTrack’s data to keep their caucus informed, and many Senate offices load GovTrack data into their back-office systems.
The data is now collected in a community project on GitHub (which began in 2012 and was spearheaded by Eric Mill at the Sunlight Foundation, Derek Willis, and myself), but the right place for this data is Congress. I never wanted to be the linchpin of congressional information (except in so far as it provided me with a career, so… thank you Congress). Once the Senate begins actually making its data available, planned for next year some time, I hope to see Congress become the authoritative source for its own information.
The history
Advocacy around legislative data began in 2007. At the request of Speaker Nancy Pelosi, who was looking for ways to reform the House, a group of government transparency advocates issued the The Open House Project report, co-written by myself and others and spearheaded by the new Sunlight Foundation. The report called for the House to make available the legislative data I had been asking for, among several other transparency recommendations. It was just seven years ago that “data” was something totally new to Congress. I surveyed the state of legislative data in 2008 — there was not much. At this time the Senate had not yet even started publishing its voting records in data (as XML). Following the report, many of us worked with Senate staff to explain why making vote data available to the public was a good thing, and only in 2009 did they start making that available (see also 2007). In 2009 we also secured favorable language in the FY 2009 omnibus appropriations bill (see also 2008), but Congress’s support agencies largely ignored the directive to make data available.
John Wonderlich at the Sunlight Foundation, who had started The Open House Project, kept the advocacy going over the next several years. But the House, under Pelosi, was not very responsive to requests for more transparency during this time. Some headway was made, but not in legislative data.
The Republican take-over of the House in 2011 marked a major shift toward transparency. They began making much more data available and promised data about bills. When one representative strangely tried to put the kibosh on data in 2012, The Washington Post ran a story about it (and about me, which was flattering), which lit the fire under House leadership and lead to the formation of the House Bulk Data Task Force. Advocates formed a new Congressional Data Coalition in 2014, spearheaded by Daniel Schuman at CREW, and we secured favorable language in the FY2015 legislative branch appropriations bill to keep the pressure on. The House task force during this time made some progress, but without the cooperation from the Senate it wasn’t able to actually do much.
That’s what changed yesterday: the Senate is on board. This closes out what has been, for me, a 13-year campaign.
Daniel wrote more about the news here.
Joshua Tauberer's Archived Blog 