Open Government Data and .NET

  • [アーティクル]

As mentioned in my previous post, I wanted to test the waters by writing a small app that allowed someone to enter a zip code, find all of their Congressional representatives , then find out the committees they're on, bills they've sponsored, how they vote based on some issues… and then provide me the tools to contact them, as well as enter a daily rating on how I think they're doing.

Despite being snowed in, it was a pretty busy day – so I only had an hour or so of free time to work on it.  Thus, tonight’s goal was just to find the representatives by zip, show their info and a picture, and then list their committees and bill sponsorships.  I’ll work on the rest tomorrow or through the weekend.

My technology choices:

  • .NET and Visual Studio
  • Windows Forms for the UI (simply because it’s fastest for me right now… I’ll switch to WPF or Silverlight once I’m through prototyping)
  • WCF REST Starter Kit – because I really dislike parsing XML.  I just started using it a day or two ago, and one of the most fantastic thing it provides is a tool that automatically creates a .NET class from an XML file… so all all you need to do is get a sample of the REST response – paste it into the class et voila!  It creates a class for you – which you can then use to access the response data.  You can get it here: https://bit.ly/JYxf
  • APIs: I’m going to use both the SunlightLabs API and the GovTrack.us API – they’re actually redundant, since the Sunlight APIs are really just a wrapper around GovTrack.us – but I wanted to mix it up a bit.  Both have REST interfaces – and the specific APIs I’m using are:

My first step was to get an API key from SunlightLabs (which I did a few days ago in anticipation of this blog) – which you can do here: https://services.sunlightlabs.com/api/register – very easy and quick.

Next, I tested the APIs via my browser to 1) make sure they work… and 2) to get the response XML for my project.  I used these:

The REST responses were valid XML – so I knew there’s be no problems with accessing the data.

So – I launched Visual Studio, started a new Winforms project (in VB.NET), and added the references to the WCF REST Starter Kit assemblies (you’ll want to install the starter kit before launching Visual Studio so you have access to the “paste as type” add-in).

From there, I added two empty classes – one to wrap the Sunlight APIs and data structure (SunlightLegislators), and one to wrap the GovTrack APIs and structure (GovTrackPerson).

Now’s the tricky part (not really)… I want to take the REST responses from the APIs and “magically” build a class structure from them so I can access the data through strongly-typed class interfaces (rather than parsing the XML… I mentioned above that a really don’t like parsing XML… it bears repeating).  Here’s what I had to do:

  • Invoke one of the APIs from the browser and select/copy all of the XML text in the response
  • Open up the empty class that I want to use, place the cursor in the class, and then select “Edit | Paste XML as Types” (picture below)
  • BAM!  You’re done.  Couldn’t be easier.

PasteXMLasTypes

Now I just need to invoke the REST API from code in my project, take the response data and have it deserialize the XML into the my classes (for those who might not be developers – deserialize means to take the xml and create a class out of it… we devs, similar to paleontologists, like to use big words to convince people we’re smart).

Here’s the code I wrote to call the SunlightAPI and have it return a collection of legislator objects (sans error handling):

Function GetLegislators(ByVal ZipCode As String) As SunlightLegislators.responseLegislators

    Dim serviceURI As String = _ "https://services.sunlightlabs.com/api/legislators.allForZip.xml?zip=" _ & ZipCode & "&apikey=" & SunlightApiKey Dim c As New Microsoft.Http.HttpClient 'Creates a new WCF HTTP Client for calling REST services Dim httpmsg As New Microsoft.Http.HttpRequestMessage() 'Create message to send to the service httpmsg.Uri = New Uri(serviceURI) 'Point it to the REST API endpoint Dim resp As Microsoft.Http.HttpResponseMessage = c.Send(httpmsg) 'Get the response Dim str As String = resp.Content.ReadAsString 'Put it in a string 'Create the deserializer and deserialize response XML into the SunlightLegislators.Response class Dim mySerializer As New _ System.Xml.Serialization.XmlSerializer(New SunlightLegislators.response().GetType()) Dim r As SunlightLegislators.response = mySerializer.Deserialize(New System.IO.StringReader(str)) Return r.legislators

End Function

 

Pretty darn easy (and probably could be easier – my two days of experience with the WCF REST stuff probably isn’t doing it justice).

The GovTrack API was a little more challenging, as the deserializer kept choking on some of the comments in the XML – so I wrote some quick code to pull them out.  Everything else is essentially the same:

 

Public Function LookupLegislatorData(ByVal govTrackID As Integer) As GovTrackLegislator.person Dim serviceURI As String = "https://www.govtrack.us/congress/person_api.xpd?id=" _ & govTrackID.ToString & "&session=111" Dim c As New Microsoft.Http.HttpClient Dim httpmsg As New Microsoft.Http.HttpRequestMessage() httpmsg.Uri = New Uri(serviceURI) Dim resp As Microsoft.Http.HttpResponseMessage = c.Send(httpmsg) Dim str As String = resp.Content.ReadAsString Dim xs As New System.Xml.Serialization.XmlSerializer(New GovTrackLegislator.person().GetType()) Dim odoc As New System.Xml.XmlDocument odoc.LoadXml(str)   'Loading into an XML Document to make removing the comments easier RemoveComments(odoc) 'The deserializer kept choking on some of the comments in the response Dim LegPerson As GovTrackLegislator.person = _ xs.Deserialize(New System.IO.StringReader(odoc.InnerXml)) Return LegPerson End Function

Sub RemoveComments(ByRef xmlDoc As Xml.XmlDocument) RemoveCommentsFromNode(xmlDoc) End Sub

Sub RemoveCommentsFromNode(ByRef XNode As Xml.XmlNode) For Each cn As Xml.XmlNode In XNode.ChildNodes If cn.HasChildNodes Then RemoveCommentsFromNode(cn) 'Crawl the whole doc recursively Dim nl As Xml.XmlNodeList = cn.SelectNodes("comment()") For Each n As Xml.XmlNode In nl cn.RemoveChild(n) Next End If Next End Sub

Not too bad… under 30 lines of code to call two REST services and put the results into .NET classes that I can then easily use from code… no XML parsing required.

The only other challenge was getting the picture… I couldn’t find an API for that, but GovTrack.us does have a library of pictures in their data directory… each photo is identified by the URL to the directory + the GovTrackID of the representative + the image size + a jpg extension.  For example: https://www.govtrack.us/data/photos/300002-100px.jpeg - So it was just a matter of assembling a string and then setting the ImageLocation property of a PictureBox control on my form.

From there – it was just a matter of putting together a quick Windows form. I used a Split Panel to separate the listview of legislators from the detail data that comes up when you click on someone – and then a tabcontrol to display the lists of committees and bills.  Only additional work was setting up an application setting to hold the API key so it’s not hardcoded as well as some code to clear it (for testing purposes).

Here’s my UI so far:

DotNetLegisTrack

So far, actual technical work on this project has taken me about an hour… and it’s less than 150 lines of code… it was almost ridiculously simple to do.  A lot of this is due to .NET, Visual Studio, and the WCF REST tools… but NONE of it would even have been remotely possible without the DATA and APIs that are made available on GovTrack.us and by SunlightLabs.

Getting back to my last post… the 5 elements to a successful Open Gov solution:

  • Data – provided by Govtrack.us
  • APIs – provided by Govtrack.us and SunlightLabs
  • Builder & tools – me, Visual Studio and .NET
  • Domain expert – me… fortunately, the context of the data is common knowledge
  • Incentive – I’m hoping that someone pays me by the word on my blogs ;)…  but in this case, just demonstrating how Microsoft technologies can be used in Open Government solutions is good enough for me

I’ve attached the code to this post – feel free to take a look… but keep in mind that this is the result of 1 hour of work… so error handling, formatting, input validation, etc. is all very limited.  If it shapes up, I’ll get it up onto codeplex as a formal project. 

In the meantime – please send me your comments and questions.

-Dan

ps. for my next trick: incorporating data from Windows Azure Dallas and Bing… stay tuned

DotNetLegisTrack.zip

Comments

  • Anonymous
    February 28, 2010
    Dan, thank you so much for this starter code.  This will really help me kick-start my project idea.

  • Anonymous
    June 27, 2010
    The rest of us would like to ask you one hour guys to leave the premises please :-)

  • Anonymous
    June 27, 2010
    Not sure what you mean, Clinton... can you elaborate?

  • Anonymous
    July 08, 2010
    Dan, this is exactly what I need.  I'm not a techie but know a few things.  How do I go about implementing this on an existing website?  Any help...steps...would be greatly appreciated.