Adventures in Open Data
Local news startup Civil Beat today posted a piece on the names and salaries of state House and Senate members and staff. As before, they posted a watermarked PDF on Slideshare with the download feature disabled, and invited people to become "full members" to use their searchable database.
But if you'd like to download and play with the data for free, it's been up for over a week at the Hawaii Open Data Project. Specifically, you can access the following documents:
2009-10 Senator and Senate Staff Salary
2009-2010 Representatives Salary (Legislator Monthly Salary)
2010 Permanent-OM-Support Staff Payroll
As you can see, the Senate provided the information in a single document, while the House separated Representatives from staff. The House staff data was provided as an Excel file, which Google Docs easily converted. I also requested job descriptions and duties, and both offices provided or referenced administrative manuals and rules that contained them. You can find everything on the wiki page.
Overdue Update
Earlier this month, I reported that my request for information was denied by the Department of Human Resources Development. (If you haven't been following the comments on that post, I encourage you to go back and read them.) I was trying to get the same data that Civil Beat had acquired and locked down, in the hopes of making it freely available.
At the time, I was discouraged. It seemed that DHRD had closed the door that it claims Civil Beat had erroneously walked through. So I feared that the other agencies I contacted -- the University of Hawaii and the state Legislature -- would similarly deny my requests.
I was pleasantly surprised to encounter a great deal of cooperation and responsiveness instead.
On the DHRD front, I filed an appeal with the state Office of Information Practices, which promptly acknowledged my inquiry. I'm now awaiting DHRD's written response to the request from the OIP (due later this week).
Meanwhile, the UH Office of Human Resources, the office of Sen. Colleen Hanabusa, and the office of Rep. Calvin Say, all fulfilled my first and only information request -- on time, completely (and electronically), and with no resistance nor fanfare.
So as in the case of the Legislature data above, if you want to see or search through a list of all UH employees and their salaries, you could certainly sign up for Civil Beat (which also offers its thoughtful analysis of the data). Or you could just download it for free, and draw your own conclusions:
UH Employees and Salaries
Whither Civil Beat?
The last few weeks have been interesting and educational, to say the least. I fully support Civil Beat's efforts to develop a sustainable business model, and to realize the dream of making money by practicing good journalism. But I happen to believe that a media outlet should be selling its reporting, analysis and commentary, and not the public information that it's based on.
I concede that it may take considerable time, effort, and money to get the information, but I feel that the information ultimately belongs to everyone, and should be made freely available. In fact, doing so helps keeps journalists credible, as anyone can analyze the data themselves to confirm or refute the journalists' conclusions. I think this is a big part of why most media outlets provide source documents and data alongside their coverage.
But, yes, the media landscape is changing, and I may simply be out of date or out of touch in my ideals and expectations.
Ragnar Carlson, editor of the Honolulu Weekly, posted a comment on Friday that noted that his newspaper was the first to get a copy of the Environmental Impact Statement for the city's multi-billion-dollar rail project. It was a government document, obtained via a public information request, but the Honolulu Weekly did not share it, even when asked by other media outlets. (The final rail EIS is now available online, courtesy the city's HonoluluTransit.org site.)
Carlson writes:
Did I feel great about it? I felt great that we got it first, and less great about holding on to our copy. But what are you going to do? Newspapers are going out of business every day. At some point, you have to be willing to say “the facts in this document are public information, the presence of it in the community is our work, that’s our business and so we are holding on to it.” If journalists continue to give away their work product, newsgathering organizations will go away. And everyone will be worse off for it.
Like many bloggers, I have the luxury of reporting on any topic I like, without having to earn a living doing so. Practicing journalism is a lot harder and more complicated for those that do it professionally. I understand why the Honolulu Weekly and Civil Beat did what they did. I simply disagree with them.
The Project
So what is the Hawaii Open Data Project? Despite the formal-sounding, un-creative name, it's basically a wiki page I started to keep track of these public information requests. Of course, I'm a newcomer and an amateur to this sort of thing, so I also added links to public information sources (courtesy Ian Lind), and to other public information quests (like Doug White's). I'm sure Larry Geller over at Disappeared News has unearthed some gems as well.
Since it's a wiki, I'm hoping other local open data advocates feel inspired to add to the collection.
So far, I've largely been following and duplicating Civil Beat's requests (and if you know of any outstanding or new requests, please let me know). Going forward, my "data liberation ninja" friend and I are brainstorming other data sets to pursue and publish. Like @kerryrm, he's cited the "8 Principles of Open Government Data," which will probably serve as the basis for future work.
He writes:
I am partial to more statistical data myself -- public health, crime, education, traffic, tourism, etc. Stuff with numbers that can be mashed up well or used to debunk false perceptions of every state thinking they rank last in some report. My goal is to normalize the data, present it in a searchable form, do some light analysis and hopefully provide a simpler way to mash up the data. My presentation skills are horrible, but hopefully, I can build a framework for someone that can.
The dream would be to have something like the set of Databases, Tables & Calculators put online by the federal Bureau of Labor Statistics. Imagine a site at data.hawaii.gov with downloadable data, online tools, and a full API to allow anyone to access and analyze government information.
Until that great day, we'll build what we can on our own. If you've got ideas, or if you'd like to contribute, please comment below... or just head over to the wiki and start something.
