Lunch@Berkman: Open Government Data for Open Accountability

Liveblogging Felipe Heusser’s presentation on Open Government Data for Open Accountability at the Berkman Center. Please excuse misrepresentation, misinterpretation, typos, and general stupidity (all of which are mine and mine alone).

Felipe’s at the Berkman Center today to talk about transparency, open data, technology, and accountability. His talk is based on three key points:

1. Transparency is a cliche, and FOIA is outdated.

Transparency is “a very sticky word,” Felipe tells us— “a pop concept that everyone likes”—and freedom of information laws have historically been one of the most popular ways to spread transparency policies.

He shows us a map of the world, shaded according to the existence/strength of freedom of information laws. He points out that this map is not highly correlated with levels of corruption, suggesting that greater transparency doesn’t necessary lead to more accountability. Regardless, transparency policy—a term Felipe is using today to mean freedom of information regulation—has spread rapidly.

Freedom of information has been the cornerstone of transparency policy since at least the late 18th century, when a Finnish priest who also served in in the Swedish-Finnish parliament developed the first freedom of the press act. Modern freedom of information laws allow you to request information from the government and also require governments to proactively disclose certain types of information.

The mindset behind FOIA, Felipe says, is not about access to data, but about access to documentation. In the age of the web, this way of thinking is becoming increasingly obsolete, he argues. Rather than the reports offered by FOIA, data records offer us neutral facts, which are more versatile. FOIA offers two-way communication, rather than an open flow of shared information. FOIA holds up a barrier between citizens and government: channels of bureaucracy in which requests can be mired for ages—up to 20 years for certain US government agencies.

2. Open data policies are necessary.

None of the above issues jibe with the way the Internet works, Felipe points out. We need open data to keep freedom of information up to date. The good news: some governments and organizations have made small steps toward open data. is one, along with, Open Kenya, World Bank data, and others.

Why does this matter? Open government data can help keep our freedom of information right up to date by allowing us to access much of the information the government holds. Open data also allows for multiway communication and sharing—it “understands the logic of information abundance.” Open data doesn’t require as many gatekeepers: you go online, search, and download, rather than demanding a formal request.

What does this mean? Open data can promote a more open accountability. Historically, government accountability has been exercised formally through clear procedures. With open data, accountability can be more informal and crowd-sourced. Rather than relying on scarce institutional watchdogs, accountability can now draw on an abundance of web-based watchdogs.

Ciudadano Inteligente's "Interest Inspector" uncovers conflicts of interest in Chile's congress.
Felipe gives an overview of one of his projects, Ciudadano Inteligente. The organization’s Interest Inspector uses open data to compare the personal interests and financial ties of Chilean congressmembers to the official policies they support to uncover conflicts of interest in what Felipe calls an “ongoing accountability exercise.”

3. Stay cautious.

There’s a lot of “talking, cocktails, and pictures” about transparency and open data, Felipe says, but even more importantly, there is still a lot of work to be done. Most of the open data experiments today are coming from the local level—cities releasing data, for example—instead of growing out of a truly national, state-level commitment to transparency. Also, most of the data available online today is not all that useful: it’s images and transportation data, with hardly anything about, for example, banking and finance.

Still, the implementation of open data policies matters, Felipe says, and it matters for accountability (and potentially for business and the delivery of public services as well). Promoting the use of these policies is crucial for securing access to information.


Doc Searls asks about personal data: MyData in the UK and Google Takeout. How does this relate?

Jennifer Shkabatur challenges both Felipe’s assertion that open data is “neutral”—data sets are still structured by someone, she points out—and the idea that open data is free of gatekeepers. As Felipe noted, the datasets that do exist openly are heavily weighted toward simpler, more fluffy data instead of making information about finance, education, and health more available. She also questions whether open data forces people to rely on NGOs and other organizations that can interpret and make use of this data—how does this lead to real accountability?

Felipe responds that documents are different from data sets in that they are more structured/created/manipulated than the data itself. Neutral facts are lists of data—the number of policemen on the streets, etc. A civil servant still creates columns and files, but the numbers are neutral. Under today’s FOI laws, governments are not compelled to release this information—they only have to release documents about this information, so if the number you want is not in a report, you’re out of luck.

With respect to gatekeepers, Felipe agrees that they still exist but argues that they are fewer under open data policies than under FOI laws. Open data allows you to, for example, cross-reference data from your own government with World Bank data, which may lead to new discoveries.

Jennifer jumps back in with a question about supply and demand: journalists use FOIA to find specific information. Relying only on data sets and hoping you can cross-reference and find informations shifts the situation from one that is demand-driven to one that is supply-driven.

Felipe responds that, even under the current FOI laws, the situation is still supply-driven. We haven’t seen yet whether there’s a change in supply and demand.

Sascha Meinrath points out that certain data sets are released in response to citizen demand, but he’s skeptical that the government will voluntarily release “data that really matters.”

Felipe notes that open data policies aren’t intended to replace FOIA, but rather to complement it.

Yochai Benkler notes that FOIA can still be helpful—we care less about transportation data and more about say, whether the Department of Homeland Security is monitoring our tweets, which is information we may be able to get from contracts and other documents available through FOIA.

Yochai also points out that there’s a lot of excitement and political/emotional energy that gets poured into open data, which can make those seem like “good actors” who are actually not all that good: the holding up of the US as a model of transparency by a mobilized global community is dangerous, he warns.* If you think the core of freedom of information is forcing the government to release data it doesn’t want to release, then FOIA is more important, and data is supplementary. If you believe the core is that once the data is out there, you can’t hide, then data is the key.

Felipe notes that was not the best example, just one of the first, and points to tools like Accesso Inteligente, which helps citizens file FOIA requests and tracks all questions and answers in a searchable database.

*Someone from the audience pushes back against this, pointing out that has many detractors.

Isaac Meister asks about “two-way sharing,” which seems to him to be a departure from, rather than a description of, the FOIA model. He asks what data private citizens should be encouraged to share with the government, what rights they may gain or lose by doing so, and what obligations the government may have to share that data.

An audience member (whose name I didn’t catch, sorry!) asks how well open data is being used by average citizens. Does the general public rely on others to interpret the data for them? Do they care that it exists? Who is using the data interpretation tools (like the Interest Inspector) that currently exist?

Felipe responds that yes, average citizens do still rely on intermediaries, usually the press. Ciudadano Inteligente has seen an increase in users of its applications (both general users and the press itself).

FOIA…and Rick Astley

As I’ve talked to technology for transparency activists over the past year, one thing that’s repeatedly come up is the importance of traditional transparency tools — partnering with the mainstream media to get the word out tends to be more effective than going it alone with phones and websites, for example. One key aspect of this “old school” activism is the use of freedom of or right to information laws (FOIA/RTI) in different countries — officially requisitioning information from the government. This doesn’t always work, of course; bureaucratic processes often draw out the process interminably, and some governments simply don’t have the capacity to keep track of the kinds of data transparency activists would like.

Today, I stumbled across of the best uses of public record requests I’ve seen since I’ve started paying attention to this field. Thank you, Oregon House of Representatives, for this gem, created with the help of a public records request for hours of film of the February 2010 legislative session:

More information on how this beauty came to be, and the role that public records requests played, here: How one Oregon lawmaker convinced his colleagues to ‘Rick Roll’ the state legislature

Thoughts from the Al Jazeera Forum

I’m spending this weekend at the Al Jazeera Forum in Doha, where I was asked to participate in a seminar on the role of the blogosphere in bringing social and political change. The question posed to us — Hisham Almiraat, Mohamed Najem, Georgia Popplewell, Ramsey Tesdell, Nasser Weddady, and myself — revolved around the article Malcom Gladwell published in the New Yorker last month claiming that online connections are purely weak ties and that social media are therefore incapable of creating or sustaining a social movement.

I side with those — David Weinberger, Zeynep Tunecki, and others — who have responded to Gladwell by saying that attempting to cleanly separate online and offline media (or connections, or actions) is a futile pursuit. Social media are an integral part of a new media ecosystem, one in which bloggers are amplified and inspired by more traditional forms of media and in which mainstream media outlets increasingly partner with citizen journalists to provide the most complete coverage they can.

Throughout the forum, panelists have repeatedly raised this theme, arguing that framing media as old versus new or mainstream versus citizen is not a productive approach to understanding the impact of online networks. Speakers who work with major news outlets have thanked bloggers for sharing their first-hand accounts of what’s happening on the ground, while bloggers have thanked major news outlets for broadcasting their reports. In a break-out discussion on Saturday, citizen journalists from Tunisia and Egypt said that Al Jazeera’s amplification of their voices helped bring the situations in these countries to a tipping point.

This idea of a media ecology is one I’ve been mulling over as I’ve been working with the Technology for Transparency Network over the past year. We’ve interviewed more than 60 organizations who use digital tools to promote transparency and accountability in their governments, and one of our key conclusions has been that organizations who build partnerships with traditional media organizations — whether these are local radio stations or major national newspapers — tend to have more success than organizations that rely solely on the Internet to get their message across. For me, it’s been interesting to hear that this is the case not only when talking about city budget transparency, but also when talking about revolutions.

I don’t think this in any way diminishes the importance of social media in the movement for government accountability. In many cases, both local radio stations and national papers (and, as I’ve heard this weekend, Al Jazeera) would be unable to report on the stories they cover without the help of bloggers and tweeters. I’m glad to see that those who are here at the forum — both traditional and citizen journalists — are willing to exlore these partnerships and acknowledge their importance.

Tools for Transparency: Google Refine

Originally posted as a guest post on the Sunlight Foundation blog.

For the past six months, I’ve served as the co-director of the Technology for Transparency Network, an organization that documents the use of online and mobile technology to promote transparency and accountability around the world. One of the most common challenges the project leaders we’ve interviewed face is making sense of large amounts of data.

In countries where governments keep detailed digital records of lobbying data and education expenditures, data wrangling is a time-consuming, labor-intensive task. In countries where these records are poorly maintained, this task becomes even harder — everything from inconsistent data entry practices to simple typos can derail data analysis.

Google Refine (formerly Freebase Gridworks) is a free, open-source tool for cleaning up, combining, and connecting messy data sets. Rather than acting like a traditional spreadsheet program, Google Refine exists “for applying transformations over many existing cells in bulk, for the purpose of cleaning up the data, extending it with more data from other sources, and getting it to some form that other tools can consume.”

At its most basic level, Google Refine helps users quickly summarize, filter and edit data sets by allowing them to view patterns and to spot and correct errors quickly. More advanced features include reconciling data sets (i.e., matching text in the set with existing database IDs) with data repository Freebase, geocoding, and fetching additional information from the Web based on existing data.

Though it runs through an Internet browser, Google Refine operates offline, making it attractive for those with limited bandwidth or privacy concerns — a group that includes many of the projects listed on the Technology for Transparency Network.

Google Refine isn’t going to solve the problem of poor data availability, but for those who manage to gain access to existing records, it can be a powerful tool for transparency.

For more information, check out the links and video below:

Tech for Transparency: New Interviews Posted

Avid readers of my blog (here’s looking at you, Rev) may remember that several months ago I announced that research was beginning for the second phase of the Technology for Transparency Network. The first phase consisted of interviews with over 30 projects around the world who are using technology to promote transparency and accountability in the government and/or private sector. Our goal in the second phase was twofold: to double the number of case studies on the site and to expand the geographic regions we covered.

Since then, I’ve been largely silent about the project — we’ve been working so hard to complete and edit the interviews that I haven’t had much time to breathe. But today I’m thrilled to announce that we have eight new case studies online, with lots more to come over the next few weeks. The case studies that have been posted so far are:

Accountability Initiative
Accountability Initiative researches and creates innovative tools to promote transparency and accountability in India’s public services.

Amatora mu Mahoro
Amatora mu Mahoro (“Peaceful Elections”) is an Ushahidi-based project created to monitor Burundi’s 2010 elections.

Association for Democratic Reforms
ADR India works to monitor national elections through country-wide SMS and helpline campaigns and an informational website. seeks to empower citizens by helping them collectively send petitions and inquiries to government bodies.

Excelências fights corruption in the Brazilian government by publishing data about politicians and government activities online.

Golos (Voice) has introduced several online tools for better election monitoring in Russia.

Mam Prawo Wiedzieć
Mam Prawo Wiedzieć helps Polish citizens access information about their elected representatives in an easy, user-friendly way.

Pera Natin ‘to!
Pera Natin ‘to! (It’s Our Money!) encourages Filipino citizens to report times when they are asked for bribes.

In addition to continuing to post new case studies (you can subscribe to our case study feed via RSS), we’ll also be publishing our final report on both phases of the project by the end of the month. In the meantime, check out @techtransparent and our Facebook page for daily updates and our podcast for interviews with the project leaders!