Tools for Transparency: Google Refine

Originally posted as a guest post on the Sunlight Foundation blog.

For the past six months, I’ve served as the co-director of the Technology for Transparency Network, an organization that documents the use of online and mobile technology to promote transparency and accountability around the world. One of the most common challenges the project leaders we’ve interviewed face is making sense of large amounts of data.

In countries where governments keep detailed digital records of lobbying data and education expenditures, data wrangling is a time-consuming, labor-intensive task. In countries where these records are poorly maintained, this task becomes even harder — everything from inconsistent data entry practices to simple typos can derail data analysis.

Google Refine (formerly Freebase Gridworks) is a free, open-source tool for cleaning up, combining, and connecting messy data sets. Rather than acting like a traditional spreadsheet program, Google Refine exists “for applying transformations over many existing cells in bulk, for the purpose of cleaning up the data, extending it with more data from other sources, and getting it to some form that other tools can consume.”

At its most basic level, Google Refine helps users quickly summarize, filter and edit data sets by allowing them to view patterns and to spot and correct errors quickly. More advanced features include reconciling data sets (i.e., matching text in the set with existing database IDs) with data repository Freebase, geocoding, and fetching additional information from the Web based on existing data.

Though it runs through an Internet browser, Google Refine operates offline, making it attractive for those with limited bandwidth or privacy concerns — a group that includes many of the projects listed on the Technology for Transparency Network.

Google Refine isn’t going to solve the problem of poor data availability, but for those who manage to gain access to existing records, it can be a powerful tool for transparency.

For more information, check out the links and video below:

Tech for Transparency: New Interviews Posted

Avid readers of my blog (here’s looking at you, Rev) may remember that several months ago I announced that research was beginning for the second phase of the Technology for Transparency Network. The first phase consisted of interviews with over 30 projects around the world who are using technology to promote transparency and accountability in the government and/or private sector. Our goal in the second phase was twofold: to double the number of case studies on the site and to expand the geographic regions we covered.

Since then, I’ve been largely silent about the project — we’ve been working so hard to complete and edit the interviews that I haven’t had much time to breathe. But today I’m thrilled to announce that we have eight new case studies online, with lots more to come over the next few weeks. The case studies that have been posted so far are:

Accountability Initiative
Accountability Initiative researches and creates innovative tools to promote transparency and accountability in India’s public services.

Amatora mu Mahoro
Amatora mu Mahoro (“Peaceful Elections”) is an Ushahidi-based project created to monitor Burundi’s 2010 elections.

Association for Democratic Reforms
ADR India works to monitor national elections through country-wide SMS and helpline campaigns and an informational website. seeks to empower citizens by helping them collectively send petitions and inquiries to government bodies.

Excelências fights corruption in the Brazilian government by publishing data about politicians and government activities online.

Golos (Voice) has introduced several online tools for better election monitoring in Russia.

Mam Prawo Wiedzieć
Mam Prawo Wiedzieć helps Polish citizens access information about their elected representatives in an easy, user-friendly way.

Pera Natin ‘to!
Pera Natin ‘to! (It’s Our Money!) encourages Filipino citizens to report times when they are asked for bribes.

In addition to continuing to post new case studies (you can subscribe to our case study feed via RSS), we’ll also be publishing our final report on both phases of the project by the end of the month. In the meantime, check out @techtransparent and our Facebook page for daily updates and our podcast for interviews with the project leaders!

Tracking Kenya’s Development Budget

I woke up early last Monday morning to interview Philip Thigo of the Kenyan Budget Tracking Tool for the Technology for Transparency Network. I don’t think I’ve ever had so much fun asking someone questions.

I woke up early last Monday morning to interview Philip Thigo of the Kenyan Budget Tracking Tool for the Technology for Transparency Network. I don’t think I’ve ever had so much fun asking someone questions.

If you listen to the podcast, you’ll hear faint sounds of Nairobi in the background: horns honking, people walking around. As Philip chatted candidly with me about the successes and struggles of encouraging greater transparency in Kenya’s national budget, I imagined him in his office, the door propped open, curtains blowing in the breeze.

Can you tell that living in New York has made me a bit desperate for sunshine and perhaps a return trip to Kampala?

Anyway, the noise makes for an interview that sounds less than studio-produced, but it also makes me happy. The sounds of life in east Africa, Philip’s laughter and his enthusiasm for his work all combined to create an awesome interview experience, and I highly recommend that you read the full case study and listen to the podcast.

Interview with Sudan Vote Monitor

On Friday I spoke with Fareed Zein, who heads the Ushahidi-based project Sudan Vote Monitor. The project lets citizens report problems with access, illegal campaigning, voter harassment and other aspects of the election.

Sudan’s first multiparty elections in over two decades began yesterday (New York Times, Al Jazeera) despite the fact that the majority of opposition parties are boycotting the entire process.

The election — already marred by a lack of actual ballots, long waits at polling places and mix-ups in the symbols printed on the ballots (many voters are illiterate, and symbols are used to represent different parties and candidates) — is widely seen as a prelude to the upcoming referendum on the independence of Southern Sudan.

On Friday I spoke with Fareed Zein, who heads the Ushahidi-based project Sudan Vote Monitor. The project lets citizens report problems with access, illegal campaigning, voter harassment and other aspects of the election. So far the site has received over 100 reports in both English and Arabic (volunteers at Meedan are helping with translation). Zein, who was busy working to get an SMS short code set up and doing some last-minute testing, spoke a little bit about his hopes for the impact of Sudan Vote Monitor:

I would say even if the election doesn’t take place we’ve already made history, and that’s not to say that that’s where we’ll stop, but this is a groundbreaking undertaking. We’ve already done a big service to just introduce the concept, introduce the possibility.

Other groups have specific activist motivations. They have a different tack. Ours is just getting access to information because the Sudanese people as well as the rest of the world have not had that in previous events. Others will take that to the next level and try to apply pressure for change.

You can read the whole interview and listen to the podcast of our chat at the Technology for Transparency Network.

Mobile Money: A Recap

I’m spending today at the Macroeconomics of Mobile Money conference at the Columbia Institute for Tele-Information (CITI).

Liveblogging. Please excuse misrepresentation, misinterpretation, typos and general stupidity.

James Alleman is giving closing comments. His takeaways:

There is a class that’s underserved by formal banking, and there are a lot of people who are ready to use mobile banking services in the developing world.

The success of existing services like M-Pesa and Menekse Gencer is impressive and lays a good foundation for future efforts. It appears as though mobile banking efforts will need a formal banking partner to be truly successful.

We still don’t have a good idea of what kinds of regulatory systems are going to be required.

Anonymity is a major question: balancing privacy with criminal threats.

Right now, security is an afterthought. This is not good.

There are too many “standards” right now.

User interface is key.

High demand for VoIP; the people who are demanding this (migrant workers, those who want to send remittances back to their families) are also good candidates for mobile banking services.

Walled gardens are a huge problem: everyone wants a piece of the action, but no one wants to cooperate.

Overall: this is a huge, growing, untapped market with many issues left to be resolved.