Tools for Transparency: Google Refine

Originally posted as a guest post on the Sunlight Foundation blog.

For the past six months, I’ve served as the co-director of the Technology for Transparency Network, an organization that documents the use of online and mobile technology to promote transparency and accountability around the world. One of the most common challenges the project leaders we’ve interviewed face is making sense of large amounts of data.

In countries where governments keep detailed digital records of lobbying data and education expenditures, data wrangling is a time-consuming, labor-intensive task. In countries where these records are poorly maintained, this task becomes even harder — everything from inconsistent data entry practices to simple typos can derail data analysis.

Google Refine (formerly Freebase Gridworks) is a free, open-source tool for cleaning up, combining, and connecting messy data sets. Rather than acting like a traditional spreadsheet program, Google Refine exists “for applying transformations over many existing cells in bulk, for the purpose of cleaning up the data, extending it with more data from other sources, and getting it to some form that other tools can consume.”

At its most basic level, Google Refine helps users quickly summarize, filter and edit data sets by allowing them to view patterns and to spot and correct errors quickly. More advanced features include reconciling data sets (i.e., matching text in the set with existing database IDs) with data repository Freebase, geocoding, and fetching additional information from the Web based on existing data.

Though it runs through an Internet browser, Google Refine operates offline, making it attractive for those with limited bandwidth or privacy concerns — a group that includes many of the projects listed on the Technology for Transparency Network.

Google Refine isn’t going to solve the problem of poor data availability, but for those who manage to gain access to existing records, it can be a powerful tool for transparency.

For more information, check out the links and video below:

SIPA Shushing Students over CableGate: Update

I was out of town this weekend, blissfully disconnected from phone and Internet. I came back to find out that since I blogged about Columbia’s School of International and Public Affairs warning students not to discuss Wikileaks online if they ever want to work for the State Department, the story has, well, exploded.

The Huffington Post got in touch with State Department spokesperson Philip J. Crowley, who said the SIPA alum who e-mailed the Office of Career Services was acting of his own volition:

This is not true. We have instructed State Department employees not to access the WikiLeaks site and download posted documents using an unclassified network since these documents are still classified. We condemn what Mr. Assange is doing, but have given no advice to anyone beyond the State Department to my knowledge….

If an employee of the State Department sent such an email, it does not represent a formal policy position.

So it looks like the blame for OCS’s ridiculous “advice” falls squarely on the shoulders of SIPA, then. Excellent.

SIPA Shushing Students over CableGate. Seriously?

Yesterday a friend forwarded me a link to a blog post about Wikileaks. Not surprising, given the number of Wikileaks-related blog posts that are floating around the Internet in the wake of the organization’s release of a quarter of a million U.S. Embassy cables. But this blog post was different: this blog post referenced the Columbia University School of International and Public Affairs (SIPA), from which I graduated six months ago.

The author reposts an e-mail sent from SIPA’s Office of Career Services to all current students. It reads:

From: “Office of Career Services”

Date: November 30, 2010 15:26:53 ESTTo:

Hi students,

We received a call today from a SIPA alumnus who is working at the State Department. He asked us to pass along the following information to anyone who will be applying for jobs in the federal government, since all would require a background investigation and in some instances a security clearance.

The documents released during the past few months through Wikileaks are still considered classified documents. He recommends that you DO NOT post links to these documents nor make comments on social media sites such as Facebook or through Twitter. Engaging in these activities would call into question your ability to deal with confidential information, which is part of most positions with the federal government.

Regards,
Office of Career Services

I’m currently happily employed at the Berkman Center for Internet & Society, but while I was at SIPA I seriously considered a career in the Foreign Service. I applied for (and was offered) a summer internship at the State Department, and I coordinated a conference on Policy Making in the Digital Age, at which the State Department’s Director of the Office of eDiplomacy and a representative of the Office of Innovative Engagement spoke.

I guess I can kiss that possible alternate career path goodbye, given that I tweeted a link yesterday to an article about CableGate. Seriously, State Department? This is all over the news. What’s more, it’s become a focal point for discussions on how digital technology is changing our expectations for government transparency (for those who’ve forgotten: the State Department is big on using tech to promote transparency in other countries. Just not here in the US?).

Seriously, SIPA? As fellow SIPA alum Ben Colmery pointed out in a comment on my Facebook wall, since when does having an opinion about a site leaking documents equate to actually leaking documents oneself? You claim to provide committed students with the necessary skills and perspectives to become responsible leaders. Apparently that means curtailing their academic freedom and teaching them how to bury their heads in the sand.

Crossposted on The Morningside Post

Update, December 6: The State Department is denying that it provided “advice to anyone beyond the State Department” regarding Wikileaks and claiming the information in the OCS email “does not represent a formal policy position.”

Tech for Transparency: New Interviews Posted

Avid readers of my blog (here’s looking at you, Rev) may remember that several months ago I announced that research was beginning for the second phase of the Technology for Transparency Network. The first phase consisted of interviews with over 30 projects around the world who are using technology to promote transparency and accountability in the government and/or private sector. Our goal in the second phase was twofold: to double the number of case studies on the site and to expand the geographic regions we covered.

Since then, I’ve been largely silent about the project — we’ve been working so hard to complete and edit the interviews that I haven’t had much time to breathe. But today I’m thrilled to announce that we have eight new case studies online, with lots more to come over the next few weeks. The case studies that have been posted so far are:

Accountability Initiative
Accountability Initiative researches and creates innovative tools to promote transparency and accountability in India’s public services.

Amatora mu Mahoro
Amatora mu Mahoro (“Peaceful Elections”) is an Ushahidi-based project created to monitor Burundi’s 2010 elections.

Association for Democratic Reforms
ADR India works to monitor national elections through country-wide SMS and helpline campaigns and an informational website.

Democrator.ru
Democrator.ru seeks to empower citizens by helping them collectively send petitions and inquiries to government bodies.

Excelências
Excelências fights corruption in the Brazilian government by publishing data about politicians and government activities online.

Golos
Golos (Voice) has introduced several online tools for better election monitoring in Russia.

Mam Prawo Wiedzieć
Mam Prawo Wiedzieć helps Polish citizens access information about their elected representatives in an easy, user-friendly way.

Pera Natin ‘to!
Pera Natin ‘to! (It’s Our Money!) encourages Filipino citizens to report times when they are asked for bribes.

In addition to continuing to post new case studies (you can subscribe to our case study feed via RSS), we’ll also be publishing our final report on both phases of the project by the end of the month. In the meantime, check out @techtransparent and our Facebook page for daily updates and our podcast for interviews with the project leaders!

Juliet Schor on “Post-Industrial Peasants”

Sociology professor Juliet Schor is at the Berkman Center today to talk about how the sustainability community — both activists and practitioners — is increasingly using the Internet to “foster new lifestyles, consumption patterns and ways of producing.”

Liveblogging Juliet Schor’s presentation “Using the Internet to ‘Save the Planet'” at the Berkman Center. Please excuse misrepresentation, misinterpretation, typos and general stupidity.

*****

Sociology professor Juliet Schor is at the Berkman Center today to talk about how the sustainability community — both activists and practitioners — is increasingly using the Internet to “foster new lifestyles, consumption patterns and ways of producing.” Her presentation is based on her recent book Plenitude: The Economics of True Wealth, in which Schor argues that by shifting to a more sustainable way of life, we can improve both the environment and our economic situation. While writing the book, Schor says, she came to believe that the sustainability and technology communities should have a much closer relationship.

It sounds crazy — “post-industrial peasants” — but there are some very important features to that: diversity of activities and income streams is key. Putting all your eggs in the basket of one employer is riskier and riskier in times of economic uncertainty. The single income stream strategy is becoming less attractive, and diversification is smart. The reason it makes sense now in a way it wouldn’t have 50 years ago is because of technology. Technology allows a single individual or a small company to be productive in ways they couldn’t have before — access to the network, access to information. This is the next stage after “big.” The large economies of scale will be less important going forward, and small-scale efforts will become more important.

Schor starts out by describing a “dramatic collapse” in biodiversity since the 1970s, the growing ecological footprints of different countries (hint: the United States is at the top, using more than four times the world average biocapacity per person), and our collective failure to reduce global carbon dioxide omissions. (She points out that recent data shows that the best way to reduce emissions is to have economic collapse, though that’s not practical as a long-term strategy.)

Schor argues that a purely technological approach won’t halt climate change — this is also a problem of scale. According to a recent paper in Nature, we have already exceeded two of nine different “planetary boundaries” (in categories such as climate change, ocean acidification, biodiversity loss, and others), and we’re close to hitting the sustainable boundaries on a number of others. The strategy of de-materialization (reducing the “material intensity” of our energy use) has had some success, but our economic growth has “more than outweighed the decline” in material intensity. On a worldwide basis, Schor says, our material intensity has actually increased by around 45%, while North America has been a particularly egregious user of materials — our material extraction has increased by about 66% since 1980. This is largely due to our use of fossil fuels and the construction boom.

The Challenges

Schor argues that the world needs to cut its ecological impact rapidly. The problem, she says, is that we’re in the midst of an unemployment crisis. This is a disaster both economically and environmentally speaking. We also shouldn’t take any paths that worsen the distribution of wealth — there’s a negative correlation between income inequality and certain environmental indicators — or decrease human development (i.e., wealth and well-being) overall.

Plenitude: The Economic Model

Switching to green technology (a clean consumption and production system) will help, Schor says. So will improving eco-knowledge, which she defines as “open source transmission and ecological skill diffusion.” We’re “centuries behind” in terms of developing both an understanding of nature as a scarce resource and technology that would allow us to increase the productivity of that resource.

Schor points to working hours, which have declined dramatically between 1870 and the 1970s (from around 3000 to around 2000 per year). Since 1973, however, the annual hours worked have been increasing the United States. A country’s ecological footprint rises with its average annual hours worked, even when income is held constant. Schor says that as we move forward, we need to focus on increasing productivity growth in fewer working hours, rather than by adding new hours. She wants to move hours from the “business as usual” economy to “self-providing” and green entrepreneurship. This will reduce market dependence and reliance on large corporations and provide more time for people to increase their skills, build local resilience, and help create a small-scale, low-impact sector of enterprises.

Schor provides an example in the form of permaculture (a high-productivity approach to agriculture) and urban agriculture. This form of micro-generation, which applies not only to farmers’ markets and fruits and vegetables but also to energy and homes (DIY yurts, anyone?), is low-cash and low-footprint in comparison to more market-driven methods and mechanisms. Schor is currently working on a number of other case studies, including a permaculture farm in the Netherlands, a converted soybean farm in Kansas build with fab lab technology. This farm is also trying to build a blueprint for other communities to follow. What’s cool about this, Schor says, is the low financial barriers to entry: communities can purchase the machines and the costs of materials are low.

Schor’s also interested in the principle of sharing: couches, homes, cars, tools, etc. She says the recession has “changed the calculus of time and money,” creating an environment that fosters these sorts of sharing schemes. Another initiative that has sprung up in this environment is the transition movement, which focuses on helping communities build local resilience.

Overall, Schor says, our constraint is much more about time — we work long hours in formal jobs, which we need in order to have access to health insurance, housing, and education. We need to find ways to allow people to “delink” from these jobs, which are high footprint jobs, to allow them to do more of this kind of small-scale activity.