Requesting and Analyzing Public Data

(Michelle Rhee, Flickr/ angela n.)

Before the headline and follow-up stories, what is the legwork necessary in creating a sound data-driven news report? As part of EWA’s series on Diving into Data, investigative reporter Jack Gillum shared his experiences as one of the lead reporters on the award-winning USA Today series that looked at suspicious student scores on standardized tests across six states and the District of Columbia.

The 2011 series of articles, which focused on the years Michelle Rhee was still chancellor of D.C. Public Schools, was one of several news accounts that documented wide-scale test results that in many cases were too good to be true. Gillum, who is now with the Associated Press, said that while he majored in computer science, old-school reporting is what drives an enterprise story. “For all the data analysis done on this project, the one thing that sort of remains clear is that all the data in the world can’t substitute old-fashioned, shoe-leather reporting,” Gillum said, “which is how we were tipped off to this story in the first place.”

[See EWA's Story Starter on Standards and Testing.]

Still, data help to “bullet-proof” stories and limit a public official’s ability to dismiss a reporter’s findings. Often, the information originated from figures culled from data the school district collected, such as erasure analysis that convincingly demonstrates which schools and classrooms had likely instances of test tampering.

But collecting erasure analysis reports—manna for school system watchdogs—is not enough to flag a district for suspicious test results. State and local school systems keep granular data that are composites of the whole picture. The idea was to develop evidence to spot aberrations in trends, which required background on how various demographic groups are likely to perform on various tests. State-level data is useful, too, because sudden jumps in school performance are visible when comparing current results to previous years.

For the USA Today series, the data were instructive. Erasure analysis reports indicated certain schools had numerous students erase their wrong answers for the right ones at unlikely levels of plausibility. The odds were one in 6.45 trillion that so many students changed their answers from a wrong answer to a correct one without assistance, Gillum noted. (Read a summary of the D.C. Inspector General report here.)

Receiving documents through Freedom of Information requests added to the series’ narrative. After a lot of haggling, the news team managed to win access to erasure analysis scores down to the student and teacher level. The students’ names were replaced with random serial codes to protect their identity, while their teachers’ names were revealed, allowing Gillum and his colleagues to “knock on doors.” That access was likely helpful, since the Rhee administration refused USA Today requests for interviews and restricted access district schools. (The New York Times wrote an article in 2011 that began with, "Why won’t Michelle Rhee talk to USA Today?")

Public records requests revealed a bevy of information, some confirming through-the-grapevine rumors of a principal using strong-arm tactics to compel the school’s teachers to increase test scores. In that case, the reporters requested public records of police reports related to that school, coming across a write-up that corroborated their source’s information. Other FOI records unveiled more sordid details, such as a senior communications official (Anita Dunn, who at one point worked for the White House of the Obama Administration) emailing her subordinate to avoid Gillum’s emails.

Gillum has an almost absolutist view of a government agency’s duty to provide access to information. “Public schools are public…they’re paid for by our tax dollars” and the information they keep should be open for scrutiny, he said.

He offered other tips:

• Make a habit of requesting FOI documents weekly or monthly. 

• Know when the school system collects crucial data like teacher pay, enrollment, discipline reports, and other items that can be scrutinized through a FOI request. 

• Have reliable data professionals from a local research group or university who can explain what the data you’ve collected means. 

• Call other districts on background to learn what data they keep and how that information is organized—the equivalent of a cabinet inventory—so you can call out your beat’s school district for withholding information. 

• Public relations officials have a job to do and you do, too. The job duties required of both will put you at odds, but that’s the nature of game. 

• Taking teachers, principals, district officials, and other sources out for coffee or lunch on your own dime will help improve relations for when a news story requires their cooperation. The dollars might add up but the connections and information gathered will be invaluable. 

• Private companies are not under any obligation to reveal information, even if their work is school-related. The only reason test company reports like erasure analysis are available through FOI is because they were paid for with public funds 

• Know the FOI restrictions in your beat. Gillum notes the U.S. Congress exempted itself from FOI requests, so one would have to request information from the executive branch to access correspondence with federal legislators. 

• With standardized testing taking on a digital format, it’s important to consider how the state will replace the practice of erasure analysis. The move from paper tests with graphite pencils to online tests and what will still require monitoring schools for suspicious test scores. Gillum suggested that reporters may be able to request the digital meta-data underneath the “official” test score responses. Meta-data are the notes in a digital file that can show a history of edits, logins, file size, and other relevant information.