Why Is Access To Public Records Still So Frustratingly Complicated?

Earlier this month, House Oversight Committee Chairman Jason Chaffetz wrote to the Federal Bureau of Investigation demanding former FBI director James Comey’s notes of conversations with President Trump.

Congress has wide discretion to investigate what it chooses, including the right to issue subpoenas demanding access to potential evidence. But Chaffetz isn’t the only one interested in reading Comey’s notes, which reports have indicated document the president attempting to influence the FBI’s investigation into Russian election interference. The American Civil Liberties Union and Electronic Privacy Information Center have both filed requests under the Freedom of Information Act for the documents.

“While we hope that Congress will be serious about its constitutional oversight role, we believe the issues raised by the Comey memos are so vital that the public should have access to them without delay,” wrote Hina Shamsi, director of the ACLU National Security Project, in a blog post.

The Freedom of Information Act, often known as FOIA, has been used by journalists, activists, and private citizens to get access to federal government records since it went into effect in 1967. And every state has passed similar laws that allow the public to get access to state and local records, generally with exemptions for files like records of ongoing investigations or personal medical records. (Florida’s are called the Sunshine Law.) The trouble, say transparency advocates and people who rely on open records laws for their day-to-day work, is that in an era when files can be searched, copied, and transmitted in minutes at minimal cost, many agencies still respond to requests with excessive delays, claims of high processing costs, and files produced in difficult-to-handle formats like scans of printed versions of digital documents.

“We’ve had situations where someone says I want the Excel spreadsheet, but one of the custodians of the record will say ‘no,’” concerned that would make it too easy to tamper with the document, says Daniel Bevarly, executive director of the National Freedom of Information Coalition.

And last year, the Electronic Frontier Foundation published a collection of FOIA horror stories, including a decade-long delay from the Defense Department and a bill from a Texas sheriff’s office for $98 in Wite-Out, presumably used to redact documents.

The problem isn’t universal. Some federal agencies have taken steps to improve how materials are released—for example, a State Department system that includes searchable copies, not just scans, of Hillary Clinton’s much-discussed emails. And some states have adopted laws requiring documents be produced in their original digital formats. “Florida, it’s written into their Sunshine laws, that any information requested be provided in its native format to the petitioner,” says Bevarly. (Those sorts of requirements aren’t always perfectly followed. The EFF’s stories include a report from a MuckRock journalist who contested a request from controversial Kentucky county clerk Kim Davis for $1,200 in photocopying costs for email records, despite a state law requiring digital documents).

But a look at Chaffetz’s letter reveals a level of specificity in how the FBI documents should be formatted, indexed, and turned over that would make most frequent FOIA users jealous. The letter extensively spells out how the FBI files should be serial-numbered, the digital format in which they should be produced, and the 37 pieces of data, including times, dates, email subjects, and document text, that should be included with each one.

Chaffetz’s letter also asks that the documents themselves be accompanied by index files formatted for Concordance. That’s a software tool from legal tech giant LexisNexis used by lawyers to organize documents turned over during litigation, in the evidence-sharing process called discovery. Just as government agencies and members of the public can find themselves overwhelmed by the volume of relevant documents to a FOIA request, lawyers can face similar issues finding relevant evidence to turn over to the other side in a complicated lawsuit, and analyzing the documents they in turn receive.

“In the past, attorneys would use sticky notes and sometimes different color sticky notes for issues, and they would have all their paper and that’s how they would go through and make notes about important issues,” says Mary Mack, executive director of the Association of Certified E-Discovery Specialists. “The paper [piles] just kept getting bigger and bigger, and they kept needing bigger and bigger conference rooms to do this.”

Eventually, as email and other digital tools became ubiquitous, paper discovery became effectively impossible, and courts and lawyers adapted increasingly sophisticated systems for digitally sharing evidence.

“There was a step where they would exchange basically just unrelated pages but they were on a disk—so they called it electronic discovery,” says Mack.

And in 2006, the federal court system adopted rules requiring litigants to share documents in their original format or “a reasonably usable form,” and various federal and state courts have since adopted more specific standards. The Delaware federal court, for example, specifically spells out that files should be produced as text-searchable images, complete with Concordance indexes, unless they’re spreadsheets, databases, or other files that are difficult to visualize as images.

But enforcing those kinds of rules may be more feasible for litigation, where court rules generally specify that the party turning over documents is responsible for the costs involved, and judges are on hand to enforce rules of discovery, than for public records. FOIA users like journalists who are considered to be acting in the public interest are often entitled to exemptions from fees, and agencies often have little wiggle room in their budget for handling complex records requests.

“I think public records are behind because when they put the FOIA laws in and the Sunshine laws in, they didn’t necessarily put a budget allocation in to do it,” says Mack.

That problem can be compounded by aging IT infrastructure that can make it difficult to search for relevant records, let alone produce them in a modern software format.

“In many jurisdictions, they’re still using mainframes,” says Alexander Howard, deputy director of the pro-transparency Sunlight Foundation. “The federal government still employs people who know [the legacy programming language] Cobol, and so do many states.”

The group advocates a potentially simple solution, which has been adopted to some extent by cities, states, and federal agencies that have launched online open data portals: Publish any records that aren’t required to be kept private.

“One of the things that Sunlight has been pushing for for over a decade now is the simple idea that public information should be available online to the public by default, except where there are demonstrated security and privacy risks and harms, or other exemptions that exist under a state’s public records law,” Howard says.