Beyond Privacy and DSARs: Public Data Requests (FOI) Are the Law in 50 States

Happy 2020!  The New Year brings Californians under the California Consumer Privacy Act (CCPA). CA consumers can ask state-based companies for all relevant data, and to request that companies delete their data. Not in compliance with CCPA? Don’t panic yet. California will not be enforcing the law for another six months. If you’re looking for solutions, we can help out.

The CCPA also gives us an opportunity to talk about another type of law in effect in all 50 states. They are known collectively as public data access laws and are modeled on the federal government’s Freedom of Information Act (FOIA).  Private companies are excluded from FOI-type laws of course.

State governments, generally the executive and legislative branches but also state agencies, have to make their records available upon request to the public. State court have their own rules for accessing court records. To make matters more complicated, local city governments can have their variant of FOI laws. And there are exemptions built into all these FOI laws to restrict access under certain conditions.

Freedom of Information for the People

US states enacted their own FOI legislation back in the 1970s, mirroring the federal law. Generally, a particular state organization (branch of government or agency) would have to initially respond to a data request within a given period of time under a state law. The cost of these requests can vary depending on  how much staff time is involved. If you want to get a feel for these laws, you can check your own state’s public or open record act (see below for a snippet of New York’s own FOI law).

One interesting point to make is that while private companies are now facing data subject access requests or DSARs for the first time, state- and federal-level entities have long had to deal with these types of requests. While the new privacy laws, such as CCPA, call for free access to data (allowing for “reasonable charges” when the requests are excessive), the state FOI laws have too often been associated with highs administrative fees for the requestor: for example this case in Michigan involves processing fees over of over $40k!

These fees are typically based on the hourly rate of the lowest-paid government worker involved. Key point: without automation, costs for manually searching through huge file systems and databases add up quickly!

Along Comes MuckRock for FOI Automation

Thankfully, there’s a great resource to understand how well states are doing in dealing with public record requests. I refer you to Muckrock, which is an non-profit that tracks FOI requests through their own clever web-based system. It’s worth nothing that even with help from Muckrock state public access requests have their own quirks, and you may still need the services of an attorney to help out with more complicated submissions.

In any case, Muckrock provides a single portal to access the separate FOI online sites, along with providing useful information on exemptions, required response times, average fees, and some other stats on previous requests. If you want to get a handle on New Jersey’s FOI law, you can click at https://www.muckrock.com/place/united-states-of-america/new-jersey/,  and the same URL pattern repeats for other states.

What if you needed to drill down a little and look at individual agencies with a state? Muckrock can help here as well. They provide RESTful Web APIs and a Github repository with sample Pythons scripts showing how to use the interface to access a ginormous database covering every federal, state, and local government agency in the US!

Let’s say you wanted to learn the number of and the average delay in processing FOI requests to the NJ State Police. You’d use the agency variant of the API and fill in with the agency id, which happens to be 839: https://www.muckrock.com/api_v1/agency/839/. Click on the link to see the results in Muckrock’s visual interface.

And the answer is that 60 requests have been tracked, only 17 completed, and there’s an average response time of 60 days. Not bad, but not great either.

How did I know the NJ State Police was assigned a Muckrock agency id of 839? Based on poking around  the APIs, I learned a brute-force approach was required, and so I downloaded the whole shebang using their export_agency_stats.py script. This effectively dumps the  database of agency ids along with stats.  With over 14,000 data rows, it takes forever. However, I’ve done all the dirty work, and you can find the Muckrock  database, as a .csv file, in my Github repository.

Yeah, it has the stats of the time of this writing. But six months from now you can search for the agency id in your Excel spreadsheet and then pull in the current stats using the API I showed above.

And the Top Ten Agencies Are …

I’m convinced this new decade will launch a new awareness of privacy among consumers with the CCPA helping to spur copycat legislation across the US. As I mentioned above, governmental agencies are under their own rules, but I suspect they’ll be more FOI requests, and new pressure to improve performance. If these agencies don’t, at least in some states, there can be significant penalties.

With the dataset I placed in my Github repository, you can do some of your own interesting  analysis. I’ll close this post with an interesting factoid that answers the question: Which agencies at the state level  — excluding cities and local governments — have the longest responses for  completing FOI requests?

Keeping in mind that this represents requests submitted through Muckrook, the answer is: