BAD DATA IS EVERYWHERE: BEWARE!

Way back in 2015, we wrote a cover story for Governing about bad data. (And by the way, we’re delighted to report that after a few absent years, we have re-established our connection with Governing, which we will be writing for and which will be periodically reprinting articles from this website).

At the time we wrote that “data is the lifeblood of state government. It's the crucial commodity that's necessary to manage projects, avoid fraud, assess program performance, keep the books in balance and deliver services efficiently. But even as the trend toward greater reliance on data has accelerated over the past decades, the information itself has fallen dangerously short of the mark. Sometimes it doesn't exist at all. But worse than that, all too often it's just wrong.”

Sadly, based on our careful reading of a number of performance audits and press reports, it turns out that even though the public sector has become even more obsessed with using data to make decisions, the quality of the data is still in serious question.

Consider, for example, a discovery near the end of last year that Florida and Texas had been sending the EPA inaccurate data about the status of their lead pipes, which the federal agency didn’t bother to check. Ultimately EPA’s Office of Inspector General discovered the flaws, but “the errors mean some states with the biggest needs may have to wait longer for funds — or will get less than they should have,” according to an article by Scripps, which wrote, “For example, a single data entry error by Houston caused the EPA to allocate nearly $120 million more to Texas than it probably should have in fiscal year 2023.”

In fact, some experts are concerned that the quality of public sector data may even be getting worse. Some months ago, we quoted Mark Zandi, chief economist at Moody’s Analytics who said this in a podcast called “On the Evidence” from Mathematica:

One of his greatest concerns is that “the quality of the data we’re using is starting to erode because a lot of it is based on surveys and survey response rates are way down across the board.”

He added that “There are concerns around privacy, cyber issues, but for whatever reason, the response rates are way down and that’s beginning to affect the data to a significant degree.”

Zandi takes this matter very seriously, and said, ““If I were king for the day and I could devote resources to one thing, that would be it. Let’s go make our data sources more resilient, better, more comprehensive, more timely.”

Sadly, many people throw statistics around without the context that truly makes them useful. Just a few weeks ago we attended a conference where one of the keynote speakers was justifiably complaining about the problem of gun violence in the United States. She told the audience that guns were the leading cause of violence in the United State, but she missed out on one important wrinkle that put that number into important perspective.

According to the Pew Charitable Trusts, “though they tend to get less public attention than gun-related murders, suicides have long accounted for the majority of U.S. gun deaths. In 2023, 58% of all gun-related deaths in the U.S. were suicides (27,300), while 38% were murders (17,927). The remaining gun deaths that year involved law enforcement (604), were accidental (463) or had undetermined circumstances (434), according to CDC data.”

Sometimes numbers that are accepted without question simply don’t make sense at all. Many, many years ago, the newspapers were full of stories that claimed that there were one million missing children each year. Milk cartons were full of warnings about the threat that children were disappearing on a regular basis.

At the time – and long before we started to write about state and local government -- It struck us that one million seemed like an extraordinarily high figure, particularly since in our experience, we barely read about scores of missing children in the New York papers, with the exception of a six-year old named Etan Patz who disappeared on route to his Manhattan school bus. We looked behind the “millions” figure and discovered that it had originated with a politician who picked that number out of a hat when speaking to the press. It was picked up from him, and used and reused, and this piece of fiction was widely felt to be fact.

We wondered about this, but never wrote an article about the greatly exaggerated number. Then the Denver Post discovered that “estimates of stranger abductions circulated at the time glossed over the reality that about 95 percent of missing-child reports were about runaways, while most of the rest involved custody disputes.” In fact the reporters won a Pulitzer Prize for this coverage in 1986.

Footnote: We’ve always regretted not having done that story ourselves, which may be a subconscious part of the reason we’re so intent on reporting about bad data every chance we get.

#StateandLocalGovernmentData #StateandLocalDataManagement #CityGovernmentData #CityDataManagement #StateandLocalDataQuality #StateandLocalBadData #StateandLocalDataCaution #StateandLocalDataGovernance #CityDataGovernance #StateandLocalDataUse #ErroneousDataReporting #MisleadingData #ExaggeratedData #PublicSectorDataQuality #DenverPostMissingChildrenPulitzerPrize #InaccurateMissingChildrenData #StateandLocalGovernmentManagement #CityGovernmentManagement #BarrettandGreeneBadData #StateandLocalPerformanceAudit #InaccurateLeadPipeData #GoverningMagazineBadData #SurveyDataQualityErosion #BandGReport #BarrettandGreeneInc