HOW TO CATCH BAD DATA BEFORE IT CATCHES YOU

We don’t have a degree in statistics. Nor are we data specialists. But after 30-plus years of using data as a basic tool in our quest to tell stories about cities, counties and states – all of which are carefully fact checked -- we’ve developed a kind of radar (not that it’s infallible by any means) for data which can be misleading, misplaced or downright wrong.

The following is a distillation of some of the approaches we’ve developed over the years that have helped us to steer clear of prose that is gritty with numbers that stand in the way of an honest interpretation of the facts.

Here are eight:    

1)    Watch out for any information that compares present day data to that which was gathered during the pandemic. In 2020 and 2021, there was an upheaval in many sectors of society. Now, a little more than five years after the onset of the pandemic, any changes that use 2020 or 2021 as a baseline are nearly sure to be misleading. That’s why some institutions that are particularly careful about their use of data compare pre-pandemic to post-pandemic. 

2)    Beware of huge ranges in costs for projects.  We frequently see data that purports to show how much a project will cost, or the number of people affected by a policy. When the range is huge, we suggest you ignore it entirely because with this little precision it’s pretty much meaningless. For example, a few years ago a Dallas based media outlet reported that “researchers say it would cost anywhere between $5 billion and $20 billon to winterize all plants in Texas.”   

3)    An additional cautionary note along the same lines. Sometimes ranges like this are translated into wording like “up to $16 billion,” in order to prove that it’s an enormous number or, for the same project, around $5 billion, when the author wants to minimize things. It’s hard to ferret out this kind of misrepresentation, but worth watching out for.

4)    Super precise numbers are rarely reliable, outside of budgets, where precision is necessary. But when a report or a study talks about something and goes out many decimal places, in most cases such precision doesn’t exist in real world activities.

5)    It’s amazing how many academic reports ground themselves in data that’s more than 25 years old. Very little that was true then is true now, and yet these ancient numbers seem to be stuck in time. For example in a 2024 paper about   “The effect of pay for performance on work attitudes,” it’s stated that “expectancy theory posits that individuals act to maximize expected satisfaction with outcomes.” This may still be true, but the source is footnoted to a paper written in 1964. 

6)    Make sure the sample size for any kind of survey is reasonably large. While anecdotal evidence can be gathered in surveys that are answered by a couple of dozen respondents, when those numbers are converted into percentages, then the flaws creep in. If the sample group is really small, then the conclusions drawn from them can be totally worthless Say, for example, you read that an astonishing 25% of people believe something that, at first glimpse, seems unlikely. But if only 16 people responded to the survey, then the four of them that make up the 25% can’t be held to be representative of society at large. 

7)    Beware of bar charts and similar comparative graphics. If data   is shown on a bar chart with a smaller range, then relatively small changes can appear to be quite huge. Let’s say, for example, that the chart shows a range from 1 to 100, and it’s moved by 3 units from 1-4 say. That movement may look relatively minimal.  But, instead, consider someone who decides to have the chart peak out at 10. The same exact change can appear to be cataclysmic.

8)    Try to find the time to read beyond the executive summaries that focus on the data in a report. For organizations that want to make a particular point, it’s a strong temptation to use the executive summary as an opportunity to boast about the positive and underplay the negative (or the other way around). With little time to read through every page in a report, it’s easy to be trapped by this phenomenon (and we suspect that occasionally that’s been true of us). But it’s worth the time to dig a little deeper before believing everything you read.

#StateandLocalGovernmentMangement #CityGovernmentManagement #CountyGovernmentManagement #StateandLocalGovernmentPerformance #CityPerformance #CountyPerformance #StateandLocalGovernmentData #StateandLocalDataVisualization #StateandLocalDataUse #CityDataCaution #CountyDataCaution #StateDataCaution #StateandLocalTransparency #StateandLocalPerformanceManagement #StateandLocalDataGovernance #CityDataUse #CityDataGovernance #CountyDataUse #CountyDataGovernance #MisleadingData #GovernmentData #InterpretingStateandLocalGovernmentData #BandGReport #BarrettandGreeneInc