THE STORY BEHIND THE DATA

Cities, counties and states are overladen with data, and with the advent of AI, it’s likely that the current flood will be turning into a tsunami. But we’re often dismayed when we speak with people in these entities only to discover that they can spout out plenty of numbers, but they can’t really understand the story behind them.

At a recent conference, for example, one of the speakers showed an alarming slide that demonstrated that a particular city had a disproportionate amount of crime. We asked if he could explain why this was the case, and it turned out that though he could show us all the numbers, he couldn’t explain the “why” behind them. (Side note: When we searched the Internet about this particular episode, we discovered that the data we’d seen was a couple of years old, and they’d improved dramatically since.)

We understand the value of purely quantitative data and we know that many academics take a qualitative approach, as well. But when one exists without the other, there's something missing.

Now, let’s go back to one of the highlights of our career, which we believe can be an example of the way we’d like data to work. It was called the Government Performance Project (GPP), and it was funded by the Pew Charitable Trusts, and published  in Governing Magazine between 1998 and 2008. The goal was to evaluate the management capacity of the nation’s states on a regular basis in such important fields as human resources, performance management, information technology, infrastructure and budgeting. On other occasions, the same project involved large cities and counties.

For each of these multiple projects, we had two teams of researchers -- one of which was made of university professors and graduate students and the other made of journalists. Together, we developed a number of criteria and sub-criteria in order to make our evaluations.

The academics and grad students culled carefully through a lengthy survey instrument which was developed with the full team. With just a few exceptions each year, we received all but a few surveys back. (This wasn’t easy, by the way, and we regularly had to make the point to reticent governments that they could choose  not to cooperate, but they couldn’t opt out of the evaluation and filling out the survey provided their best chance of the GPP getting their story right.)

Our academic partners studied the answers and translated the responses many of the answers into numeric format which could be used to yield a grade for all the major fields we were examining.

Simultaneously, the journalistic team, which we headed up, read the surveys and did hundreds of interviews with leaders and external observers in the governments and pursued each of the criteria and sub-criteria through lengthy interviews. The team of interviewers then took their best stab at using the information to evaluate what they garnered through these conversations. 

When the journalists and the academics didn’t come to the same conclusion, we researched further, discussed the issues, and came to a final decision.

The point of this self-serving reminiscence is that without the interviews, it was easy for the pure data to lead to incorrect assumptions. Meanwhile, the academic approach made the whole process far more rigorous and helped to catch issues that come up in interviews (for example, people trying to put a happy face on a bad situation).

This exercise was hugely successful, and it was remarkable how few governments disagreed with our conclusions – even when they were negative. The major issue some had was with the criteria we were using in the first place. But the project was careful to make the criteria clear.

This was a very time-consuming and expensive multi-year effort and we understand that without the support of an organization like Pew it wouldn’t have been possible. But, still it was an experience that leads us to believe that without actually digging deeper  into the gritty numbers, there’s every chance that they may be misleading for any number of reasons.

Consider this hypothetical example: A city agency carefully measures processing times for permits, and comes up with data that show its efforts cut the time to process a permit by 40%.

By itself, the city and analysts can declare victory, and boast about it to all concerned. But interviews with people involved may uncover the notion that the entity was short-changing the proper amount of effort to make sure a permit was genuinely deserved. As a result the positive data may be hiding flaws in the system that are of significance.

Of course, when conclusions are reached purely through conversations, that’s far from perfect, too. We’ve also come across papers and articles that are based exclusively on a handful of interviews. When there are hundreds of them, and care is taken to make sure the interviewees represent a cross section of the population being studied, that can work out pretty well. But we’ve seen too many instances when only a small handful or people were involved.

Ultimately,  if a major study or a report is published, we’d like to see that it’s based on a combination of carefully gathered data and journalistic interviews to put it in context. To be perfectly honest, resources -- both time and money -- make the capacity to do as huge a project as this difficult, but a collaborative approach still provides a great model for telling the story behind the data.

#StateandLocalManagement #StateandLocalGovermentPerformance #StateandLocalPerformanceManagement #GovernmentPerformanceProject #PewCharitableTrusts #GoverningMagazine #GradingTheStates #GradingTheCities #GradingTheCounties #StateandLocalManagementRanking #StateManagementEvaluation #StateandLocalHRManagement #StateandLocalInformationTechnologyManagement #StateandLocalGovernmentBudgeting #StateandLocalInfrastructureManagement #JournalistAcademicCollaboration #StateandLocalPerformanceManagementEvaluation #StateandLocalSurveyResearch #StateandLocalPerformanceMeasurement #BandGReport #BarrettandGreeneInc