We’ve been watching city, state and county efforts at performance reporting for many years, and we know it’s hard to do. We applaud the governments that have moved forward aggressively in publicizing the results of programs -- whether they are good or bad.
But even though we're fans of performance reporting, and the courage it takes to do it, that doesn't mean it's unproblematic.
We bring this up because we just read an investigative report that was issued by Douglas Hoffer, the Vermont State Auditor, about the Vermont Annual Outcome Report. At the bottom of this blog post, you'll see a short video in which Hoffer addresses his complaints with his state's outcomes report.
There are quite a few of them.
Like us, the auditor is a fan of performance reporting. The first sentence in this 25-page critique notes that performance measurement when done well, “is a powerful tool that leads to evidence-based decision making, better program management, and greater accountability.”
We don’t agree with all of his criticisms, but in our opinion, most of them have merit. In fact, a number of the problems he points out reflect the “don’ts" side of the recent ”Dos and Don’ts” paper we co-authored with the Urban Institute’s Harry Hatry and Batia Katz, and several other very knowledgeable contributors who have spent years studying this topic, including Maria Aristigueta of the University of Delaware, Don Moynihan
Of Georgetown University and Kathy Newcomer of The George Washington University. (See our favorite, do’s and don’ts in our February 16 blog post.)
Here are a few of the problems Auditor Hoffer points to:
Vague goals. The auditor worries that the goals that drive Vermont’s performance outcome reporting are too broad and undefined. For example, one goal is that “Vermont has a prosperous economy.” But it’s not clear what that means or how it will be measured.
Mysterious targets. In contrast to the broad goal for the economy, Vermont’s outcome report has targets that leave the reader with questions about how they were established. For example, the outcome report tracks the fall-related death rate and notes that the target is 116.9 per 100,000 adults. “It is not clear how this target was set up or what achieving the target means,” the investigative report notes. There is also no explanation for a 10% reduction in the target that was established in 2009.
Lack of definitions, explanation or context. The report says that 93% of Vermont is covered by “state-of-the-art” telecommunications infrastructure. Yet, Hoffer points out that Vermont’s cellular and broadband coverage needs significant expansion and improvement and that it is “difficult to know what is measured by the indicator.” As an aside, the lack of solid and consistent definitions is a prime culprit in the problem of poor state data quality.
There are other problems, and we encourage you to watch the video, which is not the most elegant, but makes many very good points.
#PerformanceReporting #Audit #Auditor #AuditReport #Performance #Vermont #PerformanceMeasures #Outcome
Comments