BEWARE SELECTION BIAS

We’ve heard repeatedly about the frustrations felt by leaders in towns, cities, and counties that regularly hold public meetings to get resident input only to find that the opinions voiced frequently come from the same small portion of the population that has time and interest to show up at meetings – often called by civic leaders, the “usual gang of suspects”.

This can often leave out people who are working multiple jobs; parents who can’t afford the expense of childcare; portions of the older population for whom travel can be difficult and others.

Even when the meetings are held online, there are plenty of folks who aren’t comfortable expressing opinions on a screen – often without getting any sense of reaction to their comments.

This is all part of a broader phenomenon confronted by communities that we’ve been thinking a lot about lately: selection bias.

One way this phenomenon exhibits itself is when states or local governments send out surveys to find out how well they’ve been doing in the eyes of the public. It’s our belief that many of the people who respond are either very happy or full of misery they want to get off their chests. Residents in the middle aren’t nearly as incentivized to speak up, and that can lead to misleading conclusions.

Though this doesn’t have anything to do with cities and states, we ourselves had such an experience recently. Often when we go to conferences, we’ll get a survey form asking what we thought of the hotel at which we stayed. And because we barely have enough time to get our work done, we are often lax at filling them out. But recently the hotel was one of the least satisfactory we can remember. We won’t go down our litany of complaints here, but we were plenty unhappy. Suddenly, we had the time to fill out the form, packed full of specific gripes.

We doubt that the hotel took much notice of our form (at least we didn’t hear back from them with any kind of apology), but cities, counties and states can take this kind of feedback pretty seriously, and can be misled to believe that they’re working in a world in which nothing they do is ever appropriately appreciated.

We’re also concerned about the conclusions reached by membership organizations, think tanks or individual public sector entities that want to get input from other places in order to guide the work they’re undertaking. Often, in that case, the input received is weighted in favor of those respondents who want to tell a positive story.  

We’ve been through this ourselves when we’ve reached out to a wide swath of similar agencies across the states or a cross section of cities or counties around the country. Even when we promise not to identify respondents by name, individuals with a less promising story to tell are also less likely to reply.  If we aren’t careful, that can too easily leave us with an unreasonably positive conclusion.

When we’ve been at conferences recently, we worry that academics have a similar issue. If their surveys address promising practices, the entities that respond are more inclined to do so if they’ve got a happy story to tell.

And here’s another problem. Data requests generate responses from places that have the data in the first place. This rules out many towns, cities and counties that lack the resources or staff capacity to gather reliable data.

There’s a kind of paradox here. Much of the policy that wends its way through county administrators, city councilmembers or boards of commissioners, is intended to help places most in need of remediation. But places with lower capacity can’t effectively get their two cents in. The result? Policy solutions may be less helpful to places that are in most need of help.

One more concern – and this is of relatively recent vintage: With a growing federal emphasis on catching and exporting men, women and children who aren’t deemed worthy of domicile in this country, you can hardly expect them to fill out any kind of form, even when they have every legal right to live where they’re living. This kind of chilling effect on individuals who want to avoid attention, easily winds up providing public leaders with misleading information and weak guidance on a host of critical policy issues..

That, we think, may be the saddest part of this whole story.

#StateandLocalGovernmentManagement #StateandLocalManagement #StateandLocalPerformance #CityGovernmentManagement #CityGovernmentPerformance #CountyGovernmentManagement #CountyGovernmentPerformance #StateandLocalPerformanceManagement #StateandLocalPerformanceMeasurement #StateandLocalGovernmentData #CityGovernmentData #CountyGovernmentData #PublicSectorMissingData #StateandLocalMissingData #SurveySelectionBias #AcademicSurveyResearch #StateandLocalSurveyResearch #StateandLocalSelectionBias #PublicHearingAttendanceBias #StateandLocalResidentOutreach #StateandLocalCitizenOutreach #BandGReport #BarrettandGreeneInc