dan

As noted in this post, our regional group of regulators and municipal stormwater permittees is moving away from quantifying trash loads and trash reductions.

That’s a good thing, but we seem to be stuck with assessing success by documenting outputs (for example, frequency of street sweeping, or portion of the drainage system equipped with capture devices) as well as documenting outcomes (for example, less trash on streets or in creeks).

Documenting outcomes is hard, and results are uncertain. A municipal permittee could try like hell to clean up the trash but trash could still increase because of factors beyond its control (for example, a demographic change, or windier weather). Or vice versa: There could be less trash over time, but for reasons that have nothing to do with the local trash reduction program. A municipal permittee’s compliance shouldn’t be subject to vagaries.

But the assessment of success shouldn’t be about compliance. It should be about continuous improvement.

To solve a problem iteratively, you need to first guess a solution. Then you need a way to tell whether you are off, and in what direction. Measuring outcomes at least gives you a chance to figure out whether what works and what doesn’t. You might give yourself a bum steer (because of vagaries), but you can at least try to consider the context and make sense of the results. In contrast, measuring outputs shows that you tried, but there’s no way it can help make your efforts more effective.

How do we measure success?

In a meeting this past week, a group of municipal staff, Regional Water Board staff, and consultants reviewed a framework for local trash reduction plans. Each of 75-odd Bay Area cities, towns, and county governments will need to prepare these plans to reduce and eliminate&#8212by 2024&#8212trash in stormwater.

Back in 2009, most of these same folks worked on requirements for interim (short-term) plans to reduce trash by 40% by 2014. Those plans have been in effect for 3 years. What have we learned (or not learned)?

Here comes the crazy: We all know those numbers can’t be met. They certainly can’t be met in the places with the toughest trash problems, and not in the midst of a recession and financial crisis.

We could make some progress, though. We could make a difference, if we marshaled what resources we have and agreed to learn as we go along.
Even that is hard, because we also need to uphold the fiction of those numbers. For Water Board staff those numbers are a bulwark from political and legal pressures brought by environmental advocates and by other regulators. For local governments, the numbers are protection from enforcement actions and lawsuits.

Beginning in 2009, we created &#8220baseline&#8221 estimates of existing trash and ways to credit activities&#8212like street sweeping, public education, and creek cleanups&#8212toward the 40% goal. Recently we’ve agreed it didn’t work because the estimates are too imprecise.

But that’s not the half of it. There’s also a conceptual error here, one that goes to the core of what scientific understanding is all about.
Things are similar; this makes science possible. Things are different; this makes science necessary. (Paraphrasing from here.)

Urban trash is similar and different. Trash in storm drains is correlated to land use and to average household income. Sweeping streets more frequently does, ceteris paribus, pick up more trash. But the correlations aren’t strong enough to be the basis for directing local cleanup efforts. You’d be taking shots in the dark.

Generalizing from particulars is fine. But assuming any and all particular instances conform to the generality? Unless the correlations are very tight&#8212something that doesn’t happen much in nature or in stormwater&#8212that practice will yield wrong results much of the time.

Wrong as in creating perverse incentives to sweep streets that are already clean, to install capture devices where trash isn’t, and to overlook obvious trash sources that weren’t anticipated and weren’t assigned &#8220credits&#8221.