Lies, Damn Lies, and Statistics

I was a political science major in the late 1980s. No one knew it at the time, but this was the last days of the Cold War so there were a lot of classes -- a lot of classes -- focused on the various aspects of US-Soviet relations. In one such class I heard a story (possibly apocryphal) that has stuck with me:

The prof claimed that back in the day the Soviet government had used two different methods to drive the production of nails in the Soviet Union. When they rewarded nail production by count, the Soviet citizens got lots and lots and lots of little tiny nails. But when the government switched the production metric to weight, the tiny nails disappeared and the citizens got lots of big, giant spikes instead.

The purpose of the story was to illustrate the benefits of the market economy over a communist/socialist economy. But looking back now, I think the prof actually sold the story short. It's not about capitalist versus communist, it's about the power of metrics. I've now spent nearly twenty years in a career that involves working with significant numbers of operational metrics, and based on that experience I've learned two rules:

First, metrics have power. Counting, and establishing a system of rewards or punishments based on the selected counts, will drive behavior, no matter how intelligent -- or stupid -- the identified metrics may be.

Second, it's hard to develop a metric that can't be subverted.

Right or wrong, healthcare is now a metric driven endeavor, and has been for a very long time. As a result, and consistent with my rules, you don't have to hang out with healthcare providers for long to hear lots of stories about efforts to subvert the metrics. My favorite (weird word, but...) example of this was a story (again, possibly apocryphal) I heard of surgeons wheeling dying patients out of the operating room and into the hallway when a surgery went sideways as, apparently, whether the patient died in the operating room had significant impact on the metrics applied to the surgeon. Do I know it happened? No. Do I believe it could, or probably did? Absolutely. 

I was thinking a lot about this after my visit with the SCCA. In cancer care, one of the primary metrics for evaluating everything is the five year survival rate. It doesn't take long Googling cancer to find examples survival curves like the one I inserted above. In that example, the conclusion we're all supposed to jump to is that the organization or provider or protocol represented by the green line is providing better care than that of the yellow. 

Maybe, but providing better care is just one way to get to that result. The other way is to limit the number of truly unhealthy patients the green line takes on. If I'm the green line, and I can figure out a way to send the sickest patients to whatever the yellow line represents, I'll get the same result.* And frankly, it's the easier approach. I don't have to do anything different than anybody else is doing, I just have to figure out a way to keep the healthier patient with me (like, perhaps, by offering large, luxurious waiting rooms with stunning views of Lake Union -- just sayin'...). 

I could, of course, be wrong. It could be that the possible impact I might have on the SCCA survival curve never once entered into anyone's calculation of whether I should be taken on as a patient. It could be that there were lots of other reasons focused on my best interests that drove the decisions that were made. It's certainly possible.

But the thing about me is I'm a cynical person. Moreover, my forty-six years on this planet have taught me that my problem is rarely that I'm too cynical; more frequently, it's that I haven't been cynical enough.

This, more than anything, has me wondering if that two-year survival median isn't going to be something of a stretch goal for me. 

Damn statistics.


* Oddly enough, it was Oncologist #2 that inadvertently reminded me of this. In explaining to Sib4 why the Sloan-Kettering trial was less promising than the results suggested, he pointed out that the folks in the trial had to be healthy enough, and wealthy enough, to travel to New York and establish a life there while they underwent treatment. Thus, he argued, at least some portion of the positive results had to be credited to selection bias.