Statistical thinking
Jun. 18th, 2016 11:12 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
flexagonFinal brain dump. On Friday I spent all day in a class called "Statistical Thinking", a non-math-focused (really!) overview of how to think like a statistician. I learned a ton, but my favorite part might have been the first part, the part that talked about decision-making and really got me thinking about very non-technical things. Heck, it might even be helping me think about politics. My work-friend H said it made her think about a tough management problem she's facing.
First, you have a default action. That's what you're going to do in the absence of any information. For example, my default action in an established romantic relationship is to continue it. My default action for a low-performing employee is to continue employing them.
Then you have a null hypothesis, which is basically the set of all worlds in which the default action is a happy thing to do. Null hypothesis: the relationship will provide a lot more good than bad in my life, going forward from now. Null hypothesis: we hired this person for a reason, and they can and will provide value over time in some environment within the company.
Then you have to think about significance, which is a stats term related to an easier question: "what would it take to make my null hypothesis look ridiculous?" You don't get to calculate this -- no data yet! -- it comes from experience, the mind and the heart, and may be deeply personal. Take that low performer at work, for instance -- I will never think that failing in a single group makes my null hypothesis look ridiculous. For me it takes two or more, where for others one is enough to convince them.
The more it takes to make your null hypothesis look ridiculous, the more extreme data you're going to need (either in quantity, or extremity of value). So you can, if you're doing actual data analysis, do some thinking about which kinds of errors you'd rather make, and collect your data and crunch the numbers to see what the data says. And the answer will always be uncertain. This gets mathy but is like life, so very like life.
Here's something nasty though. If you keep re-analyzing as data trickles in, you can be tricked into stopping an experiment too soon, as the conclusion winks in and out of what you've decided is "statistical significance". In real data analyses (drug trials, ahem) they can change the rules in a "no peeking" sort of way to make sure some large amount of data comes in before it's analyzed. And amazingly, I do this in some parts of real life: I'll give an uncomfortable-for-me work situation one year to work out, for instance. A promise I made to myself after 2008, and have kept. It's saved me both from panicking after 6 months of discomfort with a new director, and also from staying and stagnating after 12 with a not-right manager.
For the most part, though, it's impossible to keep from constant re-analysis in real life, and that's tricky. Super tricky. No wonder I had a rough winter, as romantic tribulations kept me right on the edge of "it is ridiculous to think this will work out well" for months. I did consider setting some timelines / limits, like "I won't stay in this if I'm still unhappy after six months", and I think the class says that sort of thing is a good idea.
Anyway, all of this seems like really fertile ground in understanding disagreements, because it gives me a new lens for looking at how people differ. I keep remembering old conversations in a new light. Different default actions can be remarkably hard to pick out, for instance, once people in an argument have leapt straight into discussing actual data. That means different null hypotheses might be in play, meaning different things to be proven... and there's a big big difference between someone looking for a reason to fire you and someone looking for a reason to keep you around. Big difference between people looking for reasons guns are helpful enough to keep and people looking for reasons guns are harmful enough to get rid of. And, of course, different amounts of proof people need before they're willing to say their default looks ridiculous can lead to different thresholds for action. And these different points are identifiable and can be talked about reasonably enough.
Yep, yep. Useful. I am going to keep this framework around in my head, whether I ever need to crunch another dataset or not.
First, you have a default action. That's what you're going to do in the absence of any information. For example, my default action in an established romantic relationship is to continue it. My default action for a low-performing employee is to continue employing them.
Then you have a null hypothesis, which is basically the set of all worlds in which the default action is a happy thing to do. Null hypothesis: the relationship will provide a lot more good than bad in my life, going forward from now. Null hypothesis: we hired this person for a reason, and they can and will provide value over time in some environment within the company.
Then you have to think about significance, which is a stats term related to an easier question: "what would it take to make my null hypothesis look ridiculous?" You don't get to calculate this -- no data yet! -- it comes from experience, the mind and the heart, and may be deeply personal. Take that low performer at work, for instance -- I will never think that failing in a single group makes my null hypothesis look ridiculous. For me it takes two or more, where for others one is enough to convince them.
The more it takes to make your null hypothesis look ridiculous, the more extreme data you're going to need (either in quantity, or extremity of value). So you can, if you're doing actual data analysis, do some thinking about which kinds of errors you'd rather make, and collect your data and crunch the numbers to see what the data says. And the answer will always be uncertain. This gets mathy but is like life, so very like life.
Here's something nasty though. If you keep re-analyzing as data trickles in, you can be tricked into stopping an experiment too soon, as the conclusion winks in and out of what you've decided is "statistical significance". In real data analyses (drug trials, ahem) they can change the rules in a "no peeking" sort of way to make sure some large amount of data comes in before it's analyzed. And amazingly, I do this in some parts of real life: I'll give an uncomfortable-for-me work situation one year to work out, for instance. A promise I made to myself after 2008, and have kept. It's saved me both from panicking after 6 months of discomfort with a new director, and also from staying and stagnating after 12 with a not-right manager.
For the most part, though, it's impossible to keep from constant re-analysis in real life, and that's tricky. Super tricky. No wonder I had a rough winter, as romantic tribulations kept me right on the edge of "it is ridiculous to think this will work out well" for months. I did consider setting some timelines / limits, like "I won't stay in this if I'm still unhappy after six months", and I think the class says that sort of thing is a good idea.
Anyway, all of this seems like really fertile ground in understanding disagreements, because it gives me a new lens for looking at how people differ. I keep remembering old conversations in a new light. Different default actions can be remarkably hard to pick out, for instance, once people in an argument have leapt straight into discussing actual data. That means different null hypotheses might be in play, meaning different things to be proven... and there's a big big difference between someone looking for a reason to fire you and someone looking for a reason to keep you around. Big difference between people looking for reasons guns are helpful enough to keep and people looking for reasons guns are harmful enough to get rid of. And, of course, different amounts of proof people need before they're willing to say their default looks ridiculous can lead to different thresholds for action. And these different points are identifiable and can be talked about reasonably enough.
Yep, yep. Useful. I am going to keep this framework around in my head, whether I ever need to crunch another dataset or not.
no subject
Date: 2016-06-19 06:19 pm (UTC)