Episode 38: Lies, Damn Lies, and Statistics

Transcript

But there are very few skills that I've picked up along the way that have been as pivotal for me as understanding some baseline things about statistics.
More specifically, or maybe more to the point, it's not just statistics, it's sort of the point of intersection between statistics and experimental design and the reporting out thereof.
By reporting, I mean the sort of traditional example would be someone's watching the news at night or whatever, and they say, oh, this study proves such and such.
So it's an intersection between those things.
It's understanding sort of some basic things about statistics in combination with understanding how a good, or in some cases not good, scientific experiment or study or survey or whatever, how that's put together.
And then in combination with how people go about conveying this information to the public, you take those three things, understanding that intersection point, I have found to be just a, one of the most formative and important things that I learned along the way.
And one of the, I use it probably daily, but at least weekly, the understanding of these pieces.
Now I should say, I, I, I, I do have more understanding and knowledge here than just a basic level.
Like I did go through quite a bit of statistics work, behavioral analytics type of work, business analytics kind of work.
I have throughout my career had multiple projects and things where I had to apply this information, even though it wasn't my primary job.
So like I do have more background here than just baseline, but I think my point here mostly is that if more people just had a, at least a baseline of some fundamental basic things that it would go a long way towards a better understanding of a better understanding for people of what's going on out there in the world.
And what not just scientific things are going on, although that's a part of it, but also social things, political things, having some basis in understanding these pieces can really help people analyze the world around them, I suppose.
And also evaluate the information that is being constantly shoveled at, this is actually the second or third time that I've recorded some version of this episode.
I did it before and I scrapped it because I, it, it, it, it didn't quite get across.
I think I didn't spend enough time initially narrowing down the exact point I was trying to make.
So this episode, I want to kind of make one specific point about all of this in the future.
I think what I might do is another episode that goes through maybe three or four fundamental sort of statistical concepts, ones that are not, not terribly complicated, but that are good to know about in order to, you know, just have that kind of baseline knowledge.
But for this episode, what I realized in recording kind of the first draft of this was that I was really harping on one thing in particular.
And what that tells me is that that's really the point I'm trying to get across here.
So I kind of scrapped that and doing this again.
Most people are not actively engaged with any kind of regular scientific community.
The scientific community predominantly exists within journals, peer reviewed where other science, like they, you know, you're a biologist, you publish some biology research, other biologists look at it, they critique it.
It's sort of like, it's sort of like, in some cases, it's sort of like the origination of where like internet flame wars came from.
But, you know, assuming the discourse is civil, they critique one of those work.
That's how science gets pushed forward.
It gets pushed forward in baby steps, generally speaking, where small incremental steps makes that make things better and better over time or increases understanding over time or whatever.
Most people are not engaged with that.
I don't know many people who get a true scientific journal each month delivered to them in some way and then spend time reading it.
This goes for me as well.
I don't want to give the impression like I'm sitting around reading scientific journals.
There was a time in my life that I did some of that, but that time has long since passed.
And that is not how I typically absorb information anymore.
Instead, what most people have is some combination of media barking at them all the time, right?
Be it traditional news stations or, you know, TV advertisements or TV shows or TikTok reels or YouTube reels or Instagram stories or whatever.
And there's all this information being bombarded onto people all the time.
And a good chunk of that information often comes down to some variation of studies show that blank.
Or, you know, some sort of statement around a new discovery or a new study or a new research paradigm or whatever that has suddenly shown some new dramatic thing.
The most fundamental thing that a person can come to understand about all this, in my opinion, is that as soon as you hear the word proves, your spidey sense should start tingling.
Your ears should perk up and you should be thinking very critically about every single thing that is said after that moment.
The word proves 99.9999999% of the time is not accurate.
And that's because science and all these things really stem out of some either true version or bastardized version of a scientific method.
All these surveys, all these studies, the purpose of science is to disprove things, not prove things.
So the idea is that if something happens, if there is some effect in the world, and there are 20 possible explanations, the job of science is actually to disprove 19 of those.
It is not to prove the last one.
By definition, when you study this stuff, you can't prove anything, not really.
What you do instead is lend support to something.
You can show support in favor of a particular conclusion.
That's done by disproving other things behind the scenes, but it is a factual thing, so this is the distinction.
If you hear someone reporting out saying new study proves, you should probably call bullshit on that every day and twice on Sunday.
However, if you hear someone reporting saying a new study supports the idea that, or a new study lends support to, or a new study, you know, again, supports, like supporting evidence, gives evidence to, those are good phrasings.
And that means that the source that you are listening to, or reading, or hearing, or whatever, that source has a better understanding, or maybe the writers have a better understanding, but someone in that stack of people has a better understanding of how these studies work, and what these studies actually mean, compared to someone who's running around saying, a new study proves this, and a new study proves that.
Now, I'm totally open to the idea that maybe they say prove, but they mean supports, and they just, you know, don't know any better, and that's fine.
But it is very, it becomes a vicious cycle when you have people reporting on studies, routinely putting out in there in the world, saying that they prove things when they don't.
In order to prove something, that line does get crossed eventually.
Like, there are certain things, the scientific community is willing to come together and say, we have proved that blank.
But that happens after not tens, not hundreds, but usually thousands of studies that repeatedly show the same sorts of results, that repeatedly disprove the same sets of alternative ideas.
That just repeatedly, over and over and over and over again, over the course of decades, show the same supporting evidence for the same idea over and over and over again, until there is simply nothing left, no plausible deniability left.
And the community can kind of group up and say, we have proven this.
That is a thing.
It happens eventually.
But whatever the latest story is that you saw on your TikTok reel probably didn't fit that bill.
The odds that that study, whatever it is you're seeing, is the final keystone or the final straw that has broken the camel's back and actually proven something, very, very, very infinitesimally small.
It's just a mindset shift that I think is really powerful, is this idea of moving your mindset from thinking that things have been proven to understanding that what has happened is that there has been support lended in that direction.
And I'm not going into anything here about what are the, you know, what possible experimental design flaws were there.
Was the study representative sample, was the sampling any good, are they general overgeneralizing their conclusions?
Are they interpreting the results correctly?
I'm not talking about it.
I'm going to assume all that away.
All I'm really talking about is this one key piece of next time you're hearing a story on something or you're reading about something and it says the word prove or proves or proof, that should perk you up.
It should perk your ears up and make you look real critically at whatever it is this person is going to say from here on out.
Because odds are their reporting is either designed to be sensationalized in some way or they just don't really know how to report on this kind of information.
So small terminology piece, very, very substantial mindset shift.
Again, there's some other things that I think are really worth diving into on these topics.
I'll save that for another episode, but just pay attention to this next time, next time you hear it.
And the more you become aware of it, it's sort of one of those things where once you see it, you can't unsee it, that kind of thing.
It's kind of like that.
Once you start hearing it and you internalize this sort of thing, you start questioning an awful lot of what gets reported because so much of it at the very least is being misreported and also very likely has some pretty substantial design flaws under the hood.
But at the very least, whatever it is that they're claiming has been proven probably has not been.