Never Trust the Science - On the need to identify bias & interpret data yourself

afpx

How about a big vetted database like arxiv of all hypotheses, all proposed experiments to test them, and all experimental results?

RcouF1uZ4gsC

And if a result is surprising to you, you should trust it less and look into it more deeply. And if you do so, one of two good outcomes will hopefully happen: 1. You find the result is bogus 2. You learn something new and update your internal model of the world.

like_any_other

> For example, failing to control for obvious confounders in observational data is likely to produce biased results. If we like the direction of this bias, we can do less adjustment for confounders. For example, the study showing that having a white doctor increased mortality of black babies didn't correct for birth weight - once that was done, the effect disappeared (and media interest waned): https://www.wsj.com/opinion/justice-jacksons-incredible-stat...

_aavaa_

> Finally, if you want to simply know which Science™ you can trust, I’d recommend finding and following individuals who repeatedly demonstrate competence in statistical methods and scientific interpretation. So like, the scientists themselves? > If in doubt, read the study critically yourself. I cannot believe the author manages to say this with a straight face. “Hey you average person (with maybe a college degree), go read the original academic paper yourself. Doesn’t matter that you don’t have the background, struggle with basic math (much less statistics), can’t evaluate the claims, and don’t know which questions to ask.” The age of the polymath is long dead, we’re living in The Great Endarkenment. You trust your pilot to do their job, you trust your civil engineers with the bridge you driver over, and the mechanical engineers with the controlled explosions happening in your car, but when it comes to cutting edge scientific articles, here is where you, average Joe, will be able to know better that the experts in the field who specialized in this and do it every day.

thephyber

This assumes that everyone is currently sufficiently informed enough to make the same expert observations about methodology affecting bias. This is flatly untrue for the vast majority of the population. And nobody has enough time or desire (or likely money to subscribe to the journals) to read the details of the papers and grok the nuances. Humans think in simple narratives for a reason. We shouldn’t have blind faith in science, but we also shouldn’t have to go back to first principles and do our own version of every experiment. The repeatability crisis is a thing and we know about it. P value hacking is a thing we know about. The problem described in the article is that we shouldn’t believe headlines or short summaries created by writers who aren’t incentivized to add the nuance. And nobody should believe a headline anyway - in addition to necessarily being lossy, for any for profit organization they are likely written by someone other than the writer and probably A/B tested for clicks.