in , , ,

How Should We Critique Research?

Scientific and statistical research must be read with a critical eye to understand how credible the claims are. The Reproducibility Crisis and the growth of meta-science have demonstrated that much research is of low quality and often false. But there are so many possible things any given study could be criticized for, falling short of an unobtainable ideal, that it becomes unclear which possible criticism is important and they may degenerate into mere rhetoric. How do we separate fatal flaws from unfortunate caveats from specious quibbling?

I think that what makes a criticism important is how much it could change a result if corrected and how much that would then change our decisions or actions: to what extent it is a “difference which makes a difference”. This is why issues of causal inference or biases yielding overestimates are universally important, because a ‘causal’ effect turning out to be zero effect will change almost all decisions based on such research, but other issues like measurement error or distributions, which are equally common, are often not important as they may yield much smaller changes in conclusions and hence decisions.

If we regularly ask whether a criticism would make this kind of difference, it will be clearer which ones are important criticisms, and which ones risk being rhetorical distractions and obstructing meaningful evaluation of research.

What do you think?

17829 points

Posted by Copperz


Leave a Reply





What Historical Figures Really Looked Like Will Change How You Perceive The Past

IQ rates are dropping in many developed countries and that doesn't bode well for humanity