Christie Aschwanden wrote an article for Wired titled We’re All ‘P-Hacking’ Now.
« Suppose you’re testing a pill for high blood pressure, and you find that blood pressures did indeed drop among people who took the medicine. The p-value is the probability that you’d find blood pressure reductions at least as big as the ones you measured, even if the drug was a dud and didn’t work. A p-value of 0.05 means there’s only a 5 percent chance of that scenario. By convention, a p-value of less than 0.05 gives the researcher license to say that the drug produced “statistically significant” reductions in blood pressure. »
« P-hacking as a term came into use as psychology and some other fields of science were experiencing a kind of existential crisis. Seminal findings were failing to replicate. Absurd results (ESP is real!) were passing peer review at well-respected academic journals. »
« Psychologists Uri Simonsohn, Joseph Simmons, and Leif Nelson elegantly demonstrated the problem in what is now a classic paper. “False-Positive Psychology,” published in 2011, used well-accepted methods in the field to show that the act of listening to the Beatles song “When I’m Sixty-Four” could take a year and a half off someone’s age. »
David Amici wrote an article for BiteSizeBio titled Are you Guilty of P-Hacking?
« What is P-Hacking? The term p-hacking, coined in 2014 by Regina Nuzzo in Nature News, describes the conscious or subconscious manipulation of data in a way that produces a desired p-value. P-hacking is typically done through manipulation of “researcher degrees of freedom,” or the decisions made by the investigator. These include when to stop collecting data, whether or not your data will be transformed, which statistical tests (and parameters) will be used, and so on. By simply manipulating researcher degrees of freedom, even absolutely negative data can produce a p-value under 0.05 an incredible 61% of the time. A great and timely tool to visualize this is available at 538 Science. »
« One of the authors of the study that brought researcher degrees of freedom to the forefront, Uri Simonsohn, points out that p-values just under 0.05 are extremely over-represented in the published data. There’s no great reason for this to be the case; instead, he (and others) posit that this is evidence of how widespread hacking borderline p-values has become. »