Science has a trustworthiness problem. Public trust in science, scientists, and in the worthiness of scientific research for society, has been on a steady decline since 2019, according to Pew Research Center.
To be frank, “science” is lucky its trustworthiness problem is not worse, because the public has long been unaware just how deep the rot goes. Take the mordantly named scientific “literature.” In 2022, U.S.scientists published roughly 460,000 scientific papers—Chinese scientists published roughly 900,000. Yet, an alarming proportion of those papers are never read, do not produce reliable results, and have no measurable effect on our scientific understanding of the world. If the public —who funds the bulk of scientific research—knew the full extent of this, trust in science would decline roughly to the level of those master grifters, members of Congress.
The foremost question for anyone who values science should be how to make it trustworthy again. There is no shortage of ideas. Early in its term, for example, the Biden administration set up new procedures for reporting and adjudicating scientific misconduct. How are they working out? Like any bureaucratic program, government oversight of scientific misconduct is proving to be rife with incompetence, cluelessness, inefficiency, and waste, coming nowhere close to tackling the likely scope of the problem. Like all mindless bureaucracies, government oversight of science is also prone to brute-force solutions like criminalizing scientific misconduct.
On their part, scientists have been making efforts at self-policing.
There has arisen a cottage industry, for example, of independent researchers taking the role of sniffing out scientific misconduct upon themselves. These have been notable mainly for destroying careers over seemingly small shortcomings in published work, as in the defenestration of Stanford president Marc Tessier-Lavigne.
More positively, many scientists post “preprints” on sites like bioRXive prior to submission to a journal, adding a layer of peer scrutiny of papers before they get into print in the hope that problems can be unearthed and corrected quickly. More formal efforts like The Open Science Society promote reforms such as preregistration of hypotheses prior to an experiment to guard against post-hoc adjustment of hypotheses to fit the data.
Looming large in all these discussions is the “irreproducibility crisis”, which can be summarized thus: if reproducibility of experiments is the gold standard of scientific inquiry, why is there so little of it?
Nearly twenty years ago, John Ioannidis argued that common flaws of experimental design, sampling protocols, and poor analysis make most published research impossible to replicate, and this is why, in his words, “most published research findings are false.”
Ioannidis’s assertion certainly put the cat amongst the pigeons. Nature magazine devoted a special issue to the problem, which included several mea culpas: yes, we have sinned and probably we could do better. Ecologists have chimed in to admit that “82-89% of ecological research … has limited or no use to the end user.” OK, good to know, taxpayers might say. Not to worry, taxpayers, henceforth we should embrace preregistration to “reduce research waste.” Not so fast, say others. What’s the best use of our time and resources: replicating garbage results or tending to our work? Still, others have asserted that the irreproducibility crisis is no big deal, and attention to it can be a diversion from performing “robust and efficient science.”
Psychology and behavioral sciences have come down solidly—perhaps defensively—in favor of replication and pre-registration, and—reproducible?—data back them up. Replication is better, for example than the common alternative of “meta-analysis,” which takes many studies and applies statistical magic dust to tease out something sensible. The data say so! A recent article in Nature Human Behaviour showed that due attention to replication and pre-registration can actually produce replicable and robust findings. Phew! “What reproducibility crisis?,” crowed a news article in Nature reporting on the article’s findings. The celebration was premature. About a year after the paper was published, NHB retracted the article, citing a “lack of transparency and misstatement of the hypotheses and predictions [the study] was designed to test,” as well as shortcomings in preregistration: all the shortcomings that plague the scientific literature at large, in other words.
What went wrong?
Stephanie Lee perceptively explored this question in The Chronicle of Higher Education, detailing how even the best of intentions – promoting transparency and reproducibility in scientific research – quickly degenerated into prejudice, misinterpretation, crossed signals, befuddlement, and fear of retaliation. Shucks, scientists are flawed human beings, in other words, just like the rest of us. Scientists can sometimes be “knuckleheads,” to use a current term of deflection. Hardly adequate to address science’s trustworthiness problem.
The uncomfortable truth here is this: the scientific literature is filled with irrelevant dreck. Most of it churns below the surface out of sight, and sometimes it rises to outright fraud and deliberate misrepresentation. This is not due to some failure of policy or procedure. Rather, the literature is filled with dreck because we extravagantly subsidize its production. Nearly $90 billion is shoveled into academic research annually, the majority of it from the government. Since 1953, public funding for science began to take off, and governments have poured roughly a trillion dollars into research. These unprecedented expenditures have not produced more science, as they were intended to do. What they have done, rather, is fundamentally transformed the culture of science. Stephen Turner and Darryl Chubin put it this way in their provocative essay, “The Changing Temptations of Science.” Whereas science had once been dominated by an ethic of discovery, it is now governed by an ethic of production, much of it driven by intense competition for government money. Scientists are now judged by numbers: numbers of papers published, numbers of students graduated, numbers of grant monies brought in. As measures of production have come to dominate scientific careers, new knowledge has become largely irrelevant to scientists’ careers. No wonder science is in crisis.
If science has an integrity—replicability, reliability, truthfulness, fill in blank—crisis, it will not be solved by ever more stringent policies and procedures. It can only be solved by addressing the perverse incentives built into the “ethic of productivity” problem. The solution will be painful: the entire science ecosystem, from universities to funding agencies and to Congress to the academic publishing industry, everyone is in on the grift. Taking away the network of perverse incentives will mean taking away the money incentivizing the production of dreck. Everything else is chipping away at the margins.
Image by MarekPhotoDesign.com — Adobe Stock — Asset ID#: 387186655
This article was originally published at www.mindingthecampus.org
