An 'explosion' of formulaic research papers written with AI

ave you ever wondered why eating blueberries cures cancer one day and causes it the next? Why two hundred years after the first functional gas stove was built by James Sharp they were suddenly found to cause asthma when the government wanted to ban them? Now we know the answer: computers running AI are making it all up.

A new paper ^[1] claims that “AI-assisted workflows” are producing an ‘explosion’ of “formulaic research articles, including inappropriate study designs and false discoveries” published by so-called paper mills, which, according to the authors, include many top scientific journals. The authors write:

Employing AI-supported single-factor approaches removes context from research, fails to capture interactions, avoids false discovery correction, and is an approach that can easily be adopted by paper mills.

The articles use data from the National Health and Nutrition Examination Survey (NHANES), an AI-ready dataset for which automated tools have been written. ‘Depression’ was the most frequent condition studied, with 28 papers finding positive or negative statistical correlations between depression and blood pressure, ethylene oxide levels, PFAS, blood cadmium, ratio of non-HDL to HDL cholesterol, and many other things, all of which were calculated wrong. Some analyzed multifactorial conditions as single factor problems. Some were selectively extracting data to fit their flawed design. Some were harking, which means hypothesizing after the results are in, perhaps by “subsetting,” or dividing the popula­tion up in various ways until something becomes statistically significant.

They also didn't correct for False Discovery Rate, or FDR:

Of the 28 statistically significant associations, less than half (13) remained statistically significant after FDR correction.

FDR testing is necessary when you do many hundreds of correlations. Some will always come out significant just by chance, even if you set your significance level to p<0.001. Statisticians have devised many ways of correcting for this problem, FDR being one of the weakest and Bonferroni being more stringent. Of course the real responsibility lies with the authors, whose only goal is to get a paper. In the drive to publish as much as possible, researchers forget that a correlation alone is meaningless unless it's backed up by a solid, well established mechanism. Even then, the possibility of confounding factors remains.

It should be added that the papers could very well be correct. Maybe some or all of the problems aren't multifactorial problems but single-factor problems. Unfortunately, there's no way to know, and there's also the problem of confounding factors.

A classic example of that was the ozone hole, where a potential problem was identified, a plausible mechanism was found, and evidence was found that the proposed mechanism occurred. The authors got a Nobel prize and a treaty banning the bad molecule, and . . . 38 years later . . . the ozone hole is as big as ever, though you'll never hear that in the popular press. Clearly something else, remaining to be discovered, must also be contributing, but few people care: there's no percentage in looking for it. They're not going to give another Nobel Prize for proving the first one was a ‘whoopsie.’

Maaa, China is doing biology again . . .

The authors also find that between 2021 and 2024, many of these “formulaic” manuscripts (292 out of 316) come from authors in mainland China.

They cite specific examples. In PMID 39377074, a group from Shanghai claimed in Frontiers in Endocrinology to have found a correlation between body shape index and abdominal aortic calcification. In PMID 38840158 the same group linked it to cognitive impairment.

You don't need AI or to live in Shanghai to produce statistically flawed results, nor is nutrition the only field where people do it. There are thousands of such studies from every country in every field. The authors say FDR correction should be used in all data dredging studies because the number of potential hypotheses is enormous. The problem is that researchers always say they're doing exploratory research that's not intended to be the final answer but to find as many clues as possible. That's a valid concern.

Take chronic inflammation, which is now “implicated” in almost every disease you can think of, from cancer to heart disease to Alzheimer's disease. Is it a cause, a contributing factor, or merely a side-effect as unhealthy cells die, release their contents, and activate the DNA damage response? No one knows; they're all too busy finding correlations with air pollution, microplastics, PFAS, and BPA.

Polygenic risk is a good match for AI

Polygenic risk score (PRS) is a similar questionable technique. You take a large number of patients and healthy people, analyze their DNA, and pick the top 5% or so of the gene polymorphisms, called SNPs, that show a difference. This is called a GWAS, or genome-wide association study. It was invented as a tool for exploration, and it's been very useful.

As an exploratory tactic, it's reasonable. But it wasn't meant to be used in the clinic to diagnose cardio­vasc­ular disease ^[2] or mental health disorders ^[3], which according to the experts are “highly polygenic.” What, if anything, does this mean? In fact, it means almost nothing. PRS is basically a way of multiplying hundreds or thousands of hypotheses together. You then assume the product has some objective meaning and correlate it with something else, say political belief or number of blueberries eaten per week, thereby proving that people who eat blueberries are all insane or all likely to die in some horrible way. It doesn't distinguish cause and effect, it fails to consider the patient's environment, and it has no mechanistic basis. Its connec­tion to GWAS makes it appear scientific, but it's not. This is why it was rejected by proteom­ics researchers. It's also why a PRS based on a European database can't be used for cohorts of non-European ancestry.

PRS is only popular because it gets around the fact that we have no clue what causes the disease or—especially in mental health disorders—whether the disease entity is even real.

GWAS usually corrects for the FDR to address the problem of multiple correlations, but PRS does not.^[4] Anderson et al. write:

By aggregating all variant effects into one continuous metric, one can also skirt the strict multiple comparisons corrections necessary for GWAS. In other words, rather than correcting for millions of univariate statistical tests looking for hits across the genome, just one statistical test is required to regress genome-wide PRS onto the phenotype of interest. ^[3]

Even if the numbers were corrected for the FDR, it's a statistical shipwreck: how can 100,000 numbers with no individual connection to the disease sum up to something that's predictive? These weaknesses make it worthless for doing science. Even for clinical studies, there are doubts. Khanna et al. say

[I]t should not be the only tool to identify the risk in patients because it does not consider the environmental factors, lifestyle, and family history, which could be different for every individual.^[2]

A good example of how polygenic risk score is misused is in the study by Saarinen et al. ^[5], who correlated belief in magic with polygenic risk for schizophrenia, a disorder for which the cause is totally unknown. The existence of thousands of papers praising PRS doesn't change the fact that adding up a hundred thousand genes with no discernible connection to the disorder doesn't turn them into a diagnosis.

Bad problems create worse solutions

The risk is obvious: too many bad studies can wipe out confidence in all of science and lead to bad public policy. This has been going on for decades in many fields. It's why people still think they're eating too many eggs and need to stop eating meat, driving a car, and having children to save the planet. What's new is that AI makes it all sound scientific. The idea that the compu­ter is never wrong could kill us all.

The authors propose access barriers, rejection of publications that appear formulaic, more retractions based on post-publication criticism by PubPeer, and bureaucratic controls such as licensing. That's a great way to make the problem worse. PubPeer is particularly pernicious, as many of the critiques found there are founded on speculation and personal animosity.

The source of the problem is the demand in academia to publish as much as possible. There's also a lack of statistical training in grad school. When I was a student, they gave us none of it. I had to read many books on the subject, most of which were either dry, basic, or highly mathematical. Or they were from Iowa State in 1937 and talked mainly about corn.

Here's a better solution. Forbid companies from doing drug testing on their own product. Fund mechanism-based research over data dredging and insist that any correlation must have mechanistic support. Eliminate the practice of publica­tion-counting in academia. Make statistics courses mandatory in the STEM curriculum. And find some way to accommo­date explora­tory research, maybe by labeling it as such.

That's not as simple as pulling up the ladder behind you, as the authors seem to want, but it's what you'd need if you wanted to solve the problem. The hard part is convincing people the problem exists, and for that the authors deserve praise.

may 16 2025, 2:50 pm. updated may 17 2025, 5:14 am