inay Prasad has some
interesting
comments about NIH study sections. Yes, it's a Substack,
with all that implies, but he makes some good points.
Prasad says NIH is not willing to push the envelope. He writes: “We have made no progress in Alzheimer's disease in part because of the NIH's dogmatism.” Contrary to what you may hear on TV and in the science press, this is absolutely true. Those antibodies are not a viable treatment. As I've been saying for years, they're evidence that the beta-amyloid theory has hit a dead end. Bad NIH funding priorities are the primary reason.
Prasad also cites Ioannidis, a statistician who is a favorite of people who are mad at science, which seems to be everybody these days. Ioannidis says only researchers with papers that have at least 1000 citations should be on study sections. So, once again I find myself disagreeing with Ioannidis.
What value, if any, is there in citation analysis?
In citation analysis, you count up the number of papers somebody wrote and scale them by the “ranking” of the journal. (Or you count the number of times somebody else cited them, which is basically the same thing.) This is supposed to find the best scientists, but it does not work.
In my field, Alzheimer's disease, we had a saying about papers in the top journals like Science and Nature: they were snazzy, expensive, and wrong. What they had in common was vast armies of postdocs, equipment that average labs can't afford, and a novel technique. They had impressive skill, but their findings always—not just almost always, but always—turned out to be dead ends. To find the real advances, you had to look at “minor” journals. This was the only place novel ideas could get published.
Economists like to use citation analysis and similar techniques because they see them as nice, objective measures. But any such bean-counting method would divert all the funding to wealthy institutions like Harvard and basically kill off minor schools. Now, some, maybe most, of these schools need to die. I once worked at one such place and found it to be a rat's nest of vicious administrators and researchers who sabotaged each other's papers and grants. No one at the school trusted their own former students with their medical care. Most used the real hospital across town instead. Schools like that need to be shut down, the buildings razed to the ground, and the students enrolled in welding classes.
But if you want innovative research, the last place you'd fund is a big school like Harvard or Yale. They are bastions of establishment views that hold back progress by sucking up research dollars and using them to defend ideas that only survive because their originator is a well-funded big shot. These are the ones who write review papers stating the same evidence over and over again and finding fault with any conflicting idea. If that big shot gets to review your paper or grant, he will shoot it down if your finding disagrees with his. That is the only reason most of us cite them: not because we believe their findings, but because of the risk of not doing so.
Of course NIH's insane focus on diversity must end as well. Studies on “systemic racism” and social issues like discrimination against trans people, as important as they may seem to some, are not related to either science or public health.
How can we improve grant funding?
At the moment NIH, knowing that a wrecking ball might be coming, has paused the study sections. I've been on them many times. The last thing they need is more prima donnas, which is to say big shots with papers that have been cited 10,000 times because they're dogma. Prima donnas intimidate everyone else on the study section into agreeing with them. They think they're Einstein—and who would dare tell Einstein he's wrong?
The people at CSR (Center for Scientific Review) are well aware of this and make us watch a training video telling us not to fall victim to ‘hero worship.’ That training is routinely ignored.
The best panel members are the ones who are open-minded and who do the work. That's about 1/10 of them. You have to read the grant critically. Often, scientists are terrible writers, so a grant that looks like a dumb idea might be a vitally important one, and vice versa. Sometimes they forget to state their idea altogether. Maybe they're afraid somebody will steal it, or maybe they never noticed they had a good idea. It happens more than you might think.
You also have to be cynical: does the author actually believe what he's saying? You'd be surprised how many grants come in where the authors have already published something that proves their idea is wrong. What the authors are really looking for is just money, and they have no intention of doing what they propose in their grant.
Nine out of ten grant reviewers will never catch this because they look only at the five or ten grants they're assigned, each of which may take several days to evaluate. Those who take the time to check the citations and the authors' previous work, and who can think critically and constructively, are the most valuable members on the panel.
The big shots are always the worst. They're so accustomed to unreserved praise that they expected us to assume they plan to do proper controls and consider alternative interpretations. They were either too lazy to bother writing it down (as grant writers are required to do) or they didn't actually plan to do any controls. What's the point of controls, they seemed to be asking, when I'm always right? Put them on the panels and you'd be guaranteeing an old-boy network.
An even bigger problem is that everyone wants to follow the latest fad. At one time, that was creating new transgenic mice. Now, it's creating more sophisticated kinds of induced knockouts. That's fine if the experiment calls for it. It won't work in Alzheimer's disease because mice simply don't get the disease no matter how many human genes you give them. No matter how many times the press talk about “mice with Alzheimer's disease,” there is still no such thing.
I've defended many grants that wanted to try something innovative. Sometimes I succeeded, but often the panel went with the snazzy, expensive, and wrong stuff like what they saw in Science and Nature. They preferred the safety of the herd.
As Jay B. undoubtedly knows, when told to encourage innovation, NIH will probably react by raising “Innovation” to 50% of a grant score, tweaking the study section rules, and adding a few new “high risk” Program Announcements (PAs). These band-aids won't do much. The problem is not just at NIH. It is everywhere in academia. Here's what's needed:
Adopt Open Science across the board.
Abolish Programs. Ditch all PAs, Notices of Special Interest and similar programs. These programs crush innovation dead. Scientists, not CSR administrators, should be the ones to decide what is good science.
Fund researchers directly, eliminating the parasitical academic administrators who suppress innovation by using mindless bean-counting as a metric of quality.
Whatever changes are made, the Harvard guys will write scathing letters to Nature complaining about it. But the idea that they're better scientists because they get more citations is one hundred percent wrong. Rewarding them would be the worst possible solution.
jan 25 2025, 6:01 am. last updated jan 27 2025, 4:27 pm
