ast week a colleague sent me a copy of a forensic image report that some
scientific journal had sent to them. It was remarkable in two ways: one,
the journal accused my colleague of falsifying every single image in the
paper, and two, every single allegation was provably false.
As is the current fad, the issue was with the Western blots. In one case, the analyst had tried and failed to line up two images, not realizing that they were taken on successive days and so they needed to be rotated. Once I rotated them by four degrees, they lined up perfectly. The whole document was full of beginner errors like this, and I had no trouble shredding the entire allegation.
On another image, the analyst picked bands at random, saying for instance that band 1 is similar to band 4 if you flip it, rotate it by one percent, and stretch it by ten percent. I found that, indeed, if you take your glasses off and squint, you might see some similarity. If you rotate it, you would also notice that it looks just like that black cat lying on the carpet in the next room.
It was actually a bit shocking to see a reputable scientific journal produce such a document. It was composed almost entirely of subjective judgements unsubstantiated by any kind of quantitative analysis. It also had a mocking tone that is becoming increasingly common in science. No doubt the writer was acting on the assumption that using logic, statistics, and scientific arguments was an unnecessary luxury in the noble pursuit of perpetrators of scientific malpractice. This was the sort of thing one is used to seeing in political blogs, but is now seeping into science.
Watershed segmentation is just one of many powerful tools that forensic image analysts could use. But creating pretty pictures is just the first step
The analyst kept referring to his previous shaky conclusions as proof that the current conclusion must also be valid: a type of argument to authority, with the previous paragraph assuming the role of the authority: Since I just said this, therefore that and the other thing must also be true.
My colleagues knew so little about image analysis that they had no clue what to do. They asked me for help, but they had done themselves no favors by submitting their own attempts to use NIH's OSI tools to refute the allegations, along with some new images that, indeed, looked fraudulent even to me. By doing this they made my job a lot harder.
There's math here. At least we could pretend it's boring
One can't draw conclusions about a whole field from one example, but I've seen other analyses that were just as badly flawed. I can't help wondering how many other papers have fallen prey to this new fad.
It's a classic moral panic: no one wants to take the side of a possible liar, and after two years of listening to Anthony Fauci the country is understandably fed up with scientists, real or otherwise. But making false accusations is also an injustice.
Serious allegations such as an accusation of fraud—which is, or at least should be, legally punishable, as the perpetrator has used it to benefit financially as well as reputationally—cannot be made on a subjective basis or on the basis of the history of the authors' former collaborators, as unsavory as they may be, but must be backed up by statistical analysis that is at least as rigorous as the results being questioned. Entire careers are being destroyed by this sort of incompetent analysis. It's not just the person accused whose career is destroyed: at some institutions anyone who ever collaborated with them loses their job as well.
Yet this reputable journal did not appear to be in the least embarrassed by that. After the reviewer finished snarking on every image in the paper, he (or maybe she) took upon himself or herself the duty to recommend the journal's course of action, apparently not realizing that the editors were not, I hope, actually asking for such advice.
In a law court the role of the expert witness and of the jury is strictly constrained to making determinations of fact. By exceeding that role, the analyst destroyed his or her credibility.
It's not as if this is a new field and analysts are struggling to figure out how to critique images. There are dozens of good books on the topic, such as Digital Image Forensics by Roy et al., Generalized Principal Component Analysis by Vidal et al (admittedly not a page-turner), which I'm reading now, and dozens of textbooks on multivariate statistical analysis, which is readily applied to two-dimensional images. Hell, they could even read Statistical Shape and Deformation Analysis by Zheng, Li, and Székely, which focuses on the much harder problem of analyzing three-dimensional images. We two-dimensional thinkers have it easy.
The point is that math and statistics were invented just for this purpose: to put our conclusions on a sound theoretical basis so that we can avoid having to rely on subjective feelings.
There were, to be sure, many weaknesses in the paper, but they were not attempts at deception so much as convincing evidence of an inadequate understanding about the capabilities and limitations of the equipment, much of it old and obsolete, to which the authors had access.
I discovered, for instance, that a new, still-under-warranty digital microscope for which they had paid US$ 35,000 was actually creating 8-bit indexed-color images. If you told the microscope you were using a blue filter it would set all the pixels to blue and set the other two channels to solid black, regardless of the actual colors in the sample. A simple histogram showed that despite the inflated claims on the vendor's website, it was only digitizing their images at six bits per pixel—something that hasn't been tolerable in the field since 1984. Even a ten-dollar Webcam does better.
One might think that a professional scientist would take great pains to understand the operation of their equipment: taking it apart and re-engineering it if necessary, decrypting its data file format, and perhaps even inventing their own control software. They are, after all, the smartest among us and their work is used to make life-or-death decisions and as the basis for clinical trials costing billions of dollars. But the passion for ripping stuff apart to see how it works seems to be a skill that is dying out. It is now quite common for research scientists—and not just postdocs, but ones in positions of high esteem—to feel comfortable saying they have no frickin' clue what a 16-bit grayscale image is and to have to be taught how to resize a figure in Word without altering its aspect ratio.
Granted, things are not yet so bad that scientists are writing 5000-word articles about an offensive tweet or calling each other Nazis, as Washington political pundits are doing. But, my friends, I've come to learn that this is academia, and we're always ten or twenty years behind the times.
oct 11 2022, 5:27 am. updated oct 14 2022 6:37 am
