I hope to write more about some new papers in the literature soon, as well as providing links to some pretty cool physics-related content. First, though, I want to say something about this preprint, which has people talking. The article says, in the case of an ensemble of social sciences papers, anyway, that AI-based review is better, in nearly every quantifiable way, than human peer review. This question of AI for reviewing papers and proposals has been in the air for a bit now, as in this article in Science, this editorial at Nature Nano, and the fact that the present administration has been using AI tools to "review" grants for content that allegedly goes against executive orders.
I'm an associate editor at a journal, and I see a large volume of papers and reviews. (Note that this post reflects my personal views, not an official take by the journal or its publisher.) It is 100% clear that some reviewers are using AI tools to prepare their reviews, despite clear statements that this is not ok. There is a distinct format that AI reviews take. After a summary paragraph, usually there are five specific bullet points, one or two of which tend to zero in on some particular detail or peculiarity in wording of the paper in a way that humans probably wouldn't. Often the phrasing is somewhat aggressive, with a bit of a "gotcha" tone. Of course, in any given case, I can't prove AI was used, but if you read enough of these, they stand out. To be clear: I'm not talking about people who use AI to clean up grammar. I'm talking about people feeding articles to an AI tool and asking it to spit out a review.
The point of peer review, whether it's for grants or articles, is to get the informed views of the relevant scholarly community about the work. We want to know what actual practitioners think of the results and their presentation. Purely AI reviews just do not serve this purpose.
Moreover, there are serious confidentiality issues at stake. Sure, many scientists use preprint archives these days to put up prepublication work, but plenty still do not, especially for very hot topics where people are worried about being scooped. Running submitted manuscripts or grant proposals through AI tools that can then scrape the content into their training sets is a fundamental violation of the confidentiality of the process. Imagine someone uploads your manuscript about a machine learning search for a new high temperature superconductor to Gemini to generate a review, and then amazingly SandboxAQ later scoops you on this. (Not accusing google of anything here, just a hypothetical example. Feel free to say GPT and Mattergen.) The math community already worries about this stuff.
So: If you are asked to review a paper or a grant proposal, bear in mind (1) the people doing the asking actually want your feedback, and (2) taking confidential documents and allowing them to be scraped by AI companies is bad.
Update: This paper is highly relevant, showing very interesting consistency issues with LLMs as reviewers.
Right. If the editor wanted to put in zero effort, they would just prompt the AI themselves and generate some slop. Also, the paper being read and critiqued by a human makes it more likely that other humans will understand and be able to build on the work.
ReplyDelete@anonymous 9:27: I‘d bet that more than 90% of the papers that are being published nowadays are not being read by any human apart from the reviewers (who apparently stopped reading papers either and outsourced this to AI). So your presumption that papers should be written such that humans can understand and build on them seems a bit anachronistic.
ReplyDeleteTo the extent that is true, it’s because of the incentives of the publish or perish system, and is not inherent to science, nor is it good.
DeleteI think that 90+% of the reads may be nonhuman, but I think that nearly every paper is also read by at least a few humans. (With current day "reading" more a browsing sectional reading, than an old fashioned full read
Delete...)
@anonymous 4:40: i fully agree. The problem has only been exacerbated by AI, but has existed for much longer. In the early 2000s a physics professor of mine already said half-jokingly that we’‘ve entered an era again where science is passed on only orally, as nobody reads papers anymore. An then there‘s the decades old joke that if you stack all papers that are published in Phys. Rev., the stack grows faster than the speed of light. But that‘s not a problem, cause the information content is zero.
DeleteIf the vast majority of submitted papers are of very low quality, one cannot blame the reviews for using AI to reduce their workload if they are being swamped by review requests.
Just published this afternoon in PNAS, a study that counts the incidence of fake references in science papers. Physics papers were the worst offenders, unfortunately. https://www.pnas.org/doi/10.1073/pnas.2623869123
ReplyDelete