AI Enters the Grant Game, Picking Winners

- Arms race between writing and reviewing AI

As if there weren't numerous grant requests for dead end research before LLMs. Not saying this to discredit past research but when AI is used on both sides this changes none of the fundamental issues or incentives.

Job applications, story pitches, now grant applications, everyone is overwhelmed.

Also, for a very high stakes proposal, I doubt people are just going to ask ChatGPT to do it, which would basically guarantee that their proposal is indistinguishable from some equally lazy competitors in their field.

This is preposterous. You are completely forgetting that there are actual groups of humans, people who know and respect each other, coming together and discussing the proposals.

Being involved in ranking grant applications is already a thankless job, and many scientists still do it, for the greater good if you want. They will therefore also critically look at any AI reviews, and eliminate them as soon as they are not convincing.

(I am, of course, not saying that the grant review process is without flaws - that is a different discussion.)

AI can be used as an editorial assistant, research assistant, and copy editor. This is a huge benefit already.

AI systems are also almost brilliant in helping to tighten text. Try compressing a 1500 word discussion down to 750 words for an article in Nature.

I agree that using AI to write text from scratch is lazy ghost writing—-but not the end of the world. It may be a huge problem in undergraduate teaching, but there are also some hidden up-sides.

I definitely think AI should be used to pre-review NIH grant applications to help both the applicant, NIH, and reviewers. My institution’s “pre-award” team checks the budget, check compliance with admin policies, and a few other details. They have no capacity to help with the science.

AI systems can be highly effective as surrogate pre-review reviewers. They perform at least as well as the hard-pressed unpaid human reviewers who spend 6 hours with you application if you are very lucky, and who will definitely not be perusing your latest 10 papers. AI can do both in minutes.

I give my applications to Claude with a prompt of this general format:

“You are a grumpy and highly stressed scientist who should be working on your own research but you have nobly decided to support NIH by reviewing 6 applications, each of which will take you at least 4 hours to review and another 2 hours to write-up half-fairly. Your job as reviewer today is to enumerate what you see as theoretical and experimental deficiencies of each of the three aims and sub-aims. List at three problems per aim. Having generated this list of deficiencies, then go through the application carefully and see if the applicants actually already addressed your concerns. Did you miss their pitfalls section or “alternative strategies”?

Writing a grant application is just another form of REAL science. I learn a ton every time I write (or help write) a grant application. It is more thoughtful in some important ways than DOING science and even than WRITING UP science.

Even where AI is widely lauded (such as in programming), it needs a lot of "hand holding".

The biggest risk is that an even greater amount of time would be wasted by those who would have to screen grant applications.

If the NIH responds by globally lowering the limit to two or three proposals per year, they hurt 1%er mega labs that expect to have several active grants and now need to bat web above .500 to stay afloat. So I think it's likely that we see elitist criteria as you said, maybe a sliding scale for the proposal limit where labs that currently draw large amounts of funding are allowed to submit more proposals than smaller labs.

One place this may end up is with grant proposals requiring a live presentation component. You can use AI to crank out six proposals in a day, but rehearsing and practicing six presentations will still take quite some time and effort.

Why (the fuck, I may add) would they focus on signs of commercial promise in the first place?

> AI ... tends to be a conventional wisdom machine

And it would therefore confidently pick submissions that look like older successes. Sending in a copy of something that was patented just before the model's cutoff date would be a good strategy.

The submitted proposals can involve including a year or more of "pilot data", being experiments showing that the proposed approach is feasible. In addition 3 or 4 months of writing and admin (budgets,legal requirements, etc) are needed to complete the application.

So an application can easily be 1.5 to 2 man-years of work to prepare.

A screening process then takes place. This involves the awarding organization vetting the credentials of the applicants, as well as recruiting specialists to give their expert opinion. This takes at least 6 months.

Then there will be panel of experts who review all the above and vote. The vote is "taken into consideration" by the executive of the awarding body who make the final decisions. They might award funding for three years of work for maybe 10-15% of applicants.

As you can see, it is a massively burdensome process, with typically a very poor return on investment on the behalf of researchers.

And no, I don't think current AI is anywhere near competent enough to replace most of this process, apart from maybe the admin and legal sides.

It's actually one of my favorite parts of the job, getting to read new, promising ideas that I think might genuinely help humanity.