Queueing to Publish in AI and Cs

I'm in CS and I submit both to conferences and journals (the former because it's what people actually read, the latter because of evaluation requirements in my country). And I can tell you that (IMO of course) the conference model is immensely better, and idealization of journals in the CS community is a clear case of "grass is always greener".

Revise and resubmit is evil. It gives the reviewers a lot of power over papers that ends up being used for coertion, sometimes subtle, sometimes quite overt. In most papers I have submitted to journals (and I'm talking prestigious journals, not MDPI or the likes), I have been pressured to cite specific papers that didn't make sense to cite, very likely from the reviewers themselves. And one ends up doing it, because not doing it can result in rejection and losing many months (the journal process is also slower), maybe the paper even becoming obsolete along the way. Of course, the "revise and resubmit" process can also be used to pressure authors into changing papers in subtler ways (to not question a given theory, etc.)

The slowness of the process also means that if you're unlucky with the reviewers, you lose much more time. There is a fact that we should all accept: the reviewing process always carries a huge random factor due to subjectivity. And being able to "reroll" reviewers is actually a good thing. It means that a paper that a good proportion of the community values highly will eventually get in, as opposed to being doomed because the initial very small sample (n=3) is from a rejecting minority.

Finally, in my experience reviewing quality is the other way around... there is a small minority of journals with good review quality but the majority (including prestigious ones) it's a crapshoot, not to mention when the editor desk rejects for highly subjective reasons. In the conferences I typically submit to (*ACL) the review quality is more consistent than in journals, and the process is more serious with rejects always being motivated.

not sure how many conferences you have been to, but (1) abstracts are filtered, since conferences probably get 100s of abstracts for every slot they have available, some of which end up being converted to workshops or breakout panel discussions to accommodate interesting topics, and (2) I've seen presentations where "lively" discussions have happened between the presenter(s) and the audience.

The post suggests why review quality suffers. Because of the system, there is too much reviewing going on. People get tired and produce worse reviews. Those receiving these low-quality reviews become less motivated, and in turn put less effort into reviewing as well. Bad reviewing makes the system less predictable, so you have to spray and pray with as many papers as possible if you want to keep up with publication expectations. This adds even more papers into the system, making it worse.

There are just too many negative eventualities reinforcing each other in different ways.

NeurIPS scoring system is inherently subjective. People will have wildly different interpretations of, say, 3 vs 4, or 4 vs 5. You can get lucky and draw only reviewers that, on average, "overrate" papers in their batch. The opposite can happen too, obviously. 4444 vs 3444 is just noise.

And yet, submission numbers follow the (N = actually new papers)/(p = rate of acceptance) trend, approximately. The difference between a 20% and 35% acceptance rate is ~5N vs ~3N papers in the submission pool.

The exponential scaling messes with the previous more honor-based gentleman-like reputation-based everyone-knows-everyone situation. That model has its own problems but different. Today it's more cutthroat, grind-competitive and nobody has proper incentives.

It's like working long years in a family-sized company vs job hopping between megacorps.

The actual science takes the backseat. Nobody has time to just think, you must pump out the next paper and somehow get it through peer review. As a reviewer, you don't get much out of reviewing. It used to be exciting to look at new developments from across the field in ones review stack. Today it's mostly filled with the nth resubmission of something by someone in anxious hurry to just produce something to tick a box. There is no cost to just submitting massive amounts of papers. Anyway, so it's not fun as a reviewer either, you get no reward for it and you take time away from your own research. So people now have to be forced to review in order to submit. These forced reviews do as good a job as you expect. The better case is if they are just disinterested. The worse is if they feel you are a dangerous competitor. Or they only try to assess whether you toiled "hard enough" to deserve the badge of a published paper. Intellectual curiosity etc. have taken the back seat. LLMs just make it all worse.

Nobody is truly incenticized to change this. It's a bit of a tragedy of the commons situation. Just extract as much as you can, and fight like in a war.

It's also like moving from a small village where everyone knows everyone to a big metropolis. People are all just in transit there, they want to take as much as possible before moving on. Wider impacts don't matter. Publish a bunch of stuff then get a well paying job. Who cares that these papers are not quite that scientifically valuable? Nobody reads it anyway. In 6 months it's obsolete either way. But in the meantime it increased the citation count of the PI, the department can put it into their annual report, use the numbers for rankings and for applying for state funding, the numbers look good when talking to the minister of education, it can be also pushed in press releases to do PR for the university which increases public reputation etc. The conferences rise on the impact tables because of the immense cross-citation numbers etc. The more papers, the more citations, the higher the impact factor. And this prestige then moves on to the editors, area chairs etc. and it looks good on a CV.

It mirrors a lot of other social developments where time horizons have shrunk, trust is lower, incentives are perverse and nobody quite likes how it is but nobody has unilateral power to change things in the face of institutional inertia.

Organizing activity at such scale is a hard problem. Especially because research is very decentralized by tradition. It's largely independent groups of 10-20 people centered around one main senior scientist. The network between these is informal. It's very different than megacorps. Megacorps can go sclerotic with admin bloat and paralyzed by middle manager layers. But in the distributed model, there is minimal coordination, it's an undifferentiated soup of these tiny groups, each holding on to their similar ideas and rushing to publish first.

Unfortunately, research is not like factory production, even if bureaucrats and bean counters would wish so. Simply throwing more people at it can make negative impact, analogous to the mythical man-month.

The problem is that external environment (namely hiring) creates pressure to publish in a handful of prestigious venues. Nobody wants to be the odd one out that has their papers in some less well known conferences.

I used to be in (molecular biology) research. At some point my supervisors were already working towards a paper in their mind, while I was still doubting (the statistical significance of) my findings.

In the end I left my Phd track before actually finishing it. My conclusion is that I like research(ing stuff) as a verb, but I don't like research as an institute.

> valuing where papers are published over what they contribute

And who is the arbiter of that? This is an imperfect but easy shorthand. Like valuing grades and degrees instead of what people actually took away from school.

In an ideal world we would see all this intangible worth in people's contributions. But we don't have time for that.

So the PhD committee decides on exactly that measure whether there are enough published articles for a cumulative dissertation and if that's enough. What's exactly the alternative? Calling in fresh reviews to weigh the contributions?

Depends on who is doing the "careers valuing" and how closely they're looking. At a coarse level, especially for jobs in industry, venue is a pretty simple (but obviously imperfect) indicator for quality. If you've managed to publish one or more papers at the most selective venues (esp. as main author), then I would assume there's a decent chance you are good at research, even if I don't know anything about the subfield you work on. As a further indicator, the number of citations is also a noisy but easy to check proxy for "impact".

But for academic or other high-level research jobs, whoever is doing the valuing is going to look at a lot more than just the venue.

It is difficult to judge the impact for most papers in the first few years. Additionally, impact is influenced by visibility, and when there’s a deluge of papers, guess what’s a really good way to make your paper stand out? Yup, publication in a good venue.

This has truth but is not the full story. It gets your foot in the door, but I think junior researcher overinflate the importance of numbers. At the end of the day when you interview, your future boss will ask about your research, and it doesn't matter all that much how many papers they are and how many are top published. Yes you should have some top conf papers, but one great Arxiv paper that is actually used by others can have more value than 4 meh papers that slipped through the reviews.

Remember that nobody is a passive actor in the system. Everyone sees the state of conferences and review randomness and the gaming of the system. Senior researchers are area chairs and program chairs. They are well aware and won't take paper counts as the only signal.

Again, papers are needed, but it's really not the only thing.

Creating a new, first-grade conference from scratch is hard.

NeuriPS, CVPR and ICML are solid brands that took decades to build.

Im on an academic foray to an R1 right now. One of the big items the PIs are drilling is the one paper per year mantra.

Many papers today are representing what would have happened via open source repos in yesteryear. Meaning that there is a lot of work which is useful to someone and having peer reviewed benchmarks etc. is useful to understand whether those people should care. The weakness is that some of this work is the equivalent of shovelware.

Anecdotally: I tried hook and crook to get the best flagship model at the time (Opus) to help with technical writing for a submission.

First, these models are not good at technical writing at all. They have no sense of the weight of a single sentence, they just love to blather.

Second, they can't keep the core technical story consistent throughout their completions. In other words, they can't "keep the main thing the main thing".

I had an early draft with AI writing, but by the time we submitted our work -- there was not a single piece of AI writing in the paper. And not without trying, I really did some iterations on trying to carefully craft context, give them a sense of the world model in which they needed to evaluate their additions, yada yada.

For clear and concise technical communication, it's a waste of time right now.

"there are LLM written papers being peer reviewed by LLMs, but fear it not, even if they are accepted the will not be cited because LLMs are hallucinating citations that better support their arguments!"

I just shared that with several friends, and we are all having a good laugh. Thank you very much.

It's part of the wider credential inflation trend and increased grind-competition overall.

Fees mean very little.

Go for the jugular. Impact the career of people putting out substandard papers.

Come up with a score for "citation strength" or something.

Any given bad actor with too many substandard papers to his/her credit begins to negatively impact the "citation strength" of any paper on which they are a co-author. Maybe even negatively impacting the "citation strength" of papers that even cite papers authored or co-authored by the bad actor in question?

If, say, the major journals had a system like this in place, you'd see everyone perk up and get a whole lot less careless.

The people using AI are already paying to OpenAI or whoever to create those fake papers.

It would be a disproportionate blow to researchers in countries with less resources and/or more bureaucratized systems (which e.g. demand to see a "result" if you have paid a fee), who just wouldn't submit.