Incentives in Academic Research

jruohonen

While I agree with the points raised about incentives, the logic is once again in the antagonistic camp and even plain wrong (cf. if they only would be "afraid of having their careers damaged"). No. That is: the problems are in the system. Better: if not plain fraud, the "faulty" papers should stay where they are, and editors (or conference chairs, etc.) should encourage follow-ups (incl. replications, commentaries, and whatever), which, with "modern" technology, could be linked to the papers.

malshe

I see the author's point but I respectfully disagree. If anything, there is now much more pressure to make sure the research is error-free because so many people independently look at it. In the past if you found an error in published research, you had limited ways to share your findings. Most journals are reluctant to put their papers under scanner. Now we can just write a blog or tweet out the findings. The researchers today face much more scrutiny compared to the ones in the past. In economics, people are using AI to replicate older famous papers and reporting problems with them. Tools like refine.ink can help researchers identify the errors in their research before even submitting the paper anywhere for a peer review.

bonoboTP

I think everyone who enters academia goes through this phase of disillusionment. This thing you revered so much as this temple of knowledge turns out to involve the same kind of human business you see anywhere else, with petty fights, little cliques opposing each other, grudges, backscratching, quid pro quo etc etc. I think once you come to terms with this, you come out on the other side realizing that this is nothing new. The idealized version never existed. What's new is the scale and all the quantitative metrics regarding publication. > Academic research was meant to be about breaking new ground, keeping to honesty and good scientific conduct, being clear and upfront about uncertainties, errors, and mistakes, and improving our common understanding of science, all the while training the new generation to follow these goals and principles. This is a naive idealism. Research takes effort and someone has to pay. Sometimes this someone's may profess beliefs that they only want to sponsor unrestricted objective inquiry, but everyone has biases and in an iterated game, soft preferences will be gamed too. In other words, there's is always an effect of "the one who pays orders the song". The one who pays wants results. He wants prestige through the sponsored project. He doesn't want the researcher to just sit in a room and fail to generate results for years. He wants the researcher to have groundbreaking results. What they are matters less than that they be impressive and improve prestige of the funding source, among peers (in case of nobility funding it), or among the political classes (when government funds it), or to improve the PR and make the sponsor look good and charitable (if a company funds it as a sort of donation to academia). Much of this "reward" that accrues to the funder does not really depend on the technical content of the work, only inasmuch as such quality was necessary to convince the research community to designate the work as impressive. However if this designation can be helped along via other paths, ie the quid pro quo and backscratching networks, that can be just as good for the funder as long as it doesn't rise to the level of fraud or something that may really damage the reputation. Always,always follow the money and follow the chain of decisions. Who will feel good at a cocktail party among their social peers when some impressive research result happens? Who can puff up their chest and say "We paid for that! Look how awesome we are!" and when they summarize the result in a soundbite will it impress their social peers? This is not only applicable to the actual funding provider but also other orgs in the chain like university administration who provide facilities to the lab etc.

tacostakohashi

Sounds a lot like the incentives for making a software change / PR or whatever at BigCo... Publish a PR that has the word "fix" in the description, get it approved, release it to prod... you are the hero, took action, will get promoted, etc. When it turns out that the "fix" makes everything worse... well, that's everyone _else's_ problem, like whichever users run into that change blowing up, whoever is on support that day, whoever has to undo it. The old... "concentrated benefits and diffuse costs" problem.

smath

There is a great blog post [0] that sohl dickstein had a couple years ago, about what happens when the metric being optimized is only a proxy for the thing you want to maximize — what happens eventually is “overfitting”. The original metric (which we cannot directly optimize for) actually starts to get worse. I believe publishing papers and citations are those proxy metrics. [0] https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....

auggierose

What I am seeing is not people not being diligent in their research, but people not being diligent in their reviewing of research. But maybe there is a positive feedback loop here...

toomim

We can be more specific than "the incentives are wrong in Academia", because we know what the incentive is: it's peer-review. The problem is that today's peer review, while better than nothing, is far from a perfect judge of research quality. If peer-review was a perfect judge of research quality, there would be no incentive problem. The incentives would be aligned. I happen to be working on an improvement to peer review. An early prototype can be viewed at https://peeryview.org/about . Holler at me if you're also working on this problem!

oddIdentities

>I am not sure how to fix this problem. Be brave. Go public. Name names. And don't be a part of the system. You'll earn more and do more with your life and for society at large by getting to a place in engineering where you can spend an increasing fraction of your time on research.

alpineidyll3

Couldn't agree more. Excellent piece.

senshan

Hmm... The author starts with C.Munger's quote “Show me the incentive and I will show you the outcome”, then proceeds to express dissatisfaction with the current outcome of academic research. Regrettably, the author never returns back with: a) analysis of current incentives; and b) proposal to change those for a better outcome;

t0mpr1c3

I would add that the problem is not new, and not limited to junior researchers. In my first faculty position about 20 years ago, I followed a line of research that a senior (MacArthur Fellowship-winning) colleague had pioneered. After three years and a few negative results, I came to suspect that several high-profile publications were p-hacked and likely false positives. It was extremely demotivating, to say the least. I switched to a different field and fairly soon afterward left academia altogether. A very senior scientist later confirmed my suspicions about their disregard for professional ethics.

jamesforestwest

Hasn't it always been that way? But it is sad, admittedly.

SubiculumCode

The author provides no empirical evidence to support their claims, providing only anecdotal evidence. Their claim might be true, might be truer in some places/departments/countries than in others, but I do happen to agree with the author...or I am concerned about the issue. Also anecdotal, as a peer reviewer, I too often encounter shoddy, careless, or even probably fraudulent conduct papers, and it frustrates me because peer review is not cost free: Authors are asking scientists to spend their precious time to understand the author's work, and that only works under a social contract....that authors only send out manuscripts for peer review after they have tried their best to do their best science. I hate to say it, because I know that China has great scientists, but most of the manuscripts I receive and reject due to sloppy work, or purposeful negligence in citing most directly relevant to their manuscript (often because that literature did exactly what they are trying to pass if as new), or out-right making shit up, have been manuscripts from Chinese Universities. Yes it happens from western authors as well, but what I am naming is real, and is increasingly a major burden on peer-review as a scientific institution. So much so that many have confided to me that they will refuse to serve as a reviewer of a manuscript out of China unless it is from a known lab of some standing. In science, there must be a culture motivated not by number of papers published or grants funded, but by its rigor and integrity, and like the finest craftsman who cares most cares about doing the finest of work. That matters, and that culture of integrity and personal pride, does need to be encouraged and rewarded, no matter where the lab is located. This is all to say that I agree with the author, despite their lack of empirical evidence, that these values are under threat.

augment_me

As a PhD student, I am seeing the same thing. I am currently generating about 10 papers per month for marginal increments/architectural additions that are autoreviewed in a loop by Claude + Codex until those tools don't spot any major issues and then sending them in to various conferences. Since more than half of the reviewers are just using these tools to review, it's really easy to get a good score. This year I got into ICLR, ICML, NeurIps, SIGGRAPH and ACL with 2-authors papers (me and my supervisor). Its fucked up, but I am getting a lot of praise from the department/supervisor, I got an very good internship lined up next summer, and I have plenty of time to study whatever I find meaningful because the slop generates itself. I think that this ship will sink if enough people do this, so I set up similar systems for some of my colleagues as well. The only way to break the system is to overload it until it doesn't work, so I can only hope for this.

frognumber

I think part of the problem is we don't talk about these issues openly. I see a lot of posts like (from this thread): "I followed a line of research that a senior (MacArthur Fellowship-winning) colleague had pioneered... After three years and a few negative results, I came to suspect that several high-profile publications were p-hacked and likely false positives... A very senior scientist later confirmed my suspicions about their disregard for professional ethics." These issues should be discussed openly. This also means that the consequences should be appropriate. In many domains -- anything dealing with sex / gender in almost any culture is a good example to look at -- the consequences are draconian, be that stoning or cancel culture or ... which prevents open conversations. These are all spectra.