In computer architecture conferences, “expert”/”knowledgeable” reviewers often are conflicted intellectually with at least some of the submissions they review (e.g., the submissions directly compete with or point out flaws in the reviewers’ past or current work, or solve problems that the reviewers are working on). Such conflicts exist by definition because an expert/knowledgeable reviewer is one who has previous (and likely, current) work on the topic of the submission. Such conflicts have always existed but this problem has worsened vastly due to larger PCs and more distributed and more anonymous review process. Many (all, in ASPLOS) accept/reject decisions are online, arrived at only by the reviewers without anyone else’s knowledge and the reviewer names may be hidden even from each other during most of the review process.
Problem
There are two types of papers. Those that say previous work is good, addresses one problem, but this paper addresses a different problem (i.e., largely unrelated to previous work). And those that legitimately say previous work has a fundamental flaw, meticulously show the flaw, and propose a different, clever, and better way (i.e., directly compete with previous work). The second type is crucial to moving the field forward.
Several good papers of the second type may be getting rejected due to intellectual conflicts —expert/knowledgeable reviewers protecting their own territory or the flaws of their own past work. The conflicted reviewer’s attitude is that his/her own work cannot be wrong, and that there must be something wrong with the submission. Ignoring the arguments and evidence in the paper, the reviewer manufactures “concerns” – false, small, irrelevant, anything at all. The reviewer asks some questions but ignores the author’s response, claims to remain “unconvinced” (which neither the reviewer can prove nor the authors can disprove – unscientific), or moves on after the rebuttal to new questions that the authors cannot see. The reviewer wants only one thing: to get the paper rejected.
The other reviewers do not stop this nonsense. Inevitably, there are 1–2 indiscriminately-negative (often unconflicted!) reviewers who diminish or ignore the papers’ contributions. Any remaining non-expert reviewers (unconflicted, by definition), even if initially positive, simply defer and fold to the expert’s concerns unaware of his/her conflict and actively pile on with their own objections which inevitably exist in any review. The conflicted reviewer actively influences post-rebuttal discussions when the author is not present to counter any falsehoods. Review independence, which is the whole point of five-way redundancy in the review process, is a myth; review deference is the reality. The process does not even recognize this conflict, let alone alert the other reviewers to the conflict (the conflicted reviewers reveal nothing). The result is good papers getting rejected (out of 5 reviews, 4 weak/hard rejects).
Impact
Conference impact: When the paper is resubmitted, even if a particular expert is not in the PC, some other expert is (by design, there are always some conflicted reviewers). The conflict continues and unaware co-reviewers continue to fold and pile on. These are systematic rejects – not random rejects. The result is repeated, and eventually permanent, rejection of good papers. The problem is insidious.
Such papers may be only 5-10% of all submissions which seems a small problem (an educated guess, as an author and a reviewer). But rejecting these papers with solid contributions hurts the field and is super unfair; and they could be 25–50% of the program (at typical 20% acceptance).
You would expect the acceptance rate to plummet and reveal the problem of good papers getting rejected, but the void is filled by other, weaker papers hoisted up by lenient or uncalibrated reviews (and possibly, a small fraction due to collusion). The net result is the typical 20% acceptance rate making it look all ok. But is this the mostly top 20% or a somewhat random 20% (good out, weak in)?
Human impact: As the authors repeatedly receive rejects due to conflicted reviews, the frustrated authors are also likely to act on their own conflicts when the authors are reviewers themselves. And the downward spiral continues. Bright grad students whose good papers get rejected repeatedly, sometimes permanently, have a dim view of the community. Do you think the students will want anything to do with academic or industry research after graduating?
Repeated or permanent rejection of good papers irreparably hurts the job prospects, jobs, and careers of the authors. Our entire system – graduate student jobs, faculty tenure and promotions, research careers, idea attribution and awards – is founded inextricably on publication record. Intellectual conflicts topple this edifice by dishonestly hurting the authors of good ideas. Imagine a Ph.D. student forced to graduate (or a faculty forced to go up for tenure, promotion, or a routine annual departmental review) with zero publications because all her good ideas (on different topics) were rejected repeatedly and permanently by (different) intellectually-conflicted reviewers. Such cases are happening but remain hidden due to their low volume and the stigma of a poor publication record. This is destroying careers and people.
An equally big impact is the conflicted reviewers’s ethical and moral erosion. This is an ethical and moral crisis in the making.
A moral standard
In 1900, when the famed mathematician David Hilbert announced the legendary list of unsolved problems, he envisioned a grand research program to tackle these problems. Godel’s Incompleteness theorems decimated the program. Hilbert hated Godel’s result – he was furious. Still, Hilbert was true to his craft and accepted Godel’s proof instead of engaging in deceit which would have been easy given Hilbert was a giant in mathematics at that time and Godel was a budding genius.
Acting on intellectual conflicts in reviews is unethical and immoral. Falsehoods, exaggerations, bad faith “concerns” and other manifestations are conduct unbecoming of reviewers. If some paper does better than our idea fair and square, or proposes a nice idea that eats into our territory, we must accept the fact and the paper. (Papers have to do better than previous work — our very business is “new and better”.) We should compete with good ideas by coming up with better ideas; not by dishonestly rejecting good ideas, hiding behind reviewer unaccountability and anonymity. We must compete on the court, by the rules — not jump the competition in a back alley. We must do the right thing, especially when nobody is looking. (Accepting unconflicted, good papers needs no character – only good judgement.)
Expert/knowledgeable reviewers must be scholarly and be committed to their craft and the truth. As an expert, he/she must act as a steward for the topics of his/her expertise, counter indiscriminate reviews, guide the non–expert co-reviewers to the truth and clear their misconceptions, instead of sowing seeds of doubt, raising bogus objections, and engaging in subterfuge.
Caveat
We cannot be naive: conflicted reviewers can be infinitely creative in hiding their dishonesty from authors and co-reviewers.
Non-solutions
Asking reviewers to be “objective” won’t cut it — the conflicted reviewers know exactly what they are doing. Aiming to avoid all intellectual conflicts in review assignments is likely a non-starter for PC chairs (doing so would all but preclude expert or knowledgeable reviewers, by definition). In addition to ensuring not more than one intellectually-conflicted review per submission, PC chairs should consider putting in the following measures.
Possible mitigations
Review process: The first step is for the review process to recognize not only the conflicts that may unfairly help the authors but also those that may unfairly hurt the authors. The review form for every paper should have an ethics declaration read and signed by each reviewer. Conflicted reviewers must self-identify their conflict to their co-reviewers (hidden from the authors). Co-reviewers should call out if such self-identified conflict is missing (easy: if some paper is pointing out flaws, correctly or incorrectly, in my paper or doing better than my paper, then I have a conflict though I can explain to my co-reviewers if the flaws are incorrect). Review forms should have a box for every review (hidden from the authors) where every co-reviewer agrees with the conflict status marked by the reviewer.
Co-reviewers, especially non-experts: Co-reviewers must super-vigilantly gauge whether any conflict is unfairly hurting the paper (i.e., whether any critique is accurate and proportionate to the paper’s flaw or a disproportionate excuse to reject without revealing the conflict as the true motivation), Reviewers should be asked to flag to the PC chairs if they think a paper is not getting a fair shake and to counter any unfairness. The non-experts’ common practice of folding and piling on is unacceptable. Non-expert reviewers must be instructed by the PC chair that they have a crucial role to play: they are competent researchers who have read the paper and therefore they can and must check the expert’s objections and block any bogus concerns while being acutely aware of conflicts. Any paper, where all the expert/knowledgeable reviews are conflicted, should be flagged by the PC chairs to require extra assurance from the co-reviewers and the conflicted reviewers (insisting on new reviews may be a non-starter for PC chairs). There should not be more than one highly-conflicted review per paper (where the reviewer’s entire body of work is at stake). .
Authors should be able to call out any perceived conflicts in their rebuttals which the co-reviewers and PC chairs should verify. Specifically, the rebuttal/revision form should have a box for each review for the authors to call out manifestations of conflicted reviews: falsehoods, exaggerations, bad faith “concerns”, and incommensurate scores. Such calling out should not be deemed “bad rebuttal tone” by the reviewers, as is common today. (Of course, honest mistakes and misunderstandings by reviewers are possible for which we have the main rebuttal/revision.) After all, no co-reviewer – nobody – has more skin in the game than the authors.
Post-rebuttal discussion is the most perilous time for good papers when conflicted reviewers spout (new or old) falsehoods and influence the others without the authors being present to counter. We should require each reviewer to commit to specific, 2-3 objections after reading the rebuttal and the authors should rebut this final set in a second round. If a reviewer’s objections are not significant enough or are blocked by the authors, as judged by the co-reviewers while being aware of any conflicts, then the reviewer should not influence the discussions any more and the reviewer’s reject vote should not count (without this veto, conflicted reviewers will continue to influence the outcome).
Any phase1 reject or regular reject summary must explicitly state that every reviewer agrees that any conflict, present or not, was managed carefully. All this needs a whole lot more reviewer independence than what we typically see today. “I will vote with the majority” and “I will defer to the experts” were never acceptable, more so for papers with conflicted reviews.
Entire community: Most importantly, there must be zero tolerance in our collective conscience for reviewers acting on conflicts, similar to that for plagiarism by authors.
We need to include ethical reviewing in our graduate programs and run workshops.
We likely cannot eliminate conflicts, but we can recognize them, make them visible to the co-reviewers, manage them carefully, and most importantly prevent them from hurting good ideas.
Conclusion
PC chairs and reviewers should take seriously the low-volume but high-impact problem of intellectually-conflicted reviews. This problem can cause irreparable damage to the authors’ jobs and careers, and our field’s progress. Reviewers must realize the serious, personal ethical and moral dangers of acting on their conflicts. Given the volume of submissions, the only realistic bulwark against this scourge is independent and ever-vigilant co-reviewers (a redundancy already built into the system, but woefully ineffective). PC chairs should empower authors and require co-reviewers to call out conflicted reviews. Volunteering to review does not give anyone the right to reject papers motivated by intellectual conflict. The dishonesty of intellectually-conflicted reviews must be met with zero tolerance in our collective conscience, just as plagiarism does. Truth, honesty, integrity, fairness, and scholarship in reviews are fundamental to the well-being of computer architects and computer architecture. We should emulate Hilbert.
About the author: T. N. Vijaykumar teaches and studies computer architecture as Professor in the Elmore Family School of Electrical and Computer Engineering at Purdue University.
Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.
