The “Reviewing Reviewing” (R2) Committee was formed by ACM SIGARCH and IEEE TCCA in response to concerns raised by T.N. Vijaykumar and others at the ISCA 2016 business meeting. This blog post summarizes the activities of that committee and its unsuccessful efforts to effect change in the ISCA review process.
The ACM SIGARCH and IEEE TCCA Executive Committees jointly created the R2 committee with a two-phase mandate to first identify problems with the current ISCA reviewing practices and then to propose potential solutions. The R2 co-chairs David Kaeli and Christos Kozyrakis were selected by the Executive Committees using the ISCA PC chair selection process. Each Executive Committee member ranked a list of nominees and weighted Condorcet voting was used to give each EC equal say despite different numbers of members. The rest of the R2 Committee members (Natalie Enright Jerger, Gabe Loh, Tom Wenisch and David Wood) were chosen by consensus of the TCCA chair, SIGARCH chair, and R2 co-chairs. Two of the R2 committee members are also members of the SIGARCH Executive Committee (Natalie Enright Jerger and David Wood) and four are members of the TCCA Executive Committee (Natalie Enright Jerger, David Kaeli, Gabe Loh, and Tom Wenisch). The formation of the committee was completed in mid-March 2017.
The first data collection phase began in March 2017 with a community survey to identify and gain a better understanding of any existing problems. This survey received 621 responses and results were presented at the ISCA 2017 business meeting (slides). Highlights of the survey included:
- ~74% believe in-person PC meetings are important
- ~50% feel that program committee members are overloaded
- ~50% would like to see ISCA tied to a journal
- ~50% believe integrity and fairness are paramount
- ~45% believe the current review process takes too long
- ~40% favor a revision-based model
During the Fall of 2017, the R2 Committee discussed lessons learned from the survey and reached out to several other computing research communities that have struggled with similar problems and/or have experimented with changes to their paper submission/reviewing models.
The second phase, beginning in January 2018, focused on substantially changing the ISCA review process. The results of the R2 survey are broadly in line with concerns that have been identified and addressed in a number of other computing areas. The database community has been a leader in addressing these challenges. Their first attempted solution—passing reviews between the SIGMOD and VLDB conferences—was deemed a failure for a number of reasons, including lack of conference independence (see Jagadish 2008). Their second attempt maintains conference independence, with VLDB moving to full journal-first, revision-based reviewing and SIGMOD moving to two full review-and-revision cycles per year. These multi-deadline, revision-based models are widely considered to have been very successful and variations of these models have been adopted by SIGMETRICS, IEEE Security & Privacy (“Oakland”), and a number of other conferences.
Based on the analysis of the survey results and the experience in the broader computing community, the R2 committee proposed a two-year experiment for ISCA using a multi-deadline, revision-based model. This proposal called for three deadlines, approximately four months apart. Papers submitted to each deadline could receive an Accept, Revise, or Reject decision. Authors whose manuscripts received a Revise decision would be allowed to submit a revised manuscript to one of the next two deadlines (which might be for the next year’s conference) and would be guaranteed to be re-evaluated by the same reviewers. Such a model would allow for ISCA to be paired with a journal (if adopted permanently). The full details of this proposal are available here.
This proposal was presented first to the SIGARCH Executive Committee and then to the TCCA Executive Committee (EC) for feedback and approval. SIGARCH had a number of suggestions for improvement, but was supportive of moving forward with the experiment. In contrast, TCCA was very concerned about the potential negative impact on the broader community (specifically HPCA and MICRO) and asked for substantial revisions. In particular, TCCA voted 13-5 to approve the following process to revise the proposal:
(i) We [the TCCA EC] request the R2 committee to revise the R2 proposal with two specific requests: (a) schedule the paper submission deadlines such as to prevent/minimize ISCA hurting the number of submissions to HPCA (and MICRO), and (b) investigate the possibility for multiple staggered deadlines rather than interleaved deadlines. (Note that (a) could possibly be accomplished through (b).)
(ii) Given a revised proposal that makes progress on the issues listed above, [the TCCA Chair] will then ask the TCCA EC to vote on the revised proposal. It would be strongly expected that no more changes be required.
The R2 committee took the feedback from SIGARCH and TCCA to heart and proposed a revised model that we believe incorporated SIGARCH’s suggested improvements and satisfied TCCA’s specific requests. Highlights of this revised model include:
- Divides the calendar into four-month “exclusive windows” for new submissions to ISCA, HPCA, and MICRO, consistent with deadlines in recent years, allowing each conference to innovate without unduly impacting the other conferences.
- ISCA would use three staggered deadlines within its exclusive submission window.
- Distributes reviewing load and facilitates major revisions
- Pipelines submissions, reviews, and revisions with “bulk commit”
- Decisions made in online discussions and a single in-person PC meeting
- Adds a post-PC meeting “Revision only” submission deadline for revised manuscripts that will potentially appear in the following year’s ISCA.
- Proof-of-concept schedule with HPCA 2020 using same model, demonstrating how they could work well together.
Further details of the revised proposal are available here.
The revised proposal was presented to the SIGARCH and TCCA ECs. SIGARCH unanimously approved the revised R2 proposal on an 8-0 vote. Despite the fact that this proposal includes “multiple staggered deadlines” to “minimize ISCA hurting…HPCA (and MICRO),” the vote by TCCA resulted in a 9-9 tie. Because approval required a majority vote by both ECs, the proposal was not approved.
The R2 committee will present a detailed report at the ISCA 2018 business meeting and post a final report with more details in the next few weeks.