Update: I got an ICWSM Reviewer Award this year — Thanks ICWSM!
I’m using this post as a running list of checkpoints for reviewers of computational social science and computer science conferences. But everything can be generalized to whichever field you belong to.
Here’s a short and sweet list of points to write your next peer review, courtesy ICWSM-19:
- A brief summary of the main content and contribution of the manuscript.
- A summary of the strengths of the manuscript.
- A summary of the weaknesses of the manuscript.
- The anticipated impact of this work.
- Ethical considerations of the data/problem, if any.
- Any other relevant points.
It can be incredibly hard to be a good reviewer, but it’s an important way to (a) improve your own research thinking, (b) stay up to date with the field, and (c) give back to the community. This blogpost is a rant about the reviewing practices for CS conferences, but it may apply to your own experiences reviewing elsewhere.
The problem is always that there are so many reviews to do and so little time. To cap it all, conferences often encourage lazy reviews. There are radio buttons which you use to determine the novelty, size of the contribution, the scope of research. Very good. But I took a minute to really think about this and read some pieces by other academics about what makes a good review or even good evaluation criteria. This is not it!
Talk about Significance and Validity
You’ll see in Elmqvist’s eloquent piece that these questions allow reviewers to take a shortcut out of providing actual, thoughtful, helpful feedback. Of the questions in the review form above, which ones will actually help to make the research better? Probably only 5 and 6!
Here’s an excerpt from Prof. Elmqvist: there are two glaring omissions in these rubrics that reviewers need to care about:
Significance. How important is the contribution over the existing work in the literature? What are the potential benefits to science and to society? Not all problems are worth solving and not all questions are worth answering. Discuss what this paper actually contributes to the research field.
Validity. This section discusses the methodology and the approach taken to the research, and the corresponding confidence other researchers and practitioners can have in the results. This is the part where you evaluate and discuss the validation mechanisms the authors used to prove the significance of their work. In case they provide an implementation, discuss the feeling you have about the implementation (feelings you get from reading the technical description as well as watching any video). For a theoretical argument, analyze and discuss the internal and external validity of their argument. For a user study, discuss whether the study was conducted competently, if the tasks were ecologically valid, if the data was analyzed correctly, etc.
Think about who is reading your review
What to write? The finer points
Coming back to the list I started this post with, see, for example, the in-depth review criteria for this other CS conference that I review for.
- What is this paper about, and what contributions does it make?
Please describe what problem or question this paper addresses, and the main contributions that it makes towards a solution or answer.
- Questions and Feedback for the Author(s)
- Presentation improvements
- Reviewer confidence
If you’re still reading and what to know what other smarter people think about writing reviews, I’d encourage you to have a look here.
A great Twitter thread by @cmMcConnaughy on peer reviews suggests that a review should be organized as:
Paper’s Q, arg, evidence
“This paper asks XXXXX, argues YYYYY, assessing that with ZZZZZ.”
Recommendation and its logic (in Q/arg/evidence terms)
Offer a summary judgment of the paper and an argument for that summary judgment based on those three things: question, argument, evidence and some secondary issues with the presentation
“My summary judgment is that the author(s) should be invited to revise and resubmit because this is an important question/argument [because: maybe because novel, maybe because debate] and the evidence provided tells us [fill in this blank]”
Major issues (Q/arg/evidence)
Main issues with the research (usually questions/clarifications/analysis extension or detail)
Minor issues (presentation, quibbles)
Presentational issues, too hard to interpret
What is a Major Issue?
Ok great. But um. What is a good strength or weakness? To answer that, I looked at ACL’s review form FAQ.
Are there resources you want me to link to? Drop me a line, and I’ll add them in.