Writing a peer review

Update: I got an ICWSM Reviewer Award this year — Thanks ICWSM!
I’m using this post as a running list of checkpoints for reviewers of computational social science and computer science conferences. But everything can be generalized to whichever field you belong to.

Here’s a short and sweet list of points to write your next peer review, courtesy ICWSM-19:

  • A brief summary of the main content and contribution of the manuscript.
  • A summary of the strengths of the manuscript.
  • A summary of the weaknesses of the manuscript.
  • The anticipated impact of this work.
  • Ethical considerations of the data/problem, if any.
  • Any other relevant points.

It can be incredibly hard to be a good reviewer, but it’s an important way to (a) improve your own research thinking, (b) stay up to date with the field, and (c) give back to the community. This blogpost is a rant about the reviewing practices for CS conferences, but it may apply to your own experiences reviewing elsewhere.

The problem is always that there are so many reviews to do and so little time. To cap it all, conferences often encourage lazy reviews. There are radio buttons which you use to determine the novelty, size of the contribution, the scope of research. Very good. But I took a minute to really think about this and read some pieces by other academics about what makes a good review or even good evaluation criteria. This is not it!


Talk about Significance and Validity

You’ll see in Elmqvist’s eloquent piece that these questions allow reviewers to take a shortcut out of providing actual, thoughtful, helpful feedback. Of the questions in the review form above, which ones will actually help to make the research better? Probably only 5 and 6!

Here’s an excerpt from Prof. Elmqvist: there are two glaring omissions in these rubrics that reviewers need to care about:

Significance. How important is the contribution over the existing work in the literature? What are the potential benefits to science and to society? Not all problems are worth solving and not all questions are worth answering. Discuss what this paper actually contributes to the research field.

Validity. This section discusses the methodology and the approach taken to the research, and the corresponding confidence other researchers and practitioners can have in the results. This is the part where you evaluate and discuss the validation mechanisms the authors used to prove the significance of their work. In case they provide an implementation, discuss the feeling you have about the implementation (feelings you get from reading the technical description as well as watching any video). For a theoretical argument, analyze and discuss the internal and external validity of their argument. For a user study, discuss whether the study was conducted competently, if the tasks were ecologically valid, if the data was analyzed correctly, etc.

Think about who is reading your review

More really great advice I read, says you should think about who is reading your review. See the NAACL blog and ACL’s last-minute reviewing advice.

What to write? The finer points

Coming back to the list I started this post with, see, for example, the in-depth review criteria for this other CS conference that I review for.

  • What is this paper about, and what contributions does it make?
    Please describe what problem or question this paper addresses, and the main contributions that it makes towards a solution or answer.
  • Strengths:
  • Weakness:
  • Questions and Feedback for the Author(s)
  • Presentation improvements
  • Typos
  • Reviewer confidence

If you’re still reading and what to know what other smarter people think about writing reviews, I’d encourage you to have a look here.

A great Twitter thread by @cmMcConnaughy on peer reviews suggests that a review should be organized as:

Paper’s Q, arg, evidence

“This paper asks XXXXX, argues YYYYY, assessing that with ZZZZZ.”

Recommendation and its logic (in Q/arg/evidence terms)

Offer a summary judgment of the paper and an argument for that summary judgment based on those three things: question, argument, evidence and some secondary issues with the presentation

“My summary judgment is that the author(s) should be invited to revise and resubmit because this is an important question/argument [because: maybe because novel, maybe because debate] and the evidence provided tells us [fill in this blank]”

Major issues (Q/arg/evidence)

Main issues with the research (usually questions/clarifications/analysis extension or detail)

Minor issues (presentation, quibbles)

Presentational issues, too hard to interpret

What is a Major Issue?

Ok great. But um. What is a good strength or weakness? To answer that, I looked at ACL’s review form FAQ.

Are there resources you want me to link to? Drop me a line, and I’ll add them in.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at WordPress.com.

Up ↑

%d bloggers like this: