How I think about peer review

Peer review is a responsibility for researchers. If you submit research work to be reviewed by others, I think you generally should be doing some reviewing.

Non-students should generally review at the same workload they “generate” in submissions: a good rule of thumb is conducting three reviews for every first-author submission.

How I write reviews

I write reviews using the following structure:

===Paper summary and intended contribution===

…

===Reasons to accept===

-…

===Reasons to reject===

-…

===Originality of work and potential impact of results===

…

===Quality of execution===

-Quality of execution seems {high,low,uneven,etc}.

-…

===Quality of presentation===

-Quality of presentation seems {high,low,uneven,etc}.

-…

===Adequacy of citations===

Citations are adequate to my knowledge, …

===Ethical concerns===

…

===Comments for authors===

Thank you for your submission to {venue name}! More detail-oriented comments below, in no particular order.

-…

===References===

COI Disclosure: I am not a co-author or colleague of any of the authors of the papers I cite in this review.

[1] …

[2] …

…

I include comments to the committee (that will not be visible to authors) at the end.

Why do I use this structure? It was the structure used by the reviewing form for the first place I reviewed (ICWSM 2019), and I find it to be useful for structuring my thoughts.

Recently, I have been including the following text at the bottom of my review:

LLM Disclosure: I did not use generative AI technologies in the drafting of this review.

Further reading

See Jeff Leek’s “Reviewing academic papers”.

Many journals and conferences provide an explicit reviewing guide. (For example, CHI 2024 has a nice reviewing guide.) When in doubt, following the review guide is a good idea.

In human-computer interaction specifically, I find Jacob Wobbrock and Julie Kientz’s 2016 paper “Research contributions in human-computer interaction” to be a useful guide. About systems papers specifically, see Tessa Lau’s 2010 blog post Rethinking the Systems Review Process.

I’ve written other blog posts that touch on reviewing: