Last week’s post focused on WHY it’s good to review papers.  This week, the focus is on HOW to review a paper.  By “paper” I mean a manuscript resulting from primary research, submitted to peer-reviewed scientific journal.  The ideas and opinions here are drawn from my own experience in the fields of ecology / entomology – not all the comments will apply to all fields.

Reviews of a research paper will vary tremendously depending on the experience and expertise of the individual, the research discipline, and depending on what people think a ‘review’ ought to be.  For example, reviewers sometimes approach a manuscript as if they were copy-editors, and spend a lot of time and energy fixing grammar, sentence structure, and other stylistic / editorial issues within a paper.   This is not a productive use of your time – these things can be done by (paid) copy-editors further along in the publication process.  Instead, a good review should (i) focus on the scientific quality, (ii) ensure that the overall paper delivers the research message appropriately, (iii) assess that the manuscript fits within the aims and scope of the journal, and (iv) confirm that the manuscript is aligned with the field of study.    Of course, a readable and clean paper makes this much easier (another reason authors should make sure their manuscripts are error-free, and readable, prior to submission!).

For most reviews, two reads of a paper are required.  My advice is to read a manuscript from start to finish, and then walk away for a day or so.  This lets the material settle and work away in mysterious ways, out of your conscious thought.  Upon return, do another read, only this time write down positive and negative comments about the paper, using the following list for guidance:

Big Picture: What is the conceptual framework, and is it a good one? Do the author’s make a convincing argument about the broader rationale behind their research? Do they use (recent and old) literature effectively?

Research Question: One of most important parts of a paper is the section dealing with the overall research question / hypothesis / prediction.  Think carefully about this section: it must be clear, concise, logical, and ‘testable’. Everything in the rest of the paper must relate directly to the objectives of the study.

Methods:  fundamentally, do they make sense?  Remember, the scientific method is about repeatability of an experiment: could you do the project using the described methods?  Are the methods up-to-date? Where might they lead to errors or biases?

Data management/analyses:  How were data managed after they were collected?  What was the overall data management and analysis strategy? What are the response variables measured and what is the unit of statistical replication?  What test(s) did the authors complete?  What assumptions do these tests require and did they do tests to ensure they did not violate the assumptions?  Are ‘controls’ required, and if so, what are they and how are they dealt with in a statistical framework?

Results: are the figures/tables presented in an effective manner?  Are they clear and relate directly to the research questions? Do they make sense in light of the data analysis strategy? What’s missing?  Is their supplementary information (is it accessible, elsewhere)?

Discussion:  a discussion section of a paper is meant to be somewhat speculative, but it should not wander uncontrollably from the data and results.  Does the discussion section suggest alternative explanations for patterns, or do the author’s merely use this part of their paper in an act of self-congratulation? Do the author’s point to some of the flaws in the study and offer ways to improve the experiment?  Do the author’s return to the overall (broad) conceptual framework and revisit this in a meaningful way?

Literature Cited:  did they miss important papers?  Are there any unusual trends? (e.g., too many citations of unpublished works or personal communications, too may citations of the author’s friends, ignored old literature, depended too heavily on old literature?)

Working through this list will help come up with the key issues with a manuscript.  In addition, it may be useful to think about some of the following broader points:

• Does the paper move the discipline forward in a new and exciting way or is the work largely confirmatory?

• Is the research driven by a ‘cool’ system, or a specific taxa, ecosystem or place, or is the research driven by a strong research question and the system/taxa/place are secondary?

• Does the research make linkages to larger social/economic/environmental issues?  Should it? Do the author’s use larger environmental issues or politics just to ‘look important’, or are the linkages necessary and meaningful?

• Is the work original or a re-hashing of someone else’s approach (perhaps in a different system or with a different model organism)?

• Is the work appropriate for the journal?

This last point is quite important – your review must be within the ‘context’ of the journal, and of your discipline.   You don’t want to review a natural history paper in a focused entomological journal in the same way that you would review a paper about theoretical mathematical models for a general ecological journal.   Always keep the journal’s audience in mind.

In theory, you should now be ready to write your review and submitting it to the journal.  Submitting your actual review typically involves three tasks.  For most journals, reviewers are asked to provide comments for the authors, confidential comments to the editor, and reviewers are typically asked to rate the manuscript.

The first task, and the one that takes the most time and energy, is to write all your  comments that will be forwarded to the authors.  Try to keep your comments and writing concise, and clear.   In general, a review of 1-2 pages (single-spaced) is a good goal – most reviews are much shorter than that, but if you follow the points discussed above, you will be providing a very helpful and high quality review, and it will take you more than a page to do so.

Always start your review with a quick one-paragraph summary of the work (this shows that you read it!).  You should then try to provide authors some positive feedback before delving into (constructive) criticisms.  Always keep the tone of your writing even-handed, and constructive.  Don’t be mean (this sure won’t help your scientific karma!).   I recommend first providing general ‘big picture’ criticisms, and then follow with a section of ‘specific comments’ – this can be where you can highlight those stylistic/editorial issues that have been uncovered.   In general, it is useful to provide a numbered list of your criticisms (you may even wish to use subheadings, e.g., taken from the list above) – this helps authors respond to your comments in a clear manner, and also assists the editorial board for the journal.

Without experience doing reviews, it is likely that a good review will take a long time to complete (i.e. > half a day).  However, it does get easier, and I believe that after practice, a good review can be completed within a couple of hours, provided you know the discipline well.

The review, as I have discussed it above, is really for the authors, and is to help the Editor(s) make a final decision about a manuscript.  In your comments to the authors, you must avoid making any statements about whether the paper should be published (or not) – this is not the job of a reviewer!  In fact, these kind of statements can place an editorial board in an awkward position if reviewers make claims about whether a paper is publishable – a journal may have to refuse (good) paper simply due to page constraints, or the fact that a paper may be a poor fit within a particular issue.  The Editors (and publishers) make all these final decisions, not you!

The second task is when you provide confidential comments to the editors, and this is a place where you can be more candid, and can certainly make your opinions known about whether the work is publishable.

The third and final task is where you get to score or rate the paper.  This should align with your actual review.  For example, it’s pretty bad form to be highly critical in written review and than rate the paper as ‘acceptable in its present form’.  Or, providing and short and positive review should not typically equate with a rejection!

The style of these scales vary tremendously, but most having wording something like this:

-Accept in its present form with no revisions

-Accept after minor revisions (re-review unnecessary)

-Accept after major revisions (after re-review)

-Reject but encourage re-submission in another form (e.g. short paper)

-Reject

If your review uncovers more than a couple of significant criticisms, it’s likely you will select Major revisions.  Minor revisions are typically more related to specific concerns with specific sections of the manuscript and minor revisions generally do not require authors to rework sections of their paper (e.g., new analyses, different presentations of results, rewriting a discussion).

Now you’re done.  Phew. Take a break until the next request comes in…