Editorial: Peer Reviewed

After being distracted by shopping last time, I have managed to refocus and believe I am getting closer to writing an editorial. Before I do, I would like to consider the other main duty of Overload’s editor – after gathering articles – making sure they are thoroughly reviewed before publication. Nowadays many people write blogs. You write the piece, and then people can argue with you in the comments after the fact. When finding a blog you frequently need to peruse the comments in order to get all the details. Something similar happens with question and answer websites like Stackoverflow. To get the full answer with all the caveats, one must read more than just the headline answer. In contrast, a peer reviewed composition will have been through a few initial drafts and the final version will be an opus incorporating all the comments from the reviewers, if the editor has done their job properly. If something does slip through the net, then letters to the editor are of course welcome, or further articles showing an alternative approach can be proffered.

This begs the questions, “Who are the reviewers?” and “What is their role?” In some sense, the reviewers act like a jury. Jury appears to stem from the Latin jurare meaning to swear, in the sense of a binding promise rather than a blasphemous stream of obscenities. The idea of a jury brings to mind the phrase “12 good men and true”. The “good men and true” can certainly be found in Shakespeare’s Much Ado About Nothing, though I believe woman have been allowed on juries since the 1920s in the UK and USA. Overload would welcome female reviewers – if any were to apply. I am not clear why or when courts decided they required 12 jurors. Overload currently has 7 reviewers, and from time to time seeks input from subject specialists as required. Some are more active than others – reviewers are in no way required to comment on every article, but many thanks to those who do dutifully attempt this. It seems possible that the use of twelve peers relates to Charlemagne’s Twelve Peers in the old romances [Charlemagne], and the internet suggests that peer review was first recorded in the 1970s [Peers], of course, giving no suggestion 12 reviewers are required. The number 12 crops up frequently in various contexts – both with religious significance [12 tribes of Israel] and various measuring systems (for example, feet and inches, old coinage, hours in a day). I believe is often used because it breaks into many factors – 3 groups of 4 and so on. This might make it appropriate for coinage and groups of people requiring different divisions depending on context. Other examples for twelve people taking on significance include the Magna Carta [BL]:

All evil customs relating to forests and warrens, foresters, warreners, sheriffs and their servants, or river-banks and their wardens, are at once to be investigated in every county by twelve sworn knights of the county

Whatever the evil customs were, it certainly makes sense that one person might not be sufficient to establish the validity of something. For example, Deuteronomy 19:15 says

One witness is not enough to convict anyone accused of any crime or offense they may have committed. A matter must be established by the testimony of two or three witnesses.

Of course, I am not claiming that a submitted article is equivalent to a crime or offense. It is simply that more pairs of eyes, to a point, can spot errors and inconsistencies, making a final article more polished. Mind you, the Bible also says “Judge not, lest ye be judged.” I digress. The important point is more than one person passing comment on a submission is more likely to cover all angles, including grammar, spelling, potential edge cases in any code presented, or all-out glaring bugs, omissions, alternatives that deserve consideration and so on. Though one can often find comments on the technical content of a blog, one frequently doesn’t see suggestions on writing style or alternative wordings. This is one, if somewhat small, advantage Overload, as a peer reviewed journal, offers over just writing your own blog. Of course, Overload welcomes blog entries if the writer wishes them to be published here too, but be warned, they also go through the review process.

As programmers, many of us will be used to peer reviews in the form of code reviews, either as a formal or informal process with colleagues at work, or in order to get contributions into an open source project. This can be quite an emotional experience until you get used to it. Having someone say all the things they don’t like about what you have done can be very deflating. Some submissions have caused a vast number of comments and nit-picks before finally being accepted. The same can happen in code reviews. I like to assume the amount of comments is proportional to the time people spent thinking about what you said. At least they listened, even if they missed your point. As a writer, or programmer, you will learn to take criticism, and it does become less personal the more feedback you get. However, good reviewers, be that of an article, or of code, should always say what they like as well as saying what they don’t like. It is important to encourage the positive. Sometimes, places employ pair programming, and this can be considered to obviate the need for code reviews, though again the extra pair of eyes can help. An obvious difference between pair programming and a code review is the review will tend to take place once the code is complete, in the programmer’s opinion, whereas the pairing approach means feedback is continuous and happens during the course of the work. Overload is quite willing to give feedback before a writer thinks the submission is polished and ready for a final review. Most articles do go back and forwards a few times. Mind you, so does code during a code review.

Having briefly looked at what a reviewer does, we should return to the question “Who are the reviewers?” Staying, for the moment, with the code review analogy, some organisations only allow senior people to review code. Perhaps the team leader or senior architect has to approve any code changes. This begs the question of who reviews the reviewers’ work. I personally am happy for less experienced or more junior people to review my code. They can still spot things I have missed, and the exchange can allow knowledge to pass in both directions. Perhaps I will be able to explain why I have taken a given approach. Perhaps they can tell me elements they don’t understand. This is part of the vital team building process though and takes us away from the question of who the reviewers are. Other peer reviewed journals might choose experts in the field or throw out a periodic call for reviewers asking them to submit a review they have previously written, for example, a book review as proof of competence. As an ACCU member, you can volunteer for book reviews – just checkout the website, if this is something you would like to get good at. If you wish to volunteer to be a reviewer for Overload, then get in touch with me. You will not be asked for a sample of your previous work. You will simply get included on an email list and are welcome to provide feedback on any aspect of the submitted articles. We tend to keep the reviewers anonymous, though have sometimes named individuals when asked, for example if they have produced a stunning alternative code sample. Credit where credit’s due.

Overload’s review team does usually consistent of volunteers, though historically a jury would not necessarily have been composed of volunteers. It seems members were ‘empanelled’ by a sheriff in the thirteenth century [Musson97]. The eyre was a circuit travelled by the sheriff and his men in England. The ‘justices’ arrived unannounced at irregular intervals, forming a flash-mob review panel, of sorts. Of course, the justices’ main role was raising funds for wars, rather than simply listing and reviewing the state of the eyre. As editor, I may attempt to empanel extra reviewers from time to time, if I feel we need someone with expertise in a subject not currently covered by the team, and do try to avoid wars. There has been little resistance to being ‘volunteered’ so far. If the jury of peers are not volunteers, in what sense are they peers? It seems the jury used to have to be nobles or high ranking, so nobles could not be judged by ‘less important’ people. References to the idea of peers or equals can be found in the Magna Carta [BL], for example

Earls and barons shall be fined only by their equals

and

To any man whom we have deprived or dispossessed of lands, castles, liberties, or rights, without the lawful judgement of his equals, we will at once restore these.

In a criminal court nowadays, one is supposed to be judged by peers or contemporaries though I suspect an Earl would not be allowed to insist that the panel of jurors consistent solely of Earls. Peer itself seems to be rooted in the word ‘par’ thereby tracing back to the idea of equal ranking. Ranking is of course, a relative term, and these history lessons remind us of the extremely hierarchical nature of groups of people in England in times gone by, though echoes still remain. I like to think every ACCU member is equally qualified to peer review articles, regardless of membership of the C++ committee, years of experience and so on. Everybody’s input can be equally valuable.

Another reason for peer review of academic journals is to carefully validate any claims made. It is clearly important claimed advances in medicine are carefully checked and validated. A scientific journal will insist on a methods section, so that the results can be replicated. This is part of the essence of scientific discovery, though a brief study of the history of science will show a long journey to settle on this methodology. Even with a rigorous review process, things do slip through the net. There are many examples, including out and out lies, such as the falsification of data in stem cell research [Suk]. Such lies are usually eventually uncovered and in this case the journal, Science, editorially retracted the two papers by Hwang et al. Other peer reviewed articles that eventually get called into question have not been based on fabrication. For example, the measles, mumps and rubella vaccine controversy, which tried to link the jab to an increased risk of autism [MMR]. Hopefully, Overload has never published falsified data or caused public controversy.

Let us wrap up this review of the review process with an overview, loosely based on a guide to evaluating information sources [Lloyd Sealy]. First, the goal of peer review is to assess the quality of articles submitted for publication. This involves a, possibly iterative, process: An author submits to the editor who forwards the article to experts in the field, maybe after an initial read as a sanity check. The reviewers evaluate the submission, ‘For accuracy and assess the validity of the research methodology and procedures.’ The reviewers can, and often do, suggest revisions. In theory they can reject the article, though we aim for enough feedback to iron out any problems. Writing for a peer review journal, rather than self-publishing, will give you early feedback and potential guidance you will not get elsewhere. Referencing a peer reviewed article may stand your words on more solid ground than just surfing the web for quotes that match your thinking – I do realise the irony of my references being full of urls. If you feel inspired to submit an article or join the review team then contact Overload@accu.org.

References

[12 tribes of Israel] Genesis 49:28

[BL] A British Library online translation of the Magna Carta : http://www.bl.uk/treasures/magnacarta/translation/mc_trans.html

[Charlemagne] http://www.etymonline.com/index.php?term=peer

[Lloyd Sealy] http://guides.lib.jjay.cuny.edu/content.php?pid=209679&sid=1746812 (many thanks to Roger Orr for this link)

[MMR] http://www.bbc.co.uk/sn/tvradio/programmes/horizon/mmr_prog_summary.shtml

[Musson97] ‘Twelve Good Men and True? The Character of Early Fourteenth-Century Juries’ Anthony Musson Law and History Review, Vol. 15, No. 1 (Spring, 1997), pp. 115–144

[Peers] http://thesaurus.com/browse/peers

[Suk] See http://www.sciencemag.org/site/feature/misc/webfeat/hwang2005/