Peer review: not what it's cracked up to be

Peer review is held up as the holy grail in scientific papers. The IPCC trumpets its strict policy of using only peer-reviewed literature in its proceedings (but has been caught deviating from it, but that's another story). I had no reason to doubt the efficacy of the process until I became aware of these two things: (1) the leaked emails from scientist in the "climategate" controversy and (2) the disintegration of Michael Mann's Hockey Stick.

Peer review consists of the review of papers submitted for publication by independent panels of scientists chosen by editors. The reviews are voluntary, anonymous, and unpaid. When defined this way, it is already clear that the process is not going to be in the depth that some may assume.

In the email controversy the scientists at Climatic Research Unit (CRU) at the University of East Anglia were caught attempting (and probably succeeding to some degree) using the peer-review process to prevent the publication of papers that they deemed damaging to their campaign to promote the theory of anthropomorphic global warming. There can be no doubt that that's what they were up to, and they were quite explicit about it.

With respect to Michael Mann's Hockey Stick, his original papers in 1998 and 1999 were duly peer-reviewed and published in prestigious journals, but were soon comprehensively debunked (in my opinion) by Canadians Steve McIntyre and Ross McKitrick. The story of their efforts has been told in excruciating detail and remains controversial, but the Hockey Stick, while making quite a splash at first, has gone out of style, so to speak, the victim of a torrent of criticism from many directions.

Whatever the truth of the matter, a significant lesson learned from the experience of the Hockey Stick is how superficial peer review is. Apparently, McIntyre, a retired mining engineer well versed in statistical analysis, was the first person to look closely at Mann's results, close enough to raise very substantive questions about the data and methods of analysis. Also learned from McIntyre's work was that the availability of the underlying data and methods of analysis, supposedly required by the written policies of the scientific journals to be made available, may not be at all. It was only by the energetic efforts of the two Canadians, both outsiders to the closed community of climate science, to raise some real issues with the Hockey Stick.