Trust in science and the role of proper procedure

The UK Parliament has launched an inquiry into the latest assessment report published by the IPCC. Submissions were invited and have been published on its website. Submissions were asked to comment on a number of questions, such as

• How robust are the conclusions in the AR5 Physical Science Basis report? Have the IPCC adequately addresses [sic] criticisms of previous reports? How much scope is there to question of the report’s conclusions?
• To what extent does AR5 reflect the range of views among climate scientists?
• Can any of the areas of the science now be considered settled as a result of AR5’s publication, if so which? What areas need further effort to reduce the levels of uncertainty?

I had a quick look at some of the replies. The contribution by Myles Allen caught my eye as he raises important questions about the process of knowledge production in the IPCC, especially the review process. He states:

In my view, the suggestion that a formal process can itself guarantee the absolute accuracy of any document and put its conclusions beyond question is a misrepresentation of the way science works. It is often suggested that, because  they have been restricted to peer-reviewed sources and passed through multiple  stages of review, the IPCC’s conclusions are beyond reasonable doubt. This is  misleading, because it focuses attention on the process, rather than the evidence  itself.  

 I don't think this is a convincing reply. Allen confuses absolute certainty (which you never will get in science) with the question if the knowledge is credible and trustworthy. How can this be achieved? Allen says not by procedure but by the evidence itself. But if this were the case we still would need to have someone to say what the evidence is, given that there are many research papers which come to (sometimes) different conclusions. 

The reason why IPCC reports lack credibility among some observers is that it overstates what it can achieve. It says it uses a peer review mechanism and reviews only peer reviewed research papers--both claims are obviously not true. The IPCC is answering to the UNFCCC and governments appoint its reviewers. Scientific excellence is not the main criterion (in fact, the criteria are opaque), this would be the case if the Academies of Sciences were to appoint a review board. The IPCC does not perform own research but conducts an assessment and evaluation of existing work. Academically speaking this would be a systematic review. But the IPCC does not use this terminology and has no mechanism to ensure that such a review would be conducted by an independent team. Conclusion: if a proper procedure was applied, the results of such a review would be trustworthy. 

Allen further notes:

The problem with IPCC’s response to criticisms of previous assessments is that the focus has been entirely on formalizing procedures, whereas the reports  ultimately depend on the collective scientific judgment of IPCC authors and  reviewers. In some respects, tighter procedures may even have a distorting  effect, particularly when coupled with the infrequency of IPCC reports. For  example, I am aware of two papers (on one of which I was a co-author) being  accepted on the day of the deadline for inclusion in the IPCC 5 th  Assessment, both  of which provided evidence for a downward revision of the uncertainty range for  climate sensitivity. A third paper, providing evidence for a higher value for  climate sensitivity, was accepted two days later, and hence could not be included.  The fact that the first two papers and not the third contribute to the official  consensus view on climate sensitivity for the next six years is clearly absurd.

This is another red herring against proper procedure. If there are deadlines in a review process (in any review process) they will have to be enforced. The alternative would be to have ad-hoc assessments (perhaps as soon as a new paper is published?) which would just create a parallel structure (with a slight time lag) to published research. The question seems to boil down to an appropriate time frame--but not to question the procedure of systematic review.

BTW, Allen has some fairly critical remarks about temperature reconstructions (compare this to Michael Mann’s earlier comments who feels vindicated by the AR5 report):

Likewise, I can also confirm that the particular reconstruction of Northern Hemisphere temperatures over the past millennium (the so-called “Hockey Stick”), which subsequently came in for considerable criticism, was not in any  way central to the conclusions of the 2001 Assessment regarding attribution of  causes of recent warming. We were aware of these proxy climate  reconstructions, but did not give them much weight in the attribution  assessment because the statistical methods used were (and arguably still are)  rather opaque. I remember specific discussions among the attribution chapter  authors questioning the error budgets of those reconstructions, and concluding  that it would be premature to rely on them too heavily. With the benefit of  hindsight, these discussions seem remarkable prescient, and confirm the  importance of scientific judgment in the IPCC process.