- How robust are the conclusions in the AR5 Physical Science Basis report? Have the IPCC adequately addresses [sic] criticisms of previous reports? How much scope is there to question of the report’s conclusions?
- To what extent does AR5 reflect the range of views among climate scientists?
- Can any of the areas of the science now be considered settled as a result of AR5’s publication, if so which? What areas need further effort to reduce the levels of uncertainty?
In my view, the suggestion that a formal process can itself guarantee the absolute accuracy of any document and put its conclusions beyond question is a misrepresentation of the way science works. It is often suggested that, because they have been restricted to peer-reviewed sources and passed through multiple stages of review, the IPCC’s conclusions are beyond reasonable doubt. This is misleading, because it focuses attention on the process, rather than the evidence itself.
I don't think this is a convincing reply. Allen confuses absolute certainty (which you never will get in science) with the question if the knowledge is credible and trustworthy. How can this be achieved? Allen says not by procedure but by the evidence itself. But if this were the case we still would need to have someone to say what the evidence is, given that there are many research papers which come to (sometimes) different conclusions.
The reason why IPCC reports lack credibility among some observers is that it overstates what it can achieve. It says it uses a peer review mechanism and reviews only peer reviewed research papers--both claims are obviously not true. The IPCC is answering to the UNFCCC and governments appoint its reviewers. Scientific excellence is not the main criterion (in fact, the criteria are opaque), this would be the case if the Academies of Sciences were to appoint a review board. The IPCC does not perform own research but conducts an assessment and evaluation of existing work. Academically speaking this would be a systematic review. But the IPCC does not use this terminology and has no mechanism to ensure that such a review would be conducted by an independent team. Conclusion: if a proper procedure was applied, the results of such a review would be trustworthy.
Allen further notes:
The problem with IPCC’s response to criticisms of previous assessments is that the focus has been entirely on formalizing procedures, whereas the reports ultimately depend on the collective scientific judgment of IPCC authors and reviewers. In some respects, tighter procedures may even have a distorting effect, particularly when coupled with the infrequency of IPCC reports. For example, I am aware of two papers (on one of which I was a co-author) being accepted on the day of the deadline for inclusion in the IPCC 5 th Assessment, both of which provided evidence for a downward revision of the uncertainty range for climate sensitivity. A third paper, providing evidence for a higher value for climate sensitivity, was accepted two days later, and hence could not be included. The fact that the first two papers and not the third contribute to the official consensus view on climate sensitivity for the next six years is clearly absurd.
This is another red herring against proper procedure. If there are deadlines in a review process (in any review process) they will have to be enforced. The alternative would be to have ad-hoc assessments (perhaps as soon as a new paper is published?) which would just create a parallel structure (with a slight time lag) to published research. The question seems to boil down to an appropriate time frame--but not to question the procedure of systematic review.
BTW, Allen has some fairly critical remarks
about temperature reconstructions (compare this to Michael Mann’s earlier comments
who feels vindicated by the AR5 report):
Likewise, I can also confirm that the particular reconstruction of Northern Hemisphere temperatures over the past millennium (the so-called “Hockey Stick”), which subsequently came in for considerable criticism, was not in any way central to the conclusions of the 2001 Assessment regarding attribution of causes of recent warming. We were aware of these proxy climate reconstructions, but did not give them much weight in the attribution assessment because the statistical methods used were (and arguably still are) rather opaque. I remember specific discussions among the attribution chapter authors questioning the error budgets of those reconstructions, and concluding that it would be premature to rely on them too heavily. With the benefit of hindsight, these discussions seem remarkable prescient, and confirm the importance of scientific judgment in the IPCC process.
Posts
22 comments:
I read Myles Allen's submission briefly. What I took from it was that he was suggesting that it isn't possible to produce the perfect, error-free document. He was arguing, I think, that people should recognise this. Hence, the problem is that if someone wishes to find an error in an IPCC document, they are almost guaranteed to be able to do so (which would probably be true for any undertaking of this size). Hence, it is fairly easy to then create the impression that there are problems with the document and hence spread the story that people can't trust the IPCC.
So, my view, is that if people do indeed distrust the IPCC it's more because some have been doing their utmost to suggest that it can't be trusted, than because there is any intrinsic problem with the integrity of the IPCC.
That's not to suggest that it couldn't be done differently, but - I believe - most professional climate scientists would argue that it does a pretty good job of representing our current understanding of climate science and, if anything, is rather conservative.
The first issue Myles Allan raises is writing reports on 6-year interval:
I have recently criticized the IPCC practice of producing exhaustive Assessments on
a 6-year cycle because I did not feel it best served the interests of the governments whom IPCC is intended to inform, and represented an excessive burden on the scientific community.
Richard Tol wrote on the same issue
The IPCC process assesses scientific knowledge according to a political time-scale. That implies that parts of the literature are assessed too frequently while other parts of the literature are not assessed frequently enough. Instead of a mega-report every 6-7 years, it would be better to have an IPCC Journal with frequent updates where the literature moves fast and infrequent updates where little new is written.
Writing reports on such a schedule means also that they cannot be revised when errors are found after publication or when new scientific results change the understanding.
Many people have discussed this issue before, including myself. I hope that pressure will build up enough to result in essential changes in the way IPCC operates.
being a bit naive and assume for the moment that governments are interested in the outcome of the IPCC processes, shouldn't the government themselves decide how they would like the IPCC reports to be ?
So far the discussion about the IPCC reports has been led by scientist, but I am not aware of any government voice explaining any preference. Somewhat strange, since the reports, in theory, written for them.
Would more frequent reports or more frequent updates really help with the trust issue? Or would the IPCC then be accused of alarmism? (And IMHO: It might help the discussion if the point of this blog post was summarised perhaps briefly at the end of the post. Is it about issues related to evidence, procedure, scientific judgement, all of these...?)
Raffa
#1
"Hence, the problem is that if someone wishes to find an error in an IPCC document, they are almost guaranteed to be able to do so (which would probably be true for any undertaking of this size).
Hence, it is fairly easy to then create the impression that there are problems with the document and hence spread the story that people can't trust the IPCC."
I think it would be good to discuss examples. Otherwise we are left to speculate. There are many aspects of IPCC work (WG1-3; SPM; full reports, press releases; chairman's communications, etc.) and many different criticisms.
In very general terms, the Inter Academy Council has criticized the IPCC for lack of procedural rigour and suggested reforms. Would you also direct your critique (that it is "fairly easy to then create the impression that there are problems with the document") at them?
@ReinerGrundmann
Let me see if can rephrase what I was suggesting. Let me also make clear that I'm not suggesting that criticism of the IPCC procedures don't have any merit. I would be surprised if such a complex process couldn't be improved.
What I'm suggesting is that it seems virtually impossible to have a process that is beyond criticism and impossible to produce a document that is error free. Maybe I'm wrong, but it would seem reasonable that a perfect process is likely impossible.
Therefore I was suggesting that one should also consider the possibility that the lack of trust - if it exists - is partly because there are some who like to diminish the significance of the message presented by the IPCC and hence, given the complexity of the process, will always be able to find something to criticise.
So, let me ask a question of you. Do you think that what I'm suggesting is extremely unlikely or do you at least consider the possibility that some/much of the rhetoric associated with the lack of trust in the IPCC could be being manufactured by those who would rather it wasn't trusted.
Yes I do consider such a possibility ;-) but this is a lame statement. I think on this level everyone would agree. I becomes more interesting (and difficult) if you were to discuss specific cases.
@6
Yes, in a complex processes like the IPCC errors are probably unavoidable. The question is how those errors are handled. Is trust reinforced by trying to cover them up, as in the 4th Report, or by implementing better procedures ?
@ReinerGrundmann,
Well, I'm not sure why it's lame. The reason I mentioned it was that it seems few have done so before, and I find that a little odd. This is clearly an extremely contentious topic and choosing to, apparently, ignore one option seems a little odd - to me at least.
Also, I'm fairly new to this whole debate, but as far as I'm aware, we could discuss specifics. Maybe I'm misinformed, but aren't there a few fairly high profile people who participated in - what turned out to be unfounded - attacks on the IPCC and on people involved with the IPCC. I'm reluctant to mention names or specifics since I wasn't involved in the debate at the time and most of what I've heard is via blogs or comments from others. You, having been involved for longer, could presumably comment as to whether any such thing has happened in the recent past.
@eduardo,
Sure, I agree. How you build trust is important and, certainly, covering up an error is not the way to do it. My main point is not that the IPCC should not be criticised or that it's procedures should not be changed or adapted, it's that it would seem sensible to - alongside this - not only make clear that errors are unavoidable (which is what I interpreted Myles Allen's submission as suggesting) and that not all criticisms are well-founded.
"It is often suggested that, because they have been restricted to peer-reviewed sources and passed through multiple stages of review, the IPCC’s conclusions are beyond reasonable doubt."
Its frequently forgotten that the IPCC's conclusions are NOT restricted to peer-reviewed sources, as in fact their whole publication contains wads and wads of so called grey literature from NGO's with both a financial and moral axe to grind (never mind the real scientific research teams who have exactly the same dilema and are thus obviously going to be fighting confirmation bias the whole way).
That it passes for some kind of reliable scientific document is amazing.
"The reason why IPCC reports lack credibility among some observers is that it overstates what it can achieve. It says it uses a peer review mechanism and reviews only peer reviewed research papers--both claims are obviously not true."
What is "obviously not true" is that it does not use a peer review mechanism (just not the rather exclusive form that Ruth Dixon claims) and that it claims to use only peer reviewed research papers (no, a harsh literal interpretation of what Pachauri once said is not an IPCC claim). For the actual claim of the IPCC, see Appendix A to the "Principles Governing IPCC Work" (specifically sections 4.3.3 and Annex 2).
Then there's the complaint about the IPCC reports not being called "systematic reviews". This appears to be arguing about semantics. How does the term "assessment report" differ from "systematic review", other than in the wording? I would even argue against using the term "systematic review" because it is defined as a review of the literature relevant to a specific research question. The IPCC reports review multiple research questions and also considers interaction between those research questions. That makes it much more than a systematic review.
Just for argument sake, let's take the issues I identified above as a specific case of manufactured rhetoric. Makes the discussion more interesting and difficult.
Bam
@ReinerGrundmann
You say:
“If there are deadlines in a review process (in any review process) they will have to be enforced.”
As I describe in my submission to the Commons Committee, the WGI Lead Authors Lead drove a coach and horses through the AR4 SOD publication deadlines. WGI TSU had published a deadline and severely warned authors that references to literature breaking it would be removed. When Expert Reviewers asked for that rule to be enforced, to a man, Lead Authors and Review Editors agreed to a retrospective change of the deadline the from before the start of the final review process to one month after it was finished.
It is indisputable that that sole purpose was to firm up the Palaeoclimate chapter and defend the ‘hockey stick’. Then, the TSU had to be forced under the US FOIA law to release the Reviewers’ comments and the Lead Authors responses in an accessible form. The TSU plan was to dump one paper copy in a Harvard Library to join those of the TAR. Not all the Reviewers’ comments and Authors responses were released until the second tranche of Climategate emails which showed that a legitimate Expert Reviewer request to cite papers, which met the new deadline but undermined its purpose, was censored by the WGI TSU.
As to Myles Allen’s suggestion that the ‘hockey stick’ was not taken too seriously in the IPCC 2001 WGI Detection and Attribution chapter, just read section 12.3.2 for which he was one of the Lead Authors.
“If the real world internal variability on this time-scale is no greater than that of the models, then the temperature change over the last 140 years has been unusual and therefore likely to be externally forced. This is supported by palaeo-reconstructions of the last six centuries (Mann et al., 1998) and the last 1,000 years (Briffa et al., 1998; 2000; Jones et al., 1998; Crowley, 2000; Crowley and Lowery, 2000; Mann et al., 2000), which show that the 20th century warming is highly unusual. Three of the five years (1995, 1996 and 1998) added to the instrumental record since the SAR are the warmest globally in the instrumental record, consistent with the expectation that increases in greenhouse gases will lead to sustained long-term warming.”
The IPCC Process and its clear rules ar totally unsupervised by the governments that created it and pay its participants. It is unfit for purpose and long past its sell by date.
True, a report of such lengths will very likely contain errors, in the same way, as a FORTRAN code of 1 Mio and more lines, will have some bugs.
But here, we had the situation in 2007
- for WG I no significant errors were noted; thus, such a level of negligible level of errors is possible. (Note: error in assessing the scientifically legitimate know2ledge at the time of assessment.)
- for WG II several were found, which all pointed towards a too dramatic assessment; thus the distribution of glitches was asymmetric, which is not to be expected a-prior.
In my view, WGI had really done an excellent job (without me), and WGII makes use of the excellent and rightful reputation of WGI, while having delivered material of considerably less value.
In WG1 it's possible to use only information from papers published in journals that are followed regularly by scientists. That allows for peer review after the publication, and leads to open criticism of weaknesses in conclusions that are important enough to include in IPCC reports.
If the WG2 report would be based only on papers of similar quality, it would be much thinner that it is now. AR4 WG2 report was to a very significant degree based on work that had not been subject to scrutiny by qualified peers who have studied similar questions. Many reports used were not peer reviewed, and they were not studied critically after publication.
In addition the regional parts of the WG2 report appear to suffer from being influenced by regional interests of various type. Cases of conflict of interest have probably been quite common.
extra heat in the atmosphere is NOT accumulative. If for ANY reason the atmosphere gets warmer than normal-> troposphere (oxygen &nitrogen)expands extra and wastes that extra heat in a jiffy: http://globalwarmingdenier.wordpress.com/q-a/
@von Storch
danke für ihren Beitrag. Ich hatte denselben Eindruck über den Report 2007. Und die Fehler im WGII hätte man zum großen Teil vermeiden können, wenn die Kommunikation im IPCC-Prozess selbst besser gewesen wäre. (bestes Beispiel: Himalaya-Gletscher, WGI korrekt, WGII bzw III? nicht). Es bleibt dabei: es gibt keine bessere Zusammenfassung als den IPCC-Report, aber man kann die Zusammenfassung noch besser machen.
Was ich interessant fand: immer, wenn "Skeptiker" irgendwas behaupteten, angeblich neues fanden, usw., das nicht völlig abstrus war, dann war es auch im IPCC-Report diskutiert, insb. im WGI. Das war beeindruckend. Übrig blieben dann Kleinigkeiten und Abstruses. Und sorry, Abstruses muss nicht in solch einen Report. Und Kleinigkeiten kann man ohne Aufwand verbessern.
Anonymous #17, since you do not give an alias, I call you Paul.
Paul - I do not think that it is so easy with WG II in AR4. The fact that there was a bias towards dramatizing assertions in the errors of WG II, may cause the suspicion that also in other less obvious issues there is a latent bias of that sort. (Auf gut Deutsch - da könnte gut der Wurm drin sein.)
I would subscribe to your assertion only with respect to WGI: "es gibt keine bessere Zusammenfassung als den IPCC-Report, aber man kann die Zusammenfassung noch besser machen."
On the other hand, our "regional IPCC-effort" for the Baltic Sea Region, BACC, does not suffer from such a bias, at least until what has become known so far.
The problem is that the less good WGII report is hiding behind the reputation of the superior WG I report. Let us see, how the quality of the new (AR5) report of WGII, due in April 2014, will be assessed by the blogosphere - which is the only independent review body.
You seem also to try milking reputation for WG II from that of WG I. We know by now that the Himalaya-error was not just an innocent stupidity, but was used at the highest levels of IPCC for alerting to the climate change problem; prominent institutes tried to get funding on the basis of this claim.
The UK Parliament has launched an inquiry into the latest assessment report published by the IPCC.
Why? In order to start a kind of own review process? This would be a nice example of scientified politics in Gavin's words.
Or is it a kind of poll? Anybody who likes can present his own explanations? For the dustbin or a new paper by Lewandowski? What's the purpose?
Andreas
Andreas
sorry i must have overlooked your question. The reply can be found here:
http://www.parliament.uk/business/committees/committees-a-z/commons-select/energy-and-climate-change-committee/news/ipcc---tor/
And here is some coverage on Parliament TV, ith the following witnesses:
Professor Sir Brian Hoskins, Grantham Institute, Imperial College London, Professor Myles Allen, University of Oxford University, and Dr Peter Stott, Met Office
Professor Richard Lindzen, Massachusetts Institute of Technology, Nicholas Lewis, Climate researcher, and Donna Laframboise, Author
http://www.parliamentlive.tv/Main/Player.aspx?meetingId=14741
I agree with Mike Hulme's positions (see Werner Krauss' latest post). In my words:
Politicians and the public should be debating policy not the science.
Fracking? Nuclear power plants? Renewables? This inquiry won't give any answers to the more important questions.
Andreas
Post a Comment (pop-up window,non-moderated)