Tuesday, June 10, 2014

Misrepresentation of Bray and von Storch survey

On May 26, the Online version of the Wall Street Journal posted an opinion piece  titled ‘The Myth of the Climate Change '97%' What is the origin of the false belief—constantly repeated—that almost all scientists agree about global warming?’  It was written by Joseph Bast and Roy Spencer. The brief bio informs the reader “Mr. Bast is president of the Heartland Institute. Dr. Spencer is a principal research scientist for the University of Alabama in Huntsville and the U.S. Science Team Leader for the Advanced Microwave Scanning Radiometer on NASA's Aqua satellite.”  The posting contains the statement “Rigorous international surveys conducted by German scientists Dennis Bray and Hans von Storch —most recently published in Environmental Science & Policy in 2010—have found that most climate scientists disagree with the consensus on key issues such as the reliability of climate data and computer models. They do not believe that climate processes such as cloud formation and precipitation are sufficiently understood to predict future climate change.”

Since the time of publication, the opinion piece, or parts thereof, has been widely distributed, more often than not verbatim, in the blogosphere.

We, (Bray and von Storch) the authors of the ‘Rigorous international surveys, find this disconcerting.  Bast and Spencer make claims about our work, which are inaccurate if not outright false. This is not the first time that statements of us have been misrepresented – the case of Ameling in August 2013 was another blatant one; thus it may make sense opposing publicly such claims.

In the WSJ opinion piece, Best and Spencer inform the readers that based on the results of our survey; there is disagreement ‘with the consensus on key issues such as the reliability of climate data and computer models’. What IS the consensus on the reliability of climate data and computer models?  What are our results compared to so that such a claim can be made?

A second conclusion reached by Blast and Spence reads 'They [climate scientists]  do not believe that climate processes such as cloud formation and precipitation are sufficiently understood to predict future climate change'.  On this account, first, Blast and Spencer do not seem to have taken the time to read our discussion of the use of the term prediction versus the use of the term projection, a paper which itself raised a lot of comments, i.e. prediction and projection, in climate science, are synonymous. 

But the point here is the claim of correlation between the understanding of clouds and the ability to assess the future of climate.  What the survey asks is how well the respondent thinks that climate models can deal with clouds.  Admittedly the confidence is fairly low.  However when asked ‘Concerning TEMPERATURE VALUES, how would you rate the ability of GLOBAL models to simulate mean values for the next 10 years’,  (and the next 50 years)’ the response was considerably more positive than that assigned the assessment of clouds. The correlation of the two variables was a minimal .09.

The point being, our results are modified in a way so as to support the opinions of many blog authors, not just Bast and Co. This amounts to manipulations, and to damaging our academic reputation. We, Hans von Storch and myself would like to clarify that, in undertaking the surveys, we attempt to produce results that are as objective as possible.  It is not our intention to provide fodder for this camp or that camp, we are not acting as partisans of any particular persuasion and we do not particularly appreciate being stuck with a label assigned by some third party imagination.

36 comments:

  1. Hans von Storch fand es nicht schlimm, wenn Lobbyisten wie Vahrenholt die Ergebnisse anderer autoren falsch und völlig verzerrt darstellte, warum findet er es nun schlecht, wenn Lobbyisten wie Bast und Spencer seine Arbeiten falsch und völlig verzerrt darstellen?

    Das ist schwierig zu verstehen...

    ReplyDelete
  2. The most interesting and disturbing result of the Bray/von Storch Survey was the incoherence of the answers by climate scientists. The confidence in the model results was generally high, while the confidence in the reliabilty of certain key climate processes was low. Dennis Bray himself quotes the perhaps most blatant incoherence in the above post:

    "What the survey asks is how well the respondent thinks that climate models can deal with clouds. Admittedly the confidence is fairly low. However when asked ‘Concerning TEMPERATURE VALUES, how would you rate the ability of GLOBAL models to simulate mean values for the next 10 years’, (and the next 50 years)’ the response was considerably more positive than that assigned the assessment of clouds. The correlation of the two variables was a minimal .09."

    The incoherence carries a number, 0.09. How can climate scientists believe that models can well simulate future temperatures when admitting at the same time that models can't deal with clouds well enough? Clouds are crucial and not just negligible noise.

    It would have been completely sufficient by Bast and Spencer to point out this obvious cognitive dissonance, but maybe they were to lazy to plough through the results.

    ReplyDelete
  3. nobody,
    Ihre Behauptung "Hans von Storch fand es nicht schlimm, wenn Lobbyisten wie Vahrenholt die Ergebnisse anderer autoren falsch und völlig verzerrt darstellte" überrascht mich etwas. Wann und wo soll ich das denn so ausgedrückt haben? Mögen Sie meinem Gedächtnis nachhelfen?
    Sie hätten diesen Kommentar ja auch schon bei dem Konflikt mit Ameling anbringen können.

    ReplyDelete
  4. @von Storch

    Im UBA-Thread. Sie schrieben auf die behauptung, dass Vahrenholt Ergebnisse falsch darstellt und missbraucht:

    Mißbrauch" - das ist auch so eine Sache; wer bestimmt, was richtiger Gebrauch ist, was Mißbrauch? Sicher nicht der Urheber der wissenschaftlichen Erkenntnis

    Nun ja.... allerdings haben sie auch geschrieben, dass Vahrenholt nicht immer korrekt war und doch auch Cherry pickte.


    Ich sehe ihre Aussage da schon sehr im Widerspruch zu dem Artikel heute.

    ReplyDelete
  5. Nobody, nun lassen Sie uns das mal durch-xen.

    Es geht um ihre Formulierung "falsch und völlig verzerrt darstellte" und nicht um " falsch darstellt und missbraucht". Was hat er "falsch und völlig verzerrt" dargestellt, oder handelte es sich mehr darum, daß er die Ergebnisse in einem anderen Kontext brauchte, in Ihrem Wort: "mißbrauchte", als es die Autoren intendiert hatten? Mögen Sie sagen Beispiele angeben, wo Vahrenholt "falsch und völlig verzerrt" dargestellt hat?
    Tatsächlich meine ich - durchaus im Merton'schen Sinne - daß die Verwendung von wissenschaftlichen Einsichten nicht unter dem Vorbehalt der Zustimmung des Erzeugers steht.
    Die Frage steht dann wirklich im Raum - wer bestimmt, was Mißbrauch ist?

    Das ich Vorbehalte gegen die Argumentation von Fritz Vahrenholt habe, habe ich durch meinen Beitrag "A skeptic lacking skepticism: Fritz Vahrenholt" hier auf der Klimazwiebel ziemlich deutlich gemacht, dachte ich jedenfalls.

    Ärgerlich wird es für mich, wenn jemand wissenschaftliche Aussagen falsch wiedergibt, was im Falle von Bast & Co eben der Fall war. Dann geht es um die Aussagen, nicht um die Folgerungen, die man daraus zieht.

    Karl Kuhn hat für mich recht - wenn man geschrieben hätte: "Klimaforscher trauen den Modellen wenig zu im Hinblick auf die Beschreibung von Wolken, sagt der survey; man weiß das Wolken ein zentraler Aspekt der Klimasensitivität ist, insofern müßte die Mehrheit der Klimaforscher skeptisch den Modellen und ihrer Fähigkeit gegenüber stehen, den
    Temperaturwandel zu beschreiben.", dann wäre das ärgerlich, weil das Resultat zum Zutrauen zum Beschreiben des Wandels unterschlagen worden wäre, aber man könnte vielleicht zu ihren Gunsten annehmen, daß sie dies Detail des Surveys nicht mitbekommen haben. Man hätte natürlich auch einfach uns fragen können, ob wir meinen, daß die Darstellung in WSJ sachlich in bezug auf den Survej korrekt ist.

    ReplyDelete
  6. Joseph Bast has written frequently about how Bray and von Storch take pains to avoid admitting the degree of uncertainty their surveys find.

    See: http://heartland.org/policy-documents/analysis-new-international-survey-climate-scientists-0

    And: http://heartland.org/policy-documents/scientific-consensus-global-warming

    As well as the foreword to this hefty tome: http://www.nipccreport.org/reports/2011/2011report.html

    ReplyDelete
  7. @ Jim Lakely

    I looked at the first of the links you provided. In the Report, Bast states:

    "The abstract of that article (presented on page 12 of the current report)seems to completely miss the obvious implications of the survey. No surprise,since Bray and von Storch tried to hide the plain results of their past work in a similar fashion."

    We did not try to hide any results, all of the survey results are freely available on line. What we didn't do was claim this means this or that means that - often conclusions drawn based on a normative judgment. We thought it best to leave such work for others. It is amusing that Bast accuses us of hiding what could be considered fodder for the skeptic canons while his antithesis accuses us of cherry picking to support the skeptic camp. In reality, we simply present the data without normative comment.

    ReplyDelete
  8. @Dennis Bray

    What we didn't do was claim this means this or that means that

    I have been wondering about this since I read one of your surveys for the first time: Why did you not write an interpretation, as is commonly done in scientific studies? The original authors' interpretation is usually particularly valuable, for obvious reasons. Also, in a loaded topic like here, some obvious interpretations, which you think are wrong, could have been anticipated and pre-empted with some authority. "Avoiding fodder for the deniers" is one thing, but pulling out of the necessary work half-way is something different.

    ReplyDelete
  9. I fully agree with hvw.

    ReplyDelete
  10. @HVW & KK

    There have been a number of publications resulting from the surveys. Mostly, however they focus on less controversial topics. I am sure you are aware of the difficulties publishing material that offends the status quo. To that end, the results are available for those who chose to do so. Each release of the survey results has been akin to throwing toys into the middle of a kindergarten and watching the kids argue and fight over them.

    ReplyDelete
  11. @HVW & KK

    There have been a number of publications resulting from the surveys. Mostly, however they focus on less controversial topics. I am sure you are aware of the difficulties publishing material that offends the status quo. To that end, the results are available for those who chose to do so. Each release of the survey results has been akin to throwing toys into the middle of a kindergarten and watching the kids argue and fight over them.

    ReplyDelete
  12. From Peter Manley in Green Valley News

    (http://www.gvnews.com/opinion/letters_to_editor/in-my-view-climate-change-i-m-a-denier/article_0a26e934-f0fe-11e3-b873-001a4bcf887a.html?mode=jqm)

    we learn:

    “In a May 26, 2014, Wall Street Journal opinion piece, The myth of Climate Change ‘97% by Joseph Bast and Roy Spenser the authors debunk that stat with strong arguments against the findings of four opinion pieces published in Nature, EOS, Proceedings of the National Academies of Science and in the Environmental Research Letters. The article also references the work of Dennis Bray and Hans von Storch published in Environmental Science & Policy that found ‘most climate scientists disagree with the consensus on … the reliability of climate data and computer models.’”

    Reading that, to whom would you attribute the last statement, Bast and Spencer or Bray and von Storch? I can assure you, it was not Bray or von Storch. ... a different problem of attribution.

    ReplyDelete
  13. "watching the kids argue and fight over them."

    So at least you had your fun ;-))

    ReplyDelete
  14. Na ja, wer hätte von Mr. Bast als Chef des Heartland-Instituts denn etwas anderes erwartet? In dieser Beziehung ist man ja quasi Serientäter, ich erinnere nur mal an die NIPCC-Berichte.

    Warum überhaupt darüber schreiben? Ach ja, weil aus irgendwelchen unerfindlichen Gründen der Unsinn des Heartland-Instituts im WSJ verbreitet wird, millionenfach gelesen. Die Richtigstellung erfolgt in einem kleinen, von der breiten Öffentlichkeit nicht beachteten Blog, hätte man sich daher auch fast sparen können. Heartland, GWPF, so läuft eben professionelle PR, ohne die Verbreitung durch spezielle, hilfreiche Medien liefe da gar nichts.

    Was für mich das Rätselhafte ist:

    Was bringt einen Dr. Roy Spencer dazu, gemeinsam mit "Billboard-Bast" einen Artikel zu verfassen? Auch eine (wenig) genutzte Möglichkeit, seinen Ruf unter Kollegen zu ruinieren, Erinnerungen an Bengtsson werden wach.

    Aber was soll's, wir sind tolerant, daher kann jeder seinen Ruf ruinieren, wie es ihm beliebt.

    Andreas

    ReplyDelete
  15. @Andreas: "Heartland, GWPF, so läuft eben professionelle PR, ohne die Verbreitung durch spezielle, hilfreiche Medien liefe da gar nichts."

    Da könnten deutsche Ableger eine Menge von lernen. Oder anders ausgedrückt, man sieht wie di­let­tan­ten­haft Pressearbeit hierzulande bei deutschsprachigen Skeptikervereinen abläuft. Und das finde ich gut so. ;) Selbst Heartland wird da keine Lust haben, sowas von außen zu sponsern.

    ReplyDelete
  16. My personal conclusion from reading the survey pdf itself was that there is a pretty hefty majority of climate scientists who agree with most or all details of the IPCC consensus. Then there were still some who questioned various aspects, maybe a sixth or less?
    One thing that was never published (as far as I know) was cross-tabs on the "dissenters": is it the same scientists who are taking the skeptical position on various different issues, or are they different people? This is especially important because the IPCC consensus depends on a number of critical points. Disagree on any of them and you are like to oppose mitigation.

    ReplyDelete
  17. @Mike R
    Just some clarification here. It is a little difficult to determine what you mean by 'dissenters'. If you look at the structure of the questions, typically a value of 1 would mean total disagreement and a value of 7 would mean total agreement. A dichotomous separation, the Yeas and the Nays, is a little too simple to capture the details. What the surveys capture is the degrees of disagreement that respondents have concerning the level of understanding of different physical processes. Could you be a little more precise as to what you mean by crosstabs on dissenters? I would be happy to prepare one or two tables accordingly.

    ReplyDelete
  18. @Dennis Bray. This is far from an area of expertise for me; others could answer the question much better. But if you want to allow others to analyze the data in an effective way, you should publish a simple table
    Respondent id Answer to Question 1 Answer to Question 2 ... Answer to last question
    1 6 5 ... 3
    2 6 1 ... 2
    3 7 5 ... 3
    Tab delimited or such. No personal information included.
    That way, I or anyone could try various types of exploratory analysis to see which fields are correlated in interesting ways.

    ReplyDelete
  19. Dr. Bray,

    I take your points, but suggest copyediting your post, for misspellings:

    Blast and Spencer,
    Best and Spencer
    Blast and Spence
    Etc

    ReplyDelete
  20. I meant, by the way, no _identifiable_ person information included. Any demographics you have like number of publications, or specific sub-field, or country, might be very useful in analysis.

    ReplyDelete
  21. MikeR,

    the idea is that the authors can watch the kids argue and fight over their results - not risk having them taken apart and looked inside :).

    But seriously, the raw data would be interesting to look at, indeed.

    ReplyDelete
  22. Mike R is looking for cross tabs aka contingency tables, a basic result of any multiquestion survey

    ReplyDelete
  23. Eli has come to the conclusion that a great deal of the problem with public (e.g. media) discussions of AGW is rooted in attempts to convey subtle points of agreement, disagreement and bafflement. That, and some not so honest brokers, has lead to your current situation. Therefore, he has brought forward a simple explanation of the problem suited as a starting point for discussion

    Increasing CO2 in the atmosphere at current rates will increasingly cause bad things to happen. Over the next one to two hundred years this will lead to VERY BAD things happening.

    The rest is detail.

    Thanks!

    ReplyDelete
  24. @ Mike R
    I would be happy to send the raw data and the code book to anyone that makes a request. Simply send the request by email: dennis.bray@hzg.de. I don’t have time, however, to calculate crosstabs for the entire data set, not at this time, anyway.
    It is not possible to identify a ‘person’ from the information contained in the data set.
    @hvw
    Again, a misinterpretation. What I said was ‘Each release of the survey results has been akin to throwing toys into the middle of a kindergarten and watching the kids argue and fight over them.’ I never said that was the ‘idea’ of conducting the surveys. It is simply what has happened. To obtain the raw data, simply send a request by email: dennis.bray@hzg.de.
    @Eli
    And if people disagree with your conclusion?

    ReplyDelete
  25. It would be possible to post crosstabs of variables on Klimazwiebel. Please make requests limited to only a few pairs of variables. Please look in the survey reports to obtain the exact variables you wish to include. I will limit my response to the fist four requests.

    ReplyDelete
  26. Thank you so much Dennis for making the data available so quickly.

    "Again, a misinterpretation"
    That was tongue in cheek, should have used ";)" instead of ":)" to be more clear ;).

    So what is DB and HvS's result regarding Eli's Consensus? It took Eli a while to come up with his definition, so it could not be directly asked in the survey. But we come pretty close, I guess:

    It's happening
    (26. How convinced are you that climate change, whether natural or
    anthropogenic, is occurring now?)

    We are causing it
    (27. How convinced are you that most of recent or near future climate change
    is, or will be, a result of anthropogenic causes?)

    It's going to be very bad, if business as usual continues
    (28. How convinced are you that climate change poses a very serious and
    dangerous threat to humanity?)

    Since Eli's Consensus is expressed in quite certain terms, we count the respondents that have answered 6 or 7 on a convincedness scale from 1 (not at all) to 7 (very much) to all three questions.

    54.7% (n=274) of the climate scientists agree !


    Note that exactly nothing would change in the grand scheme of things if this number was 35% or 94%.
    Hoping to have settled the consensus topic for good now, we can proceed to more interesting issues, such as climate scientist's trust in models.



    ReplyDelete
  27. @hvw

    Re the climate scientists' trust in models. It would be much more productive if such results are not used as fodder for either of the extreme positions, but put to practical use to determine research priorities as identified by the science community. This was the initial intention.
    As for 'settling the consensus topic',... hmmmm! Good luck with that one.

    ReplyDelete
  28. @hvw
    PS - you might also want to look at this paper

    https://www.academia.edu/4929792/An_Alternative_Means_of_Assessing_Climate_Models

    ReplyDelete
  29. hvw
    You can't just count all 3 questions with the same value.
    Without looking at anything else it seems fair to say that an objective set of answers could easily find:
    26. Everyone is certain that climate change wheher natural or not is occurring - because climate is always changing. Well duh! But the real question needs to be is it happening at an abnormal rate, which would garner less certainty.
    27. About half are likely utterly convinced that man has caused some of it. ie a mere toss of the coin - since the only thing that can find manmade warming are the models and these diverge from reality. There is no actual empirical evidence for manmade warming (indeed in any normal field people would notice that the official IPCC fingerprints for manmade warming are all missing).
    28. A small minority (say 10%) may think this warming might be very bad. Well there are doomsayers everywhere! (the unfortunate thing is that this minority get all the headlines).

    So voila; 53% achieved, but only 10% think we need be very worried and if question 26 was properly worded then the total percentage for the 3 questions could drop to 37%.

    Of course if there were an objetive recognition that climate change was natural then there would be no need for any research. There is thus an inbuilt bias from all climate scientists just in preserving their own jobs which must skew the percentages upwards. There is no estimate or adjustment for that bias.

    ReplyDelete
  30. Thanks! Again, it's out of my range of competence, but I have crossposted your offer on climateaudit, judithcurry, and Lucia.

    ReplyDelete
  31. @hvw. "we count the respondents that have answered 6 or 7 on a convincedness scale from 1 (not at all) to 7 (very much) to all three questions.
    54.7% (n=274) of the climate scientists agree !"
    Hvw, how do you know that 54.7% agree on all three questions - is that from the raw data? How could you tell that from the displayed results, where it's theoretically possible that none agree on all three?

    "exactly nothing would change in the grand scheme of things if this number was 35% or 94%." I can't agree with that, and I don't think John Cook would either. They are insisting on their absurd 97% consensus for a good reason.
    The reason is, 97% means, "All competent scientists in the field agree on this, except for a few crazies you can ignore. This is scientific fact." That's what a real scientific consensus looks like, more or less similar to the theory of evolution or Special Relativity.
    57% or 80% is very very different. We have a different name for that in science. We call it a open question.

    ReplyDelete
  32. Last phrase should have been, "57% or even 80% is very very different."

    ReplyDelete
  33. This comment has been removed by the author.

    ReplyDelete
  34. The full raw data set in Excel format with code book is now available at https://www.academia.edu/7454421/CliSci_2013_Data_Set_Excel_Format_with_code_book. The report of descriptive statistics can be found at
    https://www.academia.edu/5211187/A_survey_of_the_perceptions_of_climate_scientists_2013
    It would be appreciated if any user would share their findings on this blog.

    ReplyDelete
  35. You apparently sent out a new version. A number of people have voiced criticisms on twitter

    ReplyDelete