Tuesday, March 20, 2012

The politics of peer-review: a case study

 
In small talks on conferences, once in a while you hear colleagues complain about negative reviews of their articles on climate submitted for publication in scientific journals; reviews which have an overtly political touch. Recently, Hans von Storch and I submitted a short meeting report to a leading science journal and received a review which we want to put up here on klimazwiebel for discussion. In our opinion, the reviewer demands revision of our article on basis of political and not scientific criteria. 

The editor of the journal confirmed the opinion of the reviewer and wrote: “In my view, two major revisions are acquired (sic). One deals with some statements that either need editing or deleted. Please see the enclosed review for suggestions on how to revise the text”. In order to have our piece published, we deleted the indicated statement, because it was not really relevant for the meeting report. But we did so in bewilderment; a similar statement about the same meeting already was published for example in a Nature editorial. While we were accused of supporting a politically motivated agenda, we have the feeling that at least in this case the review process itself is politically infiltrated. But please judge yourself:
 
The authors provide a more or less boiler plate meeting report that would otherwise be acceptable for publication (…) were it not marred by a few inflammatory and unsupported statements.

If the authors want to publish this in a premier, high-profile members journal such as (this one, W.K.), the offending language must be dealt with in a satisfactory manner.

The piece contains the innuendo-laden statement:

"It takes transparency, collaboration with other disciplines, and well-targeted information to regain public trust and to give adequate policy advice"

That statement betrays a biased and ill-informed perspective on the part of the authors. Saying that climate scientists need to "regain publish trust" is to buy into the flawed premise that it has been lost. While the purpose of agenda-driven attacks against climate science over the past two years no doubt had the intent of trying to undermine the public trust in the science, careful polling by experts such as John Krosnick of Stanford University show that there has been no erosion of public trust, despite the best attempts of climate change deniers and media outlets that have fostered the false narrative. See e.g. http://www.aaas.org/news/​releases/2010/0322climate.​shtml

The above statement also is built on the flawed premise that (a) climate science has not bee transparent. In reality, the climate science community has undertaken
measures over the past 10 years that go well beyond the degrees of disclosure found in other areas of science (consider for example the fact that the enormous archives of model output that serve as the basis for the IPCC model projections are available to the public, and many climate models can be downloaded for free). The statement also falsely implies that there is a lack of "collaboration" with other disciplines, when by many measures, climate science is the most multi-disciplinary and interdisciplinary areas in science today.

In short, the statement quoted above is so fundamentally flawed in its underlying assumptions that it must simply be eliminated for this article to be suitable for publication in (…).

The same is true of the later sentence:

"(Meeting participant's name, W.K.) demonstrated the difficulties of the Intergovernmental Panel on Climate Change (IPCC) in a situation where science is misused to depoliticize politics and in turn becomes politicized."

Who is (XXX, W.K), and why do readers need to be informed of his misguided views of the IPCC. The statement, again, must be stricken.

I believe the meeting report will be suitable for publication contingent upon the mandatory removal of the two offensive statements noted above.

7 comments:

Richard Tol said...

There are two things here, but neither is politicization of science.

The first comment is paternalistic: It is best not to mention certain things (trust, uncertainty) because it may be abused or misunderstood.

The second comment is exclusion: he who shall not be named shall not be named (we all know who he is anyway).

As this is exactly what the meeting was about, I would reformulate rather than delete.

Replace "regain trust" with "maintain, perhaps regain trust". And so on.

There is no influence of a political agenda on the results (i.e., politicization) in the sense of "I sell wind power and therefore the Medieval Warm Period is a fantasy".

Stan said...

Apparently, the statements made at the meeting or your opinions of the statements made at the meeting are verboten. This isn't peer review. This is censorship.

Werner Krauss said...

Richard,

we had no problem in revising the article, and the revision was immediately accepted. Anyway, I think this case demonstrates that the political dimension of the climate debate deeply affected the inner working of climate science; peer-review should not be used as an instrument to "streamline" the image of climate science.

It's absolutely over the top to call a statement like climate science has to regain public trust "so fundamentally flawed in its underlying assumptions that is must simply be eliminated for this article to be suitable for publication...". It's only a small case, but - as mentioned above - there are colleagues who report similar experiences.

To give a full picture, here the first version we submitted to the journal:

Workshop report:
Postnormal Science: the case of climate science.
Hamburg, May 4th – 6th, 2011
Authors:
Werner Krauss, Hans von Storch (Helmholtz Zentrum Geesthacht)

"Climate research has long left the narrow confines of pure science and has entered the public arena. Experts from the cultural, social and natural sciences discussed the current state of climate science through the lens of “postnormal science” (Ravetz and Funtowicz 1993). Science turns postnormal when facts are uncertain, stakes high, values disputed, and decisions urgent. During the workshop, situations and practices in climate research were identified and discussed in order to provide a solid empirical basis for a more realistic definition of climate science.

In his keynote, Ravetz revised the state of art in climate science in the light of recent developments. Knowledge is easier accessible than ever before and has changed the relationship between experts and laypeople; from early on, climate science was overloaded with policy implications and strongly conditioned by culture. The participants agreed that it is no longer possible to act as if purely scientific values were adequate to set the research agenda. Instead, public and stakeholder participation helps to define research priorities. It takes transparency, collaboration with other disciplines, and well-targeted information to regain public trust and to give adequate policy advice (e.g. editorial 2011).

The “blogosphere” is an example where scientific practice and public participation merge in new ways. The question of whether or not blogs can serve as a supplement to peer review was critically discussed, while Pielke jr. demonstrated how blogging can become a part of the daily working routine. In many cases, climate blogs help to make the scientific process public and transparent.

Regional Climate Services were discussed as another mechanism to link climate research to private and public interests. The challenge for climate research is to reconcile information supply and user demands. Knowledge transfer is a dialogic process that has to take cultural values and perspectives into account (e.g. von Storch 2011).

These challenges on the regional scale are reflected on the global level. Beck demonstrated the difficulties of the Intergovernmental Panel on Climate Change (IPCC) in a situation where science is misused to depoliticize politics and in turn becomes politicized.
There was a common agreement among the participants that complexity and uncertainty management necessitates an interdisciplinary effort, including the social and cultural sciences. The workshop demonstrated convincingly that postnormal science is neither a magic formula to solve all problems nor does it mean the end of science; instead, it helps to adjust the concept of climate research to the complex realities it is already involved in.

References:
Funtowicz, S. O. and J. R.. Ravetz (1993) Science for the postnormal age. Futures, 25,
739 – 755.
Editorial (2011) Value judgements. The scientific endeavour needs to deliver
public value, nor just research papers. Nature, 473, 123 – 124.
Von Storch et al. (2011) Regional Climate Services. Zeitschrift für Umweltpolitik & Umweltrecht, 1/2011, 1-16.

eduardo said...

Werner,

similar comments were quite common a few years ago, but from what I have experienced they are disappearing in the physical/chemical area of climate research (this manuscript falls rather into the metaphysical, no insult intended here )

I recall comments like ' this sentence has to be removed because , although it is correct, it can be misused by skeptics.. ). Others were of the sort '.. this paper is typical of the von Storch group and thus I am not inclined to recommend publication'. So it seemed that Hans could even contaminate papers that he had not co-authored.

Part of the blame lies on the editors and not only on the reviewers, in my opinion. To be a good editor is hard and not all are up to the job.

Werner Krauss said...

Eduardo,

thanks for your comment. It puts peer-review and the immaculate image of science as "purely objective" into perspective.

I'll keep your comment; I'll cut it out, frame it, and put it on my desk.

And would you agree if I add that scientists know their reviewers and the sensibilities of the editors and thus sometimes practice kind of "anticipatory obedience" in order to get their stuff published? And that this atmosphere of anticipatory thinking tends to slowly corrupt science and has consequences for the framing of climate change in politics, for example?

Karl Kuhn said...

Werner,

this anticipatory obedience is exactly the problem that I see with the climate science industry. The catastrophic climate change narrative has be come the legitimatory basis for really huge amounts of research funding. This is sufficient to streamline reasearch. And I believe that this constitutes a specific form of corruption. Particularly young scientists without tenure cannot afford to swim against the mainstream without threatening their entire career.

Moreover, for many activist scientists the narrative serves as a source of personal purpose and moral superiority. These scientists are usually prominent figures and therefore often selected as reviewers, where they practice censorship. This is another form of corruption, the "good-cause corruption".

Sometimes I believe that in such highly politicized areas anonymous peer review no longer serves its purpose. The reviewers can easily find out about the authors, but themselves hide in anonymity, which makes it difficult to expose and discuss abuses. The whole climate science process should be made totally transparent. BTW, peer review is not necessarily anonymous, depends on the discipline!

Volker Doormann said...

As long as people have learned the timeless methods of science and have made science to their own consciousness politics have tried to argue in public with well known logical fallacies to govern scientists. What’s seems new is the inverted practice when science people claiming politics and politician claiming the use of the methods of science.

Politics is not compatible to science simple because of the different points of view. Politics will change the state of the art in general because of thinking the order of nature is wrong, but science simple has a motivation to recognize the order of nature. And because philosophy is the basis of science immaterial objects, like logic, math, geometry, harmony or truth vs untruth, science never can be politics; politics always looks to the near things, like nations, morality, or kings.

In the inverted practice of the science community especially in the so called climate science community arguing on the level of science has changed to a war of authorities, which was called just it became nuclear, while the term ‘climate science’ has no basis in science, because the term climate has neither a basis in physics not in science; it is term from the social society regarding different weather phenomena on different places or different ages without any knowledge about the nature of the whole heat system.

I think there are two different points on (peer) reviewers of papers. If it is obvious that in a paper it is argued on the level of politics and is therefore no science it must be rejected to hold sciences clean and holy. The other point is if there is an authority biased peer have known Sir Newton or Lord Kelvin personally and there comes new evidence conflicting with the state of the art, it becomes a problem to science, if the rules of the filter are based not in the science of philosophy but fame or proud: The Royal Academy of Science which were convinced by Sir Robert Ball has argued that communication with the planet Mars was a physical impossibility, because it would require a flag as large as Ireland, which it would be impossible to wave.

V.