Monday, March 25, 2013

Excitement around Climate of the Past


There seems to be some excitement surrounding the paper by Lüdecke et al , recently published in Climate of the Past. I was the handling editor, and here I would like to clarify some points. As handling editor, I cannot obviously reveal details about the review process itself, which is only accessible to the chief-editors of the journal.

Climate of the Past is an open review journal.


 The review process has two phases: the open phase, in which nominated reviewers (anonymously if they wish) and the public in general (non-anonymously) can submit comments on manuscripts. The authors may be then invited to submit a revised version, which is again evaluated by those same and/or additional reviewers. This second phase is not public. When a manuscript is submitted to Climate of the Past, call #1 is sent to the editors that match the key words provided by the authors. In no editor picks the manuscript the circle of editors that are called is subsequently widened. The last call is call #5 in which all editors receive a request to be handling editor. I picked the Lüdecke et al manuscript after call #4. It was quite obvious that there was no great excitement among editors to handle this manuscript, and I knew that the reason was probably that it could be controversial. Nevertheless, it was my area of expertise, and I decided to accept being the handling editor.

Manfred Mudelsse submitted a public comment to the manuscript by Lüdicke et al. in the open-review phase. Later, after the manuscript was accepted, after rounds of versions, he complained by email to Climate of the Past and publicly in Georg Hoffmann's blog that he had not been sent the revised versions for further review by him. He has also withdrawn his support for Climate of the Past as reviewer of any future manuscripts. Somewhat surprisingly, neither he nor Georg have cared to clarify a point that for me is quite relevant in this story: Manfred Mudelsse was invited to be an official reviewer of this manuscript. He did not accept the invitation. He has offered two versions of the reason why he did not accept: on February 26th , 2013 he explained that 'I rejected invitation to review owing to time shortage but then decided later (after reading the blogosphere) to join as guest commentator'); however , on September 27th 2012 he had written to me ' Lüdecke et al.: ich hatte gar nicht gemerkt/erinnert, dass ich als Referee vorgeschlagen war (oder gar das Review abgelehnt habe)'.
Why did Manfred forget to write, after almost 200 comments in Georg's blog,  that he had rejected the nomination as official reviewer, and then only later when he become 'animated' by comments in Georg Hofmann's blog decided to,yes, become an official reviewer ?

When he published his guest comment in the open-review phase, I inquired why he had not responded to the official invitation, given that he did have an interest in reviewing the manuscript. Since I had nominated him in the first place, I tried to get him on board as official reviewer, but the journal rejected this possibility, and a posteriori I think this was a wise decision: in the moment he wrote a guest comment, he was  contaminated, at least in my opinion, as an official reviewer, as it has become a priori clear that his review was going to be negative. Thereafter, I tried to nominate a reviewer that could replace him in his area of expertise: 4 further nominations were not accepted or not answered.

There were a few more public comments on this manuscript, most of them critical. However, as editor, I could not consider all these comments as independent reviews, since all of them stemmed from commentators of Georg Hoffmann's blog. I have a strong opinion that reviews of a paper have to be independent. One of the most negative breaches of the review process that we have seen pictured in the Climate Gate emails was the attempt by reviewers to agree on negative reviews, with the stated goal of bringing down a particular manuscript. As editor, my response is to give less weight to those comments that I do not consider to be independent, or to consider them as stemming from a single commenter. This is my subjective position, but I thing it is quite defensible. As a general comment, I think this is one of the weakness of the open-review process: the possibility to orchestrate a series of negative reviews that may have been prepared in collusion.

Many of the public comments on the blogosphere regarding the review process of this manuscript have been somewhat dismissive to the remaining two reviewers. I chose them, and I would have chosen them again. They are competent  scientist that regularly publish in climate research journals. They may err as any one of us, but the are respected experts in this area of research, and read with care the subsequent versions of the manuscript and the public comments . They completed their reviews in good faith. Their reviews of the final version were fairly positive. To reject the manuscript at this stage, as Manfred and others may have wished, would be equivalent to whimsically dismissing the work of these 2 reviewers throughout almost 6 months of review process. At the very least, these reviewers are not less respected than Manfred Mudelsee, and definitively more respected in the area of time series analysis that all identified commentators that I could read in Georg Hofmann's blog What Manfred is requiring here is twofold: first that Climate of the Past breaches its own rules and sends the revised manuscript to a non-official reviewer for further review; second, that his opinion is given more weight that those of these two reviewers, a position that I honestly find not defensible. Any editor has been confronted by the situation in which reviews of a manuscript diverge, and he or she has to take a (subjective) decision, unavoidably disappointing one of the reviewers. If we all would react as Manfred has done, the review process could not possible.

All in all, in my subjective opinion, the review process in this case went according to all rules of Climate of the Past, and the Chief Editors have access to its entirety. All steps and decisions taken are documented and are traceable.

As I wrote to Geog and Manfred, if they do not agree with this particular paper, what they should do is to write a rebuttal paper. And more importantly, in general, they should not willingly omit relevant parts of the truth when they are criticizing some other person. There is clear adjective for this type of behaviour that I would prefer to omit here.

17 comments:

Richard S J Tol said...

This new paper makes the same mistake as some previous ones by the same authors.

In the first step of the analysis, they remove the trend.

Their conclusions, however, are about the trend they never analyzed.

Hans von Storch said...

Thanks for this clarification, Edu. Interesting, that a blog can set such a swarm-like directed action in motion. Certainly something which must be taken into account, when assessing the merits of the review process of CP.

Anonymous said...

Eduardo,

I've read the Lüdecke discussion paper before the discussion started and even as a layman I saw some of the flaws mentioned later in the review process. So I followed for the first time an open review process. Ref#1's response disappointed me, but all other responses gave me the feeling the people involved invested a lot of work and wrote in a very constructive manner. I think open peer review is at its best if people from outside contribute, so thanks for Manfred, Karsten and Oliver for engaging.

IMHO your final response was very constructive, too. Well written, very constructive. The only thing I wondered was, that noone raised a discussion about the quality of the raw data.

But after your final response I had to learn, that open peer review is not so open as I naively thought. Some day there appeared the finalised paper with some cosmetic changes, but omitting most of the suggestions the reviewers and you made. Why? And don't you understand, that this procedure in the can lead to some frustration? I think, the main problem is, that open peer review wasn't open at all in the end.


Another point:
"It was quite obvious that there was no great excitement among editors to handle this manuscript, and I knew that the reason was probably that it could be controversial."

Really? I think, it's normal to handle a bad paper. Let's speak clearly: The paper is not "controversial", it's bad, and the only question is if it's too bad to be published. Sounds quite normal. Not usual but rather "controversial" are the authors. Skeptics affiliated with the german blog EIKE. This implies some politication, if you want or not. If you reject this paper, it's probable you get involved in a skeptic shitstorm about "excluding" skekptic papers. Now there's a dimension beyond science, maybe some editors would think about accepting a bad paper, because he wants to show that skeptic concerns are not excluded.

PS:
Ravetz wrote a lot about extended peer review. I think he could be satisfied with this example at Georg's blog Primaklima:
http://scienceblogs.de/primaklima/2012/09/19/primaklima-review-von-ludeckehempelmannweiss-multi-periodic-climate-dynamics-spectral-analysis-of-long-term-instrumental-and-proxy-temperature-records/

Andreas

eduardo said...

Andreas,

I think this is not what we are discussing here.

You assert that the paper is very bad and it should not have been published. Whereas I respect your position, you have to consider that this happens to many papers that are indeed published. I have heard the very same opinion from different independent (this time truly independent) sources about the recent Marcott et al. paper: people wondering how it is possible hat that such paper could have possibly been published. But I am also convinced that the reviewers did their job as best as they could. There are always disagreement about the quality of papers that get published, and this is why journals have set up certain rules that should be abide by. If those rules did not exist, an editor could accept or reject paper at will.

The problem is the interpretation we give to published works and our need for drama and hype. A published paper is just a study that has been vetted by a few reviewers. It does not mean that it is correct. It is published only to be confirmed or refuted (or simply ignored) by the wider community over the following years.

The rules of the peer review of this manuscript have been abide by at all times, as the chief-editors of Climate of the Past have already confirmed in writing some weeks ago (something that Georg and Manfred also preferred to silence, by the way). Within these rules, the editor has a role, the reviewers have their role, and the authors play their role as well.

The rules of the open and closed phases as as they are - set by the journal. Here, some persons wanted to have those rules to be changed in this case - or these rues were not precisely known.


'If you reject this paper, it's probable you get involved in a skeptic shitstorm
about "excluding" skekptic papers.'

This is merely your speculation. Probably you are not used to deal with persons that do not act with their own political position in the back of their minds.


' Not usual but rather "controversial" are the authors. Skeptics
affiliated with the german blog EIKE. This implies some politication, if you want or
not. '

This is a quite revelling comment from you. This means that you are already prejudging a manuscript by its authors, not by its content. This is precisely what never ever, under no circumstances, should be done. According to your view, all papers written by members of Greenpeace should be rejected right away ?


'And don't you
understand, that this procedure in the can lead to some frustration?'

Yes, I can understand it. I am reviewers of a few tenths manuscripts per year. The editors sometimes follow my advice, sometimes they dont. Most of the times, I really do not know what their final decision was.I feel that I have done my job with my review and and it is their turn to do theirs with their decision. When 10-years-old get frustrated, they stop playing or sie halten die Luft an. Grown-ups, on the contrary, understand that they are not the centre of the universe and move on.

Gator said...

So because some reviewers taking part in a public review process publicly talk about a paper, that makes their comments suspect? Stolen emails from a decade ago set the editorial policy?

A robust review should act to improve a paper if the authors and editors act in good faith. It looks like in this case the authors and editors treated this paper as just another shot in the climate wars, a political paper, rather than a scientific paper.

Günter Heß said...

Dear Mr. Zorita,

It is necessary to question peer-reviewed literature.
For my opinion, science advances because of a leaky peer-review process publishing good and bad papers.
Therefore, I agree the critics should write a rebuttal paper.
Thanks for your clarification and courage.

Best regards
Günter Heß

EliRabett said...

The paper was not bad it was WRONG. It used a data set which was known to be badly contaminated by a warming effect in earlier years which was described in DETAIL in the literature by Reinhard Boehm from the HISTALP project, one that you collaborate with on elements of the same data set.

The corrected data set is easily available from the HISTALP project on line.

Both Georg and Eli pointed this out as well as showing how this was both well known and easy to trace.

Comparing the data which has been corrected for the effect of the thermometers not being sheltered until ~ 1860 completely falsifies Luedekke's fit. (see RR)

K.a.r.S.t.e.N said...

Interesting way to put it. Your argument goes like this: As soon as someone is engaging in a blog discussion, you can't consider his or her view or comment as independent any more: "I could not consider all these comments as independent reviews, since all of them stemmed from commentators of Georg Hoffmann's blog"
No matter whether there is scientific merit in a specific argument which has been put forth, you take it as unreliable by your own definition. That's an incredibly strange argument. As an expert in this field, shouldn't you be able to figure out whether there is merit in a reviewers argument? Shouldn't you assume that the arguments are made in good faith and from a critical neutral scientific point of view, regardless of what the reviewers personal opinion about the authors is? On top of that, shouldn't one be able to spot serious flaws in a paper itself in the first place?

While you effortlessly accept that the two respected expert reviewers "completed their reviews in good faith", you deny to grant the uninvited commentators the very same privilege (for no reason other than their personal opinion about the authors). In your response to Andreas you say: "This is a quite reve[a]ling comment from you. This means that you are already prejudging a manuscript by its authors, not by its content. This is precisely what never ever, under no circumstances, should be done."
This is exactly what you did with the commentators at Georg's Blog who subsequently criticized the paper merely on scientific grounds. You prejudged them on the basis of their expressed opinion. You clearly admitted that. I am terribly sorry, but you are contradicting yourself in a blatantly obvious way.

I just can't believe that someone who argues so strongly for a reasoned debate with dissenters, dismisses the argument of someone who he thinks has a biased view out of hand. In my point of view, this level of double standard beggars belief! Lamenting about how politicized climate science has become, but politicising on your part at the highest level is pure irony. Instead of discussing the flaws in the paper, you now seem to resort in accusation of misbehaviour on Manfreds and Georgs part. Seems a pretty bold approach to me. It's difficult to know why you thinks this is a good starting point for rational discourse amongst scientists.
Don't get me wrong. You have any right to accuse me and others of being biased (which you implicitly did!). But claiming to be a "honest broker" under such a premise is pure hypocrisy. It casts a damning light on someone's honest intention. Good luck in selling your ideology within the community and beyond.

I certainly respect your decision to accept Luedecke et al.'s paper. I don't mind having bad or even wrong papers published as this indeed happens all the time. But I do have a problem with how you are framing the accompanying blog discussion plus your unfounded allegation of bias (read: non-independence).


P.S.: I humbly apologize for the offensive tone. But as I am not particularly fond of a continued debate, I wanted to make my point as clear as I could.

K.a.r.S.t.e.N said...

Two more things I've noticed:
It is interesting to hear you say that "these reviewers are not less respected than Manfred Mudelsee, and definitively more respected in the area of time series analysis that all identified commentators". Some would say (not myself to be sure) that this is some sort of appeal to authorities, something which every resident blogger here at Klimazwiebel would fervently recommend not to do. I was asked not to rely on this argument twice if I remember correctly.

Who do you think would invest a second of his time to debunk this paper? It has already been shown to be wrong on several accounts. It boils down to waste of time. Time which would better spent in advancing our understanding. Something which people like Luedecke are certainly not interested in (note: personal opinion!). Unfortunately, the content of their paper just reinforces my opinion (feel free to call it confirmation bias). That you see the Marcott et al. paper at a similar level as Luedecke et al. tells much more about your view than any other comment could ever do (feel free to think the same about my comment).

Anonymous said...

Eduardo,

I think, you misunderstood me and my position maybe because of some prejudices?

First:
You wrote that a couple of editors were not keen on handling this paper, because it's "controversial".
I merely added that there are some warning experiences in the blogosphere, don't you remember the gossip about the resigned editor of remote sensing journal dealing with the Spencer paper? Or don't you remember the CERN chief Heuer, giving just a normal interview about the cloud experiment? So if I were an editor, I would see good chances of getting my name discussed in the blogosphere handling a skeptic's paper. And I admit: If I were editor, a skeptic would have better chances of getting a paper published, because I would like to demonstrate, that skeptic complaints about "gatekeeping" are bullshit. I understand that you can't tell about your thoughts, but you can't tell me having not thought about it.

I'm a little annoyed about your critics of the Primaklima discussion. I've linked to it in my last post, so all people interested can see that the discussion was based on scientific issues. I engaged in the discussion there (more asking than contributing, of course). What's wrong about discussing a paper in a blog? Is it a natural right only for McIntyres et al?

You might be surprised, but it was me, who wished the paper getting published after revisions. The others disagreed, but not, because Lüdecke is a skeptic, but because they saw too little scientific progress. Don't know, if they are right or not, maybe my position was influenced by being a layman like Lüdecke and I sympathize with the idea of layman contributing to science, which is hard to achieve. Maybe because I found some interesting issues in the methods used, which were new for me.

And please note, that the shitstorm regarding your person didn't originate at Primaklima, it started in the comments at Tamino's and Eli Rabett's blog. IMO it was embarrassing and annoying, this kind of ad hominem discussion I was used only at skeptic blogs before, and I defended you: http://scienceblogs.de/primaklima/2013/02/22/artikel-von-eike-pressesprecher-ludecke-et-al-veroffentlicht-in-climate-of-the-past/#comment-47346

Let me repeat, Eduardo:
I enjoyed following the peer review process and I appreciated your final response very much. But I got confused, when I saw the final paper with so little changes, ignoring most of YOUR own response. I (and some others at Primaklima) speculated, that your decision might being based on arguments beyond science, let's call it "diplomacy" or "politics", exactly these reasons I discussed before.

I've felt angry about some attacks on your person, but now I'm annoyed about your attacks on Georg, Manfred and the commenters (including me) at Primaklima.

Andreas

Anonymous said...

Eduardo

"'Not usual but rather "controversial" are the authors. Skeptics affiliated with the german blog EIKE. This implies some politic[iz]ation, if you want or not.'
This is a quite revelling comment from you. This means that you are already prejudging a manuscript by its authors, not by its content. This is precisely what never ever, under no circumstances, should be done. According to your view, all papers written by members of Greenpeace should be rejected right away ?"


Bullshit, nonsense, that's of course not my point of view. I was writing about editors handling skeptic papers. And BTW, what are you implying, Eduardo?

That you've never heard of Lüdecke and EIKE or that you are the only person in the world who is able to separate knowledge from judgement? Let me tell you, that's exactly is my daily job and I think, I'm doing quite fine.

PS:
Why do you have so many prejudices about my person?

Andreas

wflamme said...

Eduardo,

am I right to assume that reviewers (too) didn't obtain at least the data or even the code from the L. paper to help them form an opinion about the substance of the paper's claims?

eduardo said...

@ 8

'Independent comments means here 'prepared independently from other comments'. This is clear from the context of my blog, for instance, '.. or to consider them as stemming from a single commenter.'


I am not opposed that the blogospehere criticizes Lüdecke's paper, or any other paper for that matter, or my decisions as editor.
What I oppose is that a commenter, in this case Manfred, who rejected to be an official reviewer in the first place, tries to blackmail the journal because he feels that an editor did not take his opinion into account. If he felt uncomfortable with my decision he had these options:

-write to me directly, telling me who would not provide reviews of manuscripts handled by me.
-ignore all further call for reviews from me

what is he point of writing the journal, even more so with a public letter ? why make the journal or other editors responsible ? why not include in that public letter that he had been nominated from the very beginning but he had not accepted the nomination ?

The answer is quite clear for any one willing to spend a couple of seconds reading the open review phase of Ludecke et al.

eduardo said...

@ 10

Andreas,

you clearly suggested that I accepted Lüdecke et a. because I was afraid of the reactions of skeptics blogs.

'If you reject this paper, it's probable you get involved in a skeptic shitstorm about "excluding" skekptic papers.'

Georg Hoffmann said...

@Eduardo

I will comment to the points of a slightly more general interests on my blog. I just want to point out two/three minor points.

1) I am not a vain person, I am really not. You wrote my name here 7 times. Four times it was spelled correctly.

2) We had an email exchange between a couple of people which was not started by me. I was just on the mailing list. At some point in the discussion I asked you explicitly if I am allowed to cite from these exchanges and specifically your mail (see ps at the end). I didnt get any answer.

I considered this email exchange as private and confidential. It was an exchange mainly between you, Manfred and several CP editors. It is certainly not up to me to publish an information such as when, why and how it came that Manfred didnt end up as the official reviewer. So I didnt omit anything what I feel entitled to distribute.

In summary, I didnt mention your name in my actual posting (which was on some of the many errors in this Luedecke paper), I didnt openly criticize you but instead just wrote this (comment section of my posting on the Luedecke Article)

"Es gab einen mehr oder minder regen E-Mail Austausch und ich moechte diesen eher privaten, jedenfalls nicht wirklich oeffentlichen Austausch hier nicht hereinsetzen. Ich kann aber, glaube ich, soviel verraten, dass nach Eduardo A) die Reviewer Experten zur DFT Analyse und hervorragende Klimaforscher waren und dass B) Eduardo soweit zufrieden mit dem Reviewprozess und dem Resultat ist."

Would you agree that this is a fair presentation of your position?
For my feeling this is a maximum of what I would "publish" of this private email exchange.

This all said and as a quite regular reader of CP I think this Luedecke paper is the worst I have seen in this journal and should never have been published. I think this at the end of the day the thing that really matters in this history.

Georg

PS Here what I asked you at that time and you didnt answer:

Question from me to Eduardo

"PS Can I publish this part of your answer?

"There were 3 revisions of the manuscript. In CP only the first round of revisions is open-review, whereas all other revisions are not. If I am not mistaken, to send a revised version in the non-public phase of the review process to a guest commentator, without the knowledge of authors, would have been a breach. I could have invited Manfred to be a reviewer in the non-public phase of the review process, if he had not commented in the open phase. But in doing so, he was already 'contaminated' as reviewer.""

So in summary you are blaming me here now of omitting something but you didnt allow me to publish it. This is a bit strange for me.

Anonymous said...

@Georg Hoffman

"I am not a vain person, I am really not."

Do you really know who you're talking about?

"Some climate scientists we know sure are notoriously thin skinned," (Antony What's)


Yeph

Manfred Mudelsee said...

I am sorry that this incidence is still active in the blogosphere. For the sake of completeness, I put on Klimazwiebel and on Primaklima a commented email of mine. My other contributions are on CPD and on Primaklima. Interested readers should have enough material to understand my acting in this incidence. I would be pleased to learn how I could have improved my acting.

Manfred Mudelsee

=== Email from Manfred Mudelsee to one past CP editor (CC to CP editors and others) explaining withdrawal of offering services for CP(D), with comments by Mudelsee from 27/03/2013 ===

Dear xxx [M 27/03/2013, name deleted]

thanks for your interest.

Things were more complex in my case (Lüdecke et al.).

http://www.clim-past.net/9/447/2013/cp-9-447-2013.html

I rejected invitation to review owing to time shortage but then decided later (after reading the blogosphere) to join as guest commentator.
[M 27/03/2013 comment: the reason why or why not to comment or review should not be relevant; what should be relevant is the content of a comment or a review, and how it is taken into account]

To summarize the CP review process: I think I raised some serious technical flaws with the paper, Editor Zorita assessed them as insignificant (on basis of reviews of others) and let the paper through. It may be that he made errors, it may be that I made errors; I think Eduardo Zorita is a well-respected climate scientists with an excellent publication record and he is also a nice person. Notwithstanding this: In essence, I am declared as incompetent regarding climate time series analysis by CP. Therefore, I have no choice but to withdraw to offer any services to CP. My time is limited, and I devote it to journals that appreciate the work of reviewers. [M 27/03/2013 addition: and of commenters as well. And it would be good if a journal informs reviewers (and commenters as well) about the process a manuscript takes. If CP rules do not allow this: change them!]

Editors of journals (of CP and other) should be aware of attempts by so-called “climate skeptics” to enter the academic publication arena. Of course scientific scepticism is welcome, but those people have a political agenda. Political agendas damage science. My advice (to CP and in general) is to exercise utmost scientific scrutinity—to all submitted papers. It serves the profile of a journal better if, in case of doubt, the decision is “rejection”.

Best regards

Manfred

P.s. In my view, enough has been said about this instance, and I do not wish that this makes many more headlines in the blogosphere. Back to work!