He specifically engages with Andrew Montford’s allegation (in his book The Hockey Stick Illusion) that ‘as many as four different journals may have had their normal procedures interfered with.’ Horton poses the rhetorical question ‘Despite peer review, are authors able to get away with dishonest or dubious research?’ to which he answers: ‘Yes, they are. Peer review does not replicate and so validate research. Peer review does not prove that a piece of research is true. The best it can do is say that, on the basis of a written account of what was done and some interrogation of the authors, the research seems on the face of it to be acceptable for publication. This claim for peer review is much softer than often portrayed to the general public. Experience shows, for example, that peer review is an extremely unreliable way to detect research misconduct. (Horton, in Muir Russell 2010: 129). But what about the allegation that outside interference took place in the review process of papers critical of AGW theories? It is worth to quote Horton at length:
‘If a research paper is especially controversial and word of it is circulating in a particular scientific community, third-party scientists or critics with an interest in the work may get to hear of it and decide to contact the journal. They might wish to warn or encourage editors. This kind of intervention is entirely normal. It is the task of editors to weigh up the passionate opinions of authors and reviewers, and to reflect on the comments (and motivations) of third parties. To an onlooker, these debates may appear as if improper pressure is being exerted on an editor. In fact, this is the ordinary to and fro of scientific debate going on behind the public screen of science. Occasionally, a line might be crossed. We experienced such a border crossing recently, where several reviewers and third parties encouraged us to delay publication of a paper for non-scientific reasons […]. Defining that line is the crucial task when judging the role of CRU scientists.’ (Horton in Muir-Russell 2010: 133).Neither Horton nor Muir-Russell provide us with a definition of such a line. It is noteworthy that Horton in the above passage says that external pressure is brought to bear on journal editors and that this is entirely normal. On the other hand he speaks of a line being crossed occasionally without making clear what this might be. He refers to a recent case at the Lancet but does not develop criteria for distinguishing ‘normal’ from ‘improper’ interference (when 'a line is crossed'). We can only guess that it might be when ‘non-scientific reasons’ are at play. This begs the question of how to deal with situations where scientists try to block papers from colleagues for reasons of politics, career advancement, or prestige--and how the controversial behaviour around CRU relates to it.
Posts
6 comments:
Just like in the previous post - drawing lines, illegal crossing, definitions of what is allowed....
The problem of peer review as presented here immediately reminds me of the process of production of knowledge at university in general. Professors are easily tempted to influence or manipulate the content of dissertations; undergraduates have no other choice than to accept, for career reasons (maybe you think twice about coming up with a skeptical dissertation when your prof is an alarmist - just as an example); presenting a panel on an important conference sometimes needs help and intervention from powerful lobbyists; on tenure track, assistant professors are forced to publish in peer reviewed journals and are happy to have good connections to editors or friends on the editorial board...There are rules, but universities are full of informal rules beyond public control. This is in stark contrast to the appearance of science in public as neutral and objective. This 'cultural' element is inherent in peer review - it is only possible to minimize it, but not to eliminate it. Thus, transparency is important, which is different from completely 'aseptic'.
One dimension of the practice of peer review has not been named by the author. In anticipation of the critical examination by colleagues researchers might refrain from applying non-conventional approaches to their research. Thus a tool aiming at enhancing the quality of science can suppress further development of the respective discipline.
In the field of sustainability science the question must be raised if a review by the “peer” (aka fellow researcher from the field) is really contributing to the aims of research. Why not introduce an additional “practitioner review” that will check articles for their applicability in “real life” and their usefulness in contributing to the good of humankind?
'third-party scientists or critics with an interest in the work may get to hear of it and decide to contact the journal'
I have to emphatically disagree with Horton on this point. That third party scientist contact the editor on their own initiative is not appropriate. I see the figure of an editor similar to that of a judge: the editor may seek evidence from technical experts if needed - (the reviewers )- but by no means should a third party try to influence a judge on a case in which this third party has some interest. I think an editor should always block any approaching attempts by other scientist which have not been contacted by the editor on his initiative.
In new journals with open review process, third party scientist can submit public comments, but this is very different from contacting the editor in private.
Henner, first of all, using "non-conventional" methods is actually quite common. It is the use of novel methodology that grants you 'fame' (assuming it works). Doing the same as others is known as the "me too" approach, and in my field means the importance of the paper is downgraded. Which makes rejection a lot more likely...
Second, if we had "practitional reviewers" on papers, many breakthroughs would probably never have happened. In many cases we cannot predict how a finding can be used, or abused!, in the future. Einstein once lamented his own theory of relativity, as it significantly contributed to the development of the atom bomb. But we also have used it for the "good" of humankind (e.g. nuclear power).
Hej Marco,
about No. 1) Point taken! I have to admit that I merely speculated on the basis of a rather not so amusing thesis defense two months ago.
About No.2) I did not suggest to introduce a “practitioner review” for fundamental research such as quantum physics or the like. I see that in these cases such a review would be pointless. I however think that a review by practitioners is helpful in the field of sustainability science. Here research is supposed to produce findings that are applicable and helpful in addressing the challenges of our time (especially climate change). This is even more valid, if we take into consideration that many research projects in this field are based on a research process that actively includes practitioners (Mode 2 knowledge production). The claim of “transdisciplinarity” is lame if only scientists are involved in the evaluation of findings.
Looking forward to read your reply!
henner:
Unfortunately, I once again have to disagree (at least in part). A clear danger of having practitioners is that they often focus on what *they* can do. Something novel, where they (know they) lack the abilities to use the new idea, may then be rejected. I've seen a few examples of that in my own field.
Note that some practitioners (in my field) have expressed similar sentiments about their fellow practitioners: their ideas were actively dismissed, because they were 'different'.
Funnily enough (or maybe not), this actually touches upon a major issue we have in stopping and adapting a variety of environmental problems: it often requires a change of behavior. Tell Americans they can save loads of money by drying their clothes on a line rather than the dryer, and the vast majority will still not take action. In Japan the government had to campaign to get Japanese businesses to increase the temperature of the airconditioning and allow Japanese businessmen to remove their tie (again, this would also save loads of money). There are hundreds such examples, some small, some large.
Post a Comment (pop-up window,non-moderated)