Frank Furedi as an interesting article in spiked on the peer review system. He finds:
The treatment of peer-reviewed science as an unquestionable form of authority is corrupting the peer-review system and damaging public debate.
Saturday, March 6, 2010
Subscribe to:
Post Comments (Atom)
Posts
25 comments:
Hans,
a very relevant post!
It bothers me for quite some time how the "final authority" of peer review is used to supress the
scientific debate. Unfortuneately this happens not only when argumenting with the general public but also
within the scientific community.
I feel completely different about this: The ultimate goal is to publish a paper in order to force the discussion about the hypothesis raised,
results presented and conclusion reached.
The publications should be debated in very many ways: Submitted and published comments are one way, but legitimate are also discussions among colleagues at scientific meetings,
in coffee breaks, in the blogsphere and.......
The dispute is part of the scientific culture, but it seems that many colleagues are offended by objections and/or are afraid that objecting and deviating from a consensus
will be misconceived and cause negative consequences.
How can the peer review system be improved? If a manuscript is distributed to 10 reviewers instead of just 2 or 3, this will delay publication a lot, and make the handling of manuscripts much more cumbersome.
An intermediate solution could be to ask for just a mark (from 1 to 10 for instance) from 10 reviewers and if the manuscripts passes a threshold it will be published. A formal review to improve clarity or details is then conducted by two or three among those 10 reviewers.
@eduardo #2
Broadening might help. But not just within the field.
When more or less sophisticated statistical methods are used, the "review board" need to include e.g. statisticans or mathematicans.
In many cases, knowing too much about the debate within the field is NOT helping a neutral assesment.
I think the article is rubbish. "the absence of evidence does not deter climate alarmists from practising their art", this is just more of the same hanging skeptic record. There are mountains of evidence for AGW. From the physical characteristics of CO2 (infrared light deflection) over the satellite measurements of the outgoing longwave radiation spectres proving the Greenhouse Effect, adding the c12/c13 ratios etc. And we have rising long-term temperature trends. There are melting glaciers, rising sea levels, etc. etc. etc.
And now "climate alarmists" are using "double standards" because of the use of grey literature in IPCC 4AR? So there is the one Himalaya mistake and the whole AR4 is painted as a mound of junk made by biased crusaders with an agenda carefully constructed out of a rigged peer review process and from grey literature? As opposed to "skeptics" telling straight out lies in the WSJ, on Fox News or the Times of London, lies that have been proven wrong a hundred thousand times?
This article is really quite upsetting.
This is an outstanding assessment of today's peer-review process. It has to be required reading for all undergraduate students who plan to go into research. It should also be read by other scientists too - as some seem to be in dire need of a refresher and ethical reorientation. How could anyone be against its message?
It should not come as surprise that the term "pal-reviewed" has been coined, which is synonymous to peer-review corruption.
"...the most disturbing, threat to the integrity of the peer-review system has been the growing influence of advocacy science. In numerous areas, most notably in climate science, research has become a cause and is increasingly both politicised and moralised."
That hits it squarely on the head.
Awhile back, some audacious scientists proclaimed themselves to be morally and intellectually superior to others, and they went on to organise themselves into some sort of Team to shut out, denounce and destroy contrarian thinkers. They appealed to authority, even calling others “not real” scientists. It was a science coup d’état. But such a system is simply not sustainable because it alienates so many other highly esteemed scientists. It creates many bitter enemies. Worse, much of it was based on fiction, and very little fact. It was doomed to crash, and oh how it has!
Yet, you'd think some lessons would have been learned by now. Hardly.
In the recent days we have yet again seen calls for and evidence of another massive campaign to "gut the sceptics" once and for all. This is like science by Machtpolitik. It is so repulsive that more rational scientists, like Judith Curry, can hardly keep their stomachs from turning.
I'm going to send this report to one famous climate-impact research institute located not far from Berlin, and also to the Honorouble Madam Chancellor herself. It is a very serious issue.
Flin
Come on. You serious?
Melting glaciers?
etc.
etc.
etc.
We have all found out just how much evidence is really real, and how much is based on unavailble data, and anecdotes taken from environmental activist groups.
When are you going to realise that climate has always changed and always will? If one looks at the climate over the last 10,000 or even millions of years, you do not see any discernable human impact today. None. It's a hypothesis. Show us the data.
Your belief is religious, and not based on data. Wake up!
@ all, specially the sceptics
is peer-review (or beer-review) a necessary albeit imperfect process?
how would you organize it?
Peer review would need to be open and transparent. For example, once a paper goes out to review, one could put the reviewed paper plus reviewers' annotations and comments online, followed by a period of comment from readers. This could be done anonymously, semi-anon. or with real identities revealed. These are secondary matters compared to the open process principle.
What happens to papers that are rejected by the editor and not sent for review? I don't have an answer, but perhaps the manuscripts and decision letters should be kept so that they could be audited at some point.
Above all, journals, especially top journals would need institutional checks to prevent nepotism and corruption. They play such an important role for the visibiity (and credibility) of results and have an immense influence on careers. To leave their operation to the personal whims and virtues of individuals is highly problematic.
Climate science over the last 20 years has provided us with an abundance of examples on how science ought NOT to proceed.
Of course checking one's work and subjecting it to scrutiny is a necessary process. It's the only way.
In summary, the requirements:
1. Subject the work to sceptics.
2. Make all raw data and methodology available TO ALL.
No crying and moaning and paranoia that others are "out to destroy me!".
Work that does not fulfil the 2 above conditions certainly cannot be regarded as accepted science, let alone "settled science". Period.
@P Gosselin: I know that climate has always changed and always will. Your religious theme is rather insulting. I will not answer any of your comments anymore. You are the religious zealot that doesn't respond to reasoning. Not me.
Good point Reiner,
Actually I feel the problem is not with the work that doesn't get published, rather it's with the crappy work that does get published.
For works that unfairly don't get published, then other outlets have to be found.
For example, the Old Mainstream Media attempted to keep certain views out, but today the internet is now a major news provider. So I guess we have to smash the science information monopolies, if they exist. It doesn't hurt for Nature and Science to have some competition. I'm sure it's coming.
That's one of the silver linings that has resulted from the bad climate science.
some random thoughts:
1) there is no perfect system of peer review, because it is done by people.
2) Peer review is not only a problem of science, but also of the public. We need educated readers; the public has to know that even peer reviewed scientific work is relative. You still have to think, to compare, to make up your mind as a reader.
3) There were (and I am sure there are) many cases when papers from skeptics passed peer review. I even remember Hans von Storch, stepping back as editor of Climate Research because a 'skeptical' paper (Soon et ?) based on poor data had passed peer review, and he was not allow to comment upon this as a new editor. Thus, it is polemical to reduce peer-review manipulation as a problem only of 'alarmist' science.
4) Of course, peer review is a currency in the science business. In order to make a career, you have to publish in peer-reviewed journals. And look at the number of peer -reviewed publications of natural scientists! I bet that even the authors themselves sometimes cannot remember having written this or that article (which indeed was mostly written by one of their PhDs or post-docs). And imagine the workload for peers to review all these incoming papers, besides the everyday business.
5) Peer review as an argument in the hockey stick debate: As far as I remember, Hans von Storch did not mention McIntyre in his critique of the hockey stick methodology, because McIntyre had no relevant peer-reviewed articles concerning this topic.
6) A quote from the Romm / Pielke debate posted here on klimazwiebel, made in defense of Pielke:
'(....) Pielke has one of the longest, if not the longest, list of peer-reviewed publications on the matter.'
Werner,
"And imagine the workload for peers to review all these incoming papers, besides the everyday business."
Yet, we're not really talking about butterfly population studies in East Bumsville, now are we?
Climate papers are being used on which to base new profound public policy that demand we make great lifestyle changes and that could have great economic consequences. These papers absolutely have to be looked at with super-scrutiny and extra rigor. At least the raw data and codes should be available. Or?
The argument that it is too much work to thoroughly peer-review all papers, and so let's skip rigorous review of climate papers, is a dubious one to me. I don't think you are going to fool very many with that one ;).
P. Gosselin - may I suggest that you argue a bit more nuanced, and consider a bit more details, please. Also, waiting for a couple of hours with responding is sometimes helpful. Maybe, the others use reason in their arguments, at least sometimes?
-- Hans
P, Gosselin - obviously, you need a break. Thus, no more posting today, please. Calm down, and try to bring forwards arguments - not mere repetitions of assertions.
I guess many Germans in this discussion do not understand the meaning of the word "peer". Peer review means review by colleagues (and by nobody else). It was originally thought to guarantee the autonomy of science. This autonomy entails the right of error (but not of unethical behavior).
Most suggestions above are about getting rid of peer review and make science a democratic endeavor that is decided by votes and political fashion. We can discuss that. But open peer review is an oxymoron.
The second option would be broadening the meaning of "peers" and include politicians and the public from both side of the aisle. However inclusion and exclusion would be always an issue.
And it will produce the same effect that is criticized now. Science would become just politics - maybe less authoritarian and more democratic. However it would not help to reestablish "objective science" (if such a thing ever existed).
And again (and in this sense the posted article is right), peer review is not about final truth. It is about checking on consistency and methodical standards. The probation of scientific findings develops over time. A published paper is never the last word (despite peer review). Most of them get falsified later, at least in some details.
I agree the role of peer review has been misrepresented by some climate researchers. Peer review does not mean final truth. But it should mean that it is the best knowledge available.
By the way, political motivated science is not as new and it actually produced big breakthroughs. One might think of the Manhattan Project or the roots of the Internet which lie also in military research. Cancer research is another example.
I do not defend climate research. I just think that if we stick to an unrealistic idealistic picture of science (and also climate researchers did), we will not solve the institutional problems that occur when thousands of people work together on an policy relevant issue.
PS: Of course there are more or less open forms of peer review around if it comes to policy of certain journals. I actually wrote about that. Some pioneering methods of self-recruiting reviewers can be found in the very field of atmospheric science (the IPCC process is actually an example).
Another interesting example http://www.atmos-chem-phys-discuss.net/papers_in_open_discussion.html. This journal has an open discussion of every contribution.
Yawn...peer review is not perfect, therefore lazy sloppy bloggers think they are doing better. It's like someone saying a knife is not as good as a gun, therefore use bare hands. And...yawn, yawn...we've seen this logic flaw for years now.
It seems to me that the main question raised by Furedi's remark is to what extent a scientist's work may be agenda-driven (rather than peer-review per se). It is right that political agenda should not be disguised as objctive knowledge. (In Roger Pielke Jr.'s words, we should not be stealth issue advocates.) On the other hand, some scientific subjects make sense only with some premise about policy choice. As a matter of science, we should be disintersted whether the premise be fulfilled or not. But as a matter of motive for a scientist to do the study, it is likely that the premise is something very desirable or very undesirable.
My opinion is that we should keep agenda in the periphery and prevent it to penetrate into the core of scientific documents.
I mentioned in my comments on the article "Over the top?" on 31 January that a scientific journal article can be separated to "hard" and "soft" parts. (And that even the "hard" part of a paper is reliable, the "soft" part may often not.) This time I rephrase them as "core" and "periphery" parts. The "core" consists of the description of data and methods, results, and conclusion. The "periphery" consists of the introduction, as well as discussion, remarks, implications etc. which are given after the conclusion.
I think that the "core" parts should be written in a disinterested manner to describe a view shared with people who may have different agenda. This should be an explicit point in the peer review process of journals.
On the other hand, I think it un-productive to try to eradicate agenda from "periphery". I do not think that crude advocacy may be included even in the peripheral parts. But, if the author finds relevance of the core results in a certain policy position, I think it is more informative to show it than hide it.
It is the readers, especially journalists, that should take care that the peripheral parts may contain something other than science in its narrow sense. Please give weight on the core part when you report about a scientific article.
p.s. In the context of IPCC, the insistence by the WG1-related scientists may have harmed reputation of WG2 and WG3 whose references are not always peer-reviewed journal articles. It is one of many elements of mutual misunderstanding we need to continue correcting.
@Werner Krauss: Concerning your random thoughts, point 2: "Peer review is not only a problem of science, but also of the public". I do not see peer review as a problem. Publishing a paper in a scientific, peer reviewed journal ensures a basic quality of the paper. Not more and not less. Published papers are not the final answer. They can be commented on, corrected, even withdrawn. This is the process how science is done, isn't it?
And this is where i totally disagree with Furedi. Peer review is not a holy grail. Publishing papers in a peer reviewed journal, however, shows that the participant in the scientific discussion has something significant to say and does follow scientific standards and methodology to a certain extent. So the publishing is, in my eyes, a kind of entry qualification that shows that someone at least tries to add to our understanding of the world.
And since, as you mention in your point 3, sceptics happen to publish papers as well, this entry qualification isn't as impossible to achieve as Furedi suggests in his article.
Peer review as in the good ol boys network or
the scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards.For example, if you’re doing an experiment, you should report everything that you think might make it invalid—not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked—to make sure the other fellow can tell they have been eliminated. In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.Surely you're joking Mr.Feynman...
Or,
do we have to re-invent peer review again, surely you're joking Mr.Jones...
comment/24 contains some interesting lines, but unfortunately in a style, which has been earlier explained as being not acceptable. It would have been easy to write this in a polite but nevertheless clear way ... but in this form, I had to delete it. Unfortunately. -- HvS
Post a Comment (pop-up window,non-moderated)