A mistake with consequences ?

A new paper in press in Journal of Climate by Jason Smerdon from the Lamont Doherty Earth Observatory and collaborators documents surprising, and somewhat inexplicable, errors in some previous pseudo-proxy studies by Mann and collaborators.



Background: The reconstruction of past climates ís based on the analysis of biological or geochemical archives, like tree-rings, which may be sensitive to environmental conditions. These archives need to be somehow translated to variations in temperature or precipitation. This is mostly accomplished by statistical methods: in the simple case that the annual variations in tree-ring widths are correlated to variations of local summer temperatures, this correlation allows for a calibration of tree-ring widths in terms of mean growth-season temperature, which can be then extrapolated back into the past. Assuming it remains valid also for long-term climate variations, it can be used to reconstruct past temperatures. One of the question that always remains open is whether or not these empirical methods also provide a reasonable answer at longer time scales. This is not guaranteed. The archives may had reacted differently to environmental changes in the past. Also, although a statistical method may perform well for the rapid interannual variations scales, it may not be that good at longer timescales, such as decadal and centennial variability
In particular, the performance of statistical reconstruction methods is not easy to test at longer time scales, since the observational record is obviously short. This is the rationale of pseudo-proxy studies, which use artificial data generated in long climate simulations. In these simulations, the target variable to be reconstructed, is known. If artificial proxies could be also somehow generated within the climate simulation, the reconstruction method could be applied to these pseudo-proxies and the result could be compared with the known target temperature. A pseudo-proxy can be approximately  generated  by taking the temperature simulated at one of the model grid-cells and distorting these data with random numbers to emulate the imperfect relationship between real temperatures and real tree-ring widths.
There are several pseudo-proxy studies around that have tested in this way different climate reconstructions methods. The conclusions of these studies sometimes diverge. One of the current controversies involves the Regularized Expectation Maximization method. Some groups (Christiansen et al, Smerdon et al ) have found that this method leads to a similar underestimations of past variations as found in early reconstruction methods, whereas other group including Mann and collaborators, here and here, find that the RegEM method performs well. The new paper by Smerdon et al has identified basic and surprising errors in the testing of the RegEM method by the Mann group. The errors are not difficult to understand and do not involve the implementation of the RegEM method itself: in one case, when interpolating the climate model data onto a different grid, the data were rotated around the Earth 180 degrees, so that model data that should be located on the Greenwich Meridian were erroneously placed at 180 degrees longitude; in another case the data in the Western Hemisphere were spatially smoothed, while the data in the Eastern Hemisphere were not. These errors bear some consequences: the location of the pseudo-proxies did not match the location of real proxies any more; the spatial covariance of the temperature data was not correct; when the authors though they they were testing the skill of the method to reconstruct the temperature in the ENSO region by proxies located in North America, they were actually testing the reconstruction of temperature in other region by proxies located somewhere else.
Stuff happens and errors like these can creep in every study by any group. There is no reason whatsoever to think that malice played any role here. It is, however, surprising that these errors went undetected for several years, affecting two key manuscripts that used the same data sets. My conclusion is that those stress-tests were really too weak to detect any possible low performance of the method.
The most recent climate reconstructions by Mann et al published in Science in 2009 were conducted with the RegEM method, supported by the good skill displayed by the RegEM in those previous stress-tests. I am unsure as to what extent those reconstructions may be now compromised. Interesting food for thought for the authors of the next IPCC Report.



Update: Rutherford et al. have a response here, of which I ignore the status (in preparation, accepted, in press..)

Print this post