Thursday, January 26, 2012

Models and reality


,In view of the interest on scepticism, I thought that the following text on models, written as a sort of personal introduction within the Post Normal Science workshop (more info here and here held in Hamburg last year, could be interesting for some readers of Klimazwiebel. It is a bit too long and there is no guarantee that at the end the reader will be satisfied. On similar tone, but much better written, is the article by Sibylle Anderl in FAZ.

 
My view of science, rightly or wrongly, is strongly influenced by my university background. Physics students are confronted from the very beginning with the concepts of theory and models. I would argue that these two concepts are not really separated and I would use both words as synonymous here - some models are theoretical or fundamental, like Newton's theory of gravitation, and other models are practical or numerical implementations that follow through the main ides expressed in theoretical models. A more important point is, however, that models and reality are deemed clearly separated, and actually physics makes use of quite different models that aim to describe different aspects of the same purported 'reality'. These a very common situation in quantum physics, in which subatomic particles- electrons, protons, etc., are handled as either particles or as waves, depending on the experiential situation. Many examples of this sort of dichotomy can be mentioned: the nucleus can be described as a drop of a 'nuclear liquid or as a set of neutrons and protons moving in a shell structure similar to electrons in a atom; phase transitions are brought about by the average influence of a whole solid body or by just the neighbouring atoms, etc. It is not unusual that in exams the student is asked to explain a phenomenon within a certain model. In the mind of a progressing physics student, the concept of reality loses value vary rapidly, and it is very seldom referred to, if at all. This Orwellian doublespeak does not seem to cause dramatic clashes, at least to most of us. A theory is just a tool to reduce the wealth of experimental observations to a common framework, or to make predictions about the outcome of as yet not available experimental results -arguably, both aspects, prediction and reductionism, being two sides of the same coin. A model is certainly not the 'reality', and even does not attempt to map reality one-to-one. The concept of existence (reality) is not central in physical models.
I think it is important to keep in mind the limitations of this 'irrational' concept of science or of scientific activity. Basically, the scientific activity consist of designing models (theories) that condense observations and test them against other observations. Predictions are not useful per se, but only as a tool to benchmark models. This utilitarian concept of models, i.e. quite detached of the concept of reality, is underlined by the fact that very often the building bricks used in those theories cannot be found in the real world. For example, Newton's model of gravitation was formulated by defining the functional form of the force between two point masses separated by a given distance. Obviously, nobody had seen at that time, or later, 'a point mass' . Models in Modern physics are much more alien to the daily experience. Also, climate models, and models of fluid motions in general, incorporate concepts that have only a limited range of validity, and thus they cannot be thought as 'reality'. One familiar concept is density. Density only contains a meaning at (our) macroscopic scales and increasingly loses its connection to (our) reality at atomic scales, where it would rather be equivalent to the rather loose concept of density of a forest. It seems therefore clear that models cannot attempt to map 'reality', in as much as 'reality' is not a well defined concept either.
I deem the sort of useful, down-to-Earth predictions, surely important and based on complex science, but a fundamentally different activity from that of model building and testing. Perhaps this is the reason of much of the controversy surrounding post-normal science. It could also be related to the eternal squabbles between the two dominant schools of statistical thought, frequentist and Bayesian. One of the most important aims of frequentist school is precisely hypothesis testing, which we could interpret here as model testing The frequentist try to estimate to what extent some preconceived hypothesis or models are compatible with observations and to what extent the agreement between models and observations could possibly be due to chance. Models and hypothesis are thus not proven by the statistical analysis, they are only disproved or deemed incompatible with experiments. This is exactly the viewpoint of classical science.
This lies at the centre of the attribution of anthropogenic climate change, since models that disregard the anthropogenic climate forcing are incompatible with observations, whereas models that do include those forcings are compatible with observations (actually less incompatible, as explained later). The concept of attribution is distinct from that of a useful or accurate prediction for the the future climate. This difference stems not only from the uncertainty in the possible future history of anthropogenic emissions, which is of course a crucial external condition for climate prediction but which does not form part of climate science. The difference in both sorts of activities is neatly illustrated by considering that the IPCC takes into account about 20 or so climate models, all of them claiming to describe the same 'aspects of an underlying reality' and each of them providing different predictions for the future climate. They are thus competing models. The classical scientific activity would be directed at separating the wheat from the chaff until hopefully one of these models remains. Even more strictly, classical scientific activity would be aimed at testing all climate models against present observations with the unhidden purpose of proving them wrong. This would not be quite difficult, because we already know that all climate models are wrong, in the sense that not any single one of them can reproduce all available observations, even taking into account the observation uncertainty.
However, climate prediction, and actually economic and many other sorts of predictions, have a different goal, namely to use as efficiently as possible all the available tools we have at hand (models, observations, experience, insights, etc) to deliver the most 'reasonable' future evolution of a given system. This is the world of Bayesian methods. Now all models are used, since all models are more or less equally wrong, and this is what we have anyway. All observations are used, since this is also the maximum amount of information and insight we may have about the system. Predictions are not good or bad per se, and they may even change and do change when new information (new data, new models) becomes available. this does not invalidate the methods used in former predictions, the predictions are just up-dated. Predictions are more or less efficient or more or less reasonable. This is a stark contrast to classical science, and much more similar to my understanding of what post-normal science is.
As an Orwellian doublespeaker by education, I do not feel especially uneasy. when confronted with this situation, as far as one knows on which court one is playing the game.

38 comments:

Georg Hoffmann said...

I give a triple A to this text. Cudos.

Hans von Storch said...

What, Georg, has Merton to do with this? CUDOS = Communalism, Universalism etc.? :-)

An issue, one should keep in mind is that the word "model" goes with rather different meanings in different quarters - we (Peter Müller and me) have worked that out in our 2004-book Computer Modelling in Atmospheric and Oceanic Sciences - Building Knowledge. Springer Verlag Berlin - Heidelberg - New York, 304pp, ISN 1437-028X. An important source for us was the book

Hesse, M.B., 1970: Models and analogies in science. University of Notre Dame Press, Notre Dame 184 pp.

which is addressing the complex specifically from the side of physics (which is not really too close to what we are used in climate science). A helpful element in her analysis is the usage of the concept of analogs, which can be positive, negative or neutral. The former two refer to validation, while the neutral ones allow the utility of a model.

eduardo said...

Danke, Georg. Der Text ist aber eher Standard and Poor

@ReinerGrundmann said...

Eduardo
very interesting indeed. I wonder how you would interpret the story about UK climate impacts which I posted yesterday. In the BBC online comment, David Shukman quotes a scientist who has been involved with the report. His comment on the role of models is: "They're the best we've got, they're all we've got."

Harry Dale Huffman said...

Ridiculous. This view, of "models over reality", is no longer physics at all, it is literally the writing of fiction, that heeds (and remolds) only the evidence that can be made to support and advance the writer's story. Zeno reborn (remember him, who "proved" that there is no motion?), and still standing in the way of true insight.

eduardo said...

Werner,

I do not think it is an unusual situation in science. Models/theories are always imperfect. Consider a seemingly well established theory, Newton's theory of gravitation. It is not correct, or rather not completely correct, but it is good enough for certain applications. The same can be said of a more advanced theory of gravitation, Einstein’s. It is not complete correct either, but also good enough for some applications, etc, etc. The question with climate models is not whether they are correct or incorrect. They are for sure incorrect, but maybe good enough for some applications, and not correct enough for others.

All calculations based on physics, without very few exceptions that are learned by physics students quite early, are approximate. Their validity can however be tested in experiments, something that it is not easily feasible with climate models

eduardo said...

Harry,

I am quite surprised by your comment. I do not see where the text supports the view of 'models over reality'. It rather express the idea that the connection between both is weaker that it may seem at first sight

@ReinerGrundmann said...

Eduardo
I think you intended to reply to myself, not Werner ;-)

I thought of the statement of the scientist quoted by Shukman as being in line with your Bayesian interpretation of the role of models.

But I wonder if the frequentist approach is favoured by skeptics who insist on testing "to what extent some preconceived hypothesis or models are compatible with observations and to what extent the agreement between models and observations could possibly be due to chance. Models and hypothesis are thus not proven by the statistical analysis, they are only disproved or deemed incompatible with experiments. This is exactly the viewpoint of classical science."

This is what I though you were arguing, making a link of frequentist methodology to those denying AGW. But you seem to come to the opposite conclusion.

eduardo said...

#8 Sorry, Reiner, yes, I did want to reply to you.

I agree with you. I possibly did not explain myself clearly enough. The tension between frequentist and Bayesian schools is also a reflection of the tension between classical and post-normal science - I interpret post-normal sciences, perhaps wrongly, as those in which experiments are impossible and difficult to conduct, and yet strong expectations are placed on those sciences. Climate science is a clear example.
In the play field of classical sciences, there is no room for climate-style uncertainty. If something is uncertain, a well design experiment will set any discussion straight. In sciences where this is not possible, like climate, economics, or cosmology, uncertainty abounds leading to a multiplicity of models and predictions that cannot be easily resolved. In those situations, where nonetheless predictions are required, Bayesian methods come quite naturally into play. We can see today the predictions of economic growth for 2012 issued by a dozen on models worldwide, all different. The extremes of these predictions can deviate quite strongly. We also know that none of this models predicted the difficult times we are living in now. Yet very few voices claim the economics is a rogue science. A true skeptic would require that all economics go back to their universities until they come up with one, and only one, predictive and successful model, and leave us alone in the mean time. The policy makers will, of course, bring them to heel and require their advice even though the predictions are known to be uncertain.

Anonymous said...

As an undergraduate, my geophysics professor (a very distinguished scientist) forbade the use of the word "model". The allowed term was "numerical experiment". If you look carefully, some of my peers (now working at the UK Met Office) maintain this distinction.

wflamme said...

Eduardo,

two questions:

1) Why doesn't your wrongness-forgiving Bayes' view include models that dismiss or trivialize anthropogenic CO2?

2) By what standards have you classified "all (wf: IPCC) models are more or less equally wrong"?

EliRabett said...

Models used in teaching are not so much wrong as they are incomplete because they are designed to illuminate issues or ideas. Experiments in lab courses are simplified so that those ideas can be actualized. The wave/particle duality issue is a excellent example of this, we can model/observe behavior that illustrates either, but the reality, which is what the maturing student begins to appreciate, is that the issue is not wave OR particle but wave AND particle and that he or she needs to understand when one type of behavior dominates or when both manifest simultaneously. Calling this Orwellian doublespeak is evidence of scientific immaturity.

The price of adding complexity to models of actual situations is a loss of casual understanding at the price of increasing accuracy as one attempts to trace the influence of various factors. Climate models are good examples of this. The broad outlines of the situation can be had by fairly simple models. Adding additional complexity (better models of oceans and clouds for example) makes things, well, more complex, but not necessarily more accurate and certainly more difficult to understand.

Testing very complex models requires pushing the inputs to extremes to discover where the model fails. Fortunately for the modelers, many people appear to favor doing so.

@ReinerGrundmann said...

Eduardo
"The tension between frequentist and Bayesian schools is also a reflection of the tension between classical and post-normal science - I interpret post-normal sciences, perhaps wrongly, as those in which experiments are impossible and difficult to conduct, and yet strong expectations are placed on those sciences. Climate science is a clear example."

Lets assume (not sure if I totally agree) this is the case. This would mean that all (or most) social sciences are by definition post-normal and that the climate sciences could learn from them -- after all, social sciences have been around for much longer and always been involved in policy making.

You give the example of economics with its notorious wrong predictions. Why do policy makers still listen to them?
Is there something in our political institutions that makes economics (despite all its shortcomings) an accepted tool for decision making? If so, aren't the social mechanisms (rhetorical strategies, boundary work, keeping high level of abstraction) more important than the epistemic core (i.e. usefulness and truthfulness of models)?

Another example is ozone science It did not suffer a loss of credibility after the discovery of the ozone hole which NONE of the models had predicted (and they could not model it for several years afterwards).

eduardo said...

#11

In a model ensemble that tries to describe the climate variations and would include 'all plausible' models, according to more or less established knowledge: the greenhouse effect of CO2 is a established knowledge on the same theoretical footing as the effect of solar radiation. Note the problem is to quantify those effects when other processes, say feedbacks, are taken into account. So a model that would dismiss the greenhouse effect of CO2 molecules would be clearly wrong from the outset. An analogy with economic models would be that creating money stokes inflation, the uncertainty lies on how much

I have qualified them as wrong because they all cannot be right at the same time, since they provide different answers to the very same question. They are quite few comparative studies so far to be able to tell that model A or model B is superior to all others.

wflamme said...

Eduardo (#14),

as to Q1:
I consider your response as being inconsistent with the Bayes' view you've laid down:

First you use hard, classical science to generally dismiss some models and keep the rest which are based on classical physics. But then you choose a much more tolerant bayesian view to explain why the ones you've kept nevertheless fail the criteria for which you've kept them in the first place.

Well, I'm a layman, just someone with a statistical background based upon interest and self-education. IMO the Bayes view is a consistent framework about how to optimally refine one's given prejudice facing new (or not yet considered) information. However it doesn't regard leaving and entering this framework at one's will as being optimal.

Re Q2:
Eduardo, according to your text you didn't qualify them as (simply) 'wrong' but as 'more or less equally wrong'.

Does that make a difference? Well yes ... because if it didn't you should dismiss no 'wrong' model at all.
If it doesn't however then there must be new/unconsidered evidence which according to you is quite missing (or inconclusive).

So I must conclude that up until now this is not logic, not classical science/physics, not Bayes but rather 'writing prose anything goes'.
Don't consider it an insult - after all, we're all humans.

eduardo said...

#15

Wflame,

I can barely understand what you have written, especially concerning the second question.

Please, keep in mind that as a climate scientist, I have a very low IQ.

eduardo said...

#13

I cannot know why or to what extent policy makers trust economic models. It would be interesting as a social study. From my perspective, it is clear that the do not heed climate models. For instance, the projections for Spain are indeed catastrophic, if they would come to pass they would make Spain inhabitable. However, I do not feel that policy makers there are particularly worried, not now and not 5 years ago. My conclusion is that they do not believe that those projections may be right.

Form other comments, it seems that the post is not very clear. Perhaps I will try again. All models and all theories are wrong, in the sense that they cannot describe nature perfectly, even we we restrict the concept of nature to a particular phenomenon. Again, a clear example is Newton's theory of gravitation. We know it is wrong, and not only because some free parameters in the theory are uncertain. It is structurally wrong. It yields, however, a very good approximation in most of the situations on Earth and the solar system.

Now imagine we have two theories of gravitation A and B, which when applied to calculate the orbit of an artificial satellite, yield differing answers. How do we proceed if we cannot design experiments to rule out one of both theories?
Actually, this situation occurs in tragical moments when in an aircraft two or more computers give diverging information to the pilots. I do not know the answer, but I can see the conflict between a pure scientific approach to solve this problem and the needs of the pilots.

wflamme said...

"I can barely understand what you have written, especially concerning the second question.
Please, keep in mind that as a climate scientist, I have a very low IQ."


Eduardo, you must be joking indeed since you bemoaned there "are quite few comparative studies so far to be able to tell that model A or model B is superior to all others".

IMO the only thing one can conclude from a lack of comparative studies is a lack of comparative studies - but not that the IPCC models are 'more or less equally wrong'. Would you consent to that?

Hans von Storch said...

wflamme, sure, the assertion "more or less equally wrong" is valid. All models are "wrong" in a trivial sense, so that the usage of the term "wrong" is also wrong. What needs to be determined, if a model is suitable for a purpose (analysis, forecast, sensitivity study, siulation tool of XXXXX).

Anonymous said...

eduardo

Thanks for bringing up that important subject.

Given the great many number of smart people involved in model building, I wonder since quite some time whether the thinking about what exactly they are doing is not quite underrepresented.

Maybe more than semantic nitpick:

You equate models to theories. This view is disputed by epistemologists such as Maragaret Morisson and Nancy Cartwright who frame models "as mediators" (between theories and empirical data) that construct an environment wherein theories and laws of nature can be developed and tested in the first place. These philosophers even claim that to be the case for hardcore physicist's models, but in the case of climate models it is even clear to me: The models draw from a large number of theories from different fields, from hypotheses, heuristics and questionable crutches because we have nothing better yet.

Looking at the actual implementation of the models it gets much worse: They are likely full of bugs. Also funny stuff like that happens: If you run the same code, the same binary, with the same input on the same hardware at two different times you frequently get different results. Numerical climate models are lightyears away from the status of a scientific theory such as gravitation.

Your focus on "validation" or "confirmation" seems to be restricted to comparison with observed data. I do not believe anything because some random code matches better some way to few and uncertain measurements than some other code. There are also general problems with that idea of validation alone (Konikow and Bredehoeft, Oreskes). My trust in these models derives to a large extent from what M. B. Beck calls "internal validity". The trust that they are put together in a correct and skillful way by people who know their stuff and take into account the state of the science. I even accept clearly non-physical features built in, for example artificial diffusion, because experts in numerics tell me that this is the OK. Maybe the trust in expert judgement here makes postmodernism creep in.

I think the current practice of running several different codes that were iteratively developed over decades by different groups, plotting the output in a composite graph and selling that as an assessment of model uncertainty is frivolous, to put it in a nice way. What would be needed now is a Grand Unified GCM: Build a model from scratch that contains a number of the best ones we have now as special cases; dynamical cores and all process descriptions as swappable modules. Validating the new code against the old ones you would catch a large number of the bugs and actually make a step towards code validation. Then you could do serious sensitivity analysis and assessment of model uncertainty with planned numerical experiments. Huge job requiring big resources? Sure! But absolutely straightforward with guaranteed success. Given what is at stake, its a poor excuse to say that this is mostly CS engineering and not so much science and therefore boring.

-hvw

wflamme said...

@HvS

Hans, please explain the difference between "'wrong'" and "wrong" in this context. Otherwise I'm afraid I will not get it and I wonder if anybody will.

Following eduardo's original explanation ...
"because we already know that all climate models are wrong, in the sense that not any single one of them can reproduce all available observations, even taking into account the observation uncertainty"
... the following definitions seems plausible:

a) wrong in a trivial sense: not able to reproduce all available observations

b) wrong: not able to reproduce all available observations taking into account the observation uncertainty (Eduardo)

c) equally wrong: (requires a key performance indicator to quantify the wrongness (type b) of a model)

Two models are equally wrong when these indicators match taking into account the observation uncertainty

d) more or less equally wrong: (probably requires a key performance indicator to quantify the wrongness (type b) of a model - my original question to Eduardo)

Definition still missing.

Anonymous said...

There's a new blog from a modeler, title:

ALL MODELS ARE WRONG ... but some are useful
http://allmodelsarewrong.com/

First topic: discussion of the title ;-)

Andreas

eduardo said...

@18

wflamme,

i was not joking. I jzst did not understand what you were trying to say. If you read your previous comment starting with ' Does that make a difference? Well yes ... because if it didn't you should dismiss no 'wrong' model at all......' maybe you would agree that it is not very clearly written. Perhaps I am wrong but my impression is that you are trying to set up some logical traps and in the end declare some type of victory, a game that I am not playing. I can try answer your questions and accept your criticisms if they are formulated clearly.

'IMO the only thing one can conclude from a lack of comparative studies is a lack of comparative studies - but not that the IPCC models are 'more or less equally wrong'. Would you consent to that?'

If you prefer to formulate it in that way, it is fine with me. I was arguing that due the lack of comparative studies it is difficult to single out models that are clearly superior to others. I still do not seethe subtle relevance of this question anyway.

It is very difficult to design a performance indicator. Some models may be better at simulating the observed mean temperature, others better at simulating the observed mean precipitation, ye t others may be better at simulating the observed variability, etc. There are many variables and many statistical measures one can think of.

eduardo said...

@20

I am not qualified to contradict philosophers (Werner would like this oine , I think), but nevertheless I can give a short account of my view of this. I had read a bit Cartwright when I was postdoc and I could but disagree with her. I though that obviously theories are something that exist, and can be true or false, independently of human minds. Surely Newton's law would be something that does not need a human mind to exist. Now, I am not so sure. Let us consider the example of a climate model a Newton's . There are clear differences: Newton's law is can be very compact and has a wide range of application. A climate model is a fortran code with half a million lines, applicable only to the Earth's climate. However, both are in essence two algorithms that use some input to produce some output. Can we be sure that the inhabitants of some planet in Andromeda would have formulated Newton's law as we have done, or even that they would be using the mathematical language as we use it ? Maybe not. Maybe Newton's law is one way, among many others, to condense some regular behaviour that we see in Nature in a very compact form. But we cannot know if we see all possible outcomes in Nature. Actually, we do see outcomes in Nature that cannot be condensed by Newton's law. Therefore, I would see theories and computer models on the same philosophical footing, if you like: They are human devices to predict yet unobserved phenomena, but in essence they are not different. The fact that we prefer Newton's law because it is very compact and a climate model is not, is just a matter of preference. Why should a natural law be compact ?

'Looking at the actual implementation of the models it gets much worse: They are likely full of bugs. Also funny stuff like that happens: If you run the same code, the same binary, with the same input on the same hardware at..'

This is correct, but considering a model as an algorithm, a model is 'the model plus its bugs'. The same model with different bugs is another model. Why are you sure that Newton's law has no bugs ? why do you consider the inverse squared law correct and not an exponent that deviates from 2 in the billionth decimal place ?

Sure, if you run a climate model code on different computers you may get differer results. but this is not a problem of the algorithm per se. If you implement Newton's law on a computer to calculate the orbit of a planet you also get different results on different computers.

'than some other code. There are also general problems with that idea of validation alone (Konikow and Bredehoeft, Oreskes). My trust in these models derives to a large extent from what M. B. Beck calls "internal validity". The trust that they are put together in a correct and skillful way by '

But then validation of a model or theory is just a matter of degree of believe. What is the difference between Newton's law and the account of the Bible ?

'I think the current practice of running several different codes that were iteratively developed over decades by different groups, plotting the output in a composite graph and selling that as an assessment of model uncertainty is frivolous, to put it in a nice way..'

and you are not alone in that opinion. The problem of how to combine different models that disagree - when they had to agree - , without knowing which one is better is quite acute. You may encounter different opinions on this. Some would say it is better to carry on with many different models and so we will have a measure about how wrong we may be. Others, like you, advocate the construction of a supermodel. I would be on your side if we had a possibility to test that supermodel, but that is not so easy. The problem with most models are not the bugs - by this I mean now coding errors contrary to the intention of the coder - but rather a lack of knowledge about the physical processes that are relevant at long time scales.

Mikel Mariñelarena said...

Hi Edurado,

In a comment on this blog just over 2 years ago you stated that 15 years of no observed warming would be "a sign of a serious problem with climate models": http://klimazwiebel.blogspot.com/2010/01/ten-years-of-solitude.html?showComment=1262942895200#c4477677993989895712

At the moment both RSS and HadCRUT show a trend of ~0C for the past 15 years (1997-2011). Do you still hold that view? What do you make of this long stasis (according to those 2 records) with regards to the IPCC climate models?

Thanks,

Mikel

wflamme said...

Eduardo,

it's so simple. In your original article you stated:

"This lies at the centre of the attribution of anthropogenic climate change, since models that disregard the anthropogenic climate forcing are incompatible with observations, whereas models that do include those forcings are compatible with observations (actually less incompatible, as explained later)."

Later you explained that the models that include anthropogenic forcings are "more or less equally wrong"

Let me paint a picture here: Let us assume that models reproducing observations within its uncertainty are graded 'A' then - since they are 'all more or less equally wrong' - they'd probably all rate B-C ... imperfect to some extent but not much different in their rating. Whereas models that dont't accout for these forcings simply grade F (if not G :-) ).

To obtain this picture evaluation standards to quantify 'the amount of wrongness' must be applied to each model. Only if such criteria exist it can be concluded that IPCC models are 'more or less equally wrong' and models that trivialize CO2 forcings are simply dead wrong.

(Btw: I don't mind if that result has been obtained by classical or bayesian means. I can live with both.)
But now I learn that these assessment frameworks are missing or controversial and comparative studies are too few ... so how was this rating result obtained then?

eduardo said...

@25

Mikel,

yes, I still have that *opinion* (emphasis added). A more serious analysis would involve to calculate 15 year trends in the temperature simulated by all IPCC models in the period, say 1950 and 2020, to estimate if there any many 15-year periods with near-zero trend (statistically not significant). I havent done that, but I think they will be difficult to find. If that is the case, my conclusion would be that either the climate sensitivity is smaller than in most models, or internal variability is larger and more persistent than in most models. Since there have been no strong volcanic eruptions in the last 15 years that could have had a cooling effect, it is an indication of clear model deficiencies.
I think we could also see this type of disagreement in the simulated sea-level trends

@ReinerGrundmann said...

Eduardo
I had a look at publication dates of papers in the ISI database which use the term Bayesian in the title and "climate change" and "model" in the text. There are about 120 papers published from 1992 until 2011. Half of them were published since 2009. I wonder what could explain this recent rise? Is it a realization that the "classical" approach does not yield the desired results?

But does the new approach not bring the charge of arbitrariness, i.e. the position to declare many observations as "consistent with" the models?

OBothe said...

I wonder what could explain this recent rise? Is it a realization that the "classical" approach does not yield the desired results?

If you search for Bayesian in any context it will show a similar increase. On the one hand, more and more people become aware of the Bayesian type of thinking (e.g. von Storch and Zwiers, 1999, as one standard of statistics in climate research concentrated on the frequentist point of view)
which, on the other hand, is probably to some extent due to the higher computational costs of Bayesian methods. That is, Bayesian thinking becomes feasible as computing becomes more efficient and faster.

@ReinerGrundmann said...

OBothe
Looking at Google trends this shows a clear decline since 2005. So something special is going on in climate science.
http://www.google.com/trends/?q=bayesian

OBothe said...

ok. i just followed your example and put "Bayesian" into the webofknowledge search. I got a nice increase of new publications per year starting in 1991 considering only every second year.

The interest in statistics seems to be declining as well http://www.google.com/trends/?q=statistics ;)

eduardo said...

@ 26

Wolfgang,

I suspect we have a problem of terminology. I have been trying to understand your way of looking at this. My impression is that we have a confusion between the term 'model' and the term 'forcing'. For instance, the same climate model can be used to simulate the climate of the 20th century with or without changes in CO2. It would be the same model , i.e. the same algorithm but with different input data. The forcing is something external to the model and is 'prescribed' , it is an input with which we feed the model (this is not completely correct, but for our purposes we can assume it is correct). When you write 'models that trivialize CO2' do you mean models for which the effect of CO2 is very small or simulations in which the CO2 concentrations have been kept constant ?

The second possibility is very easy to implement in practice : just conduct the simulation prescribing no change in CO2. The second possibility would be quite difficult, because the 'effect of CO2' is to a large extent due to the climate feedbacks, i.e. increase in water vapour, changes in cloud cover, etc' which are very similar to the feedbacks that are linked to other forcings, and cannot be controlled by the modeller in an easy way.

For instance, sometimes one reads in the blogosphere that models underestimate the effect of solar irradiance and overestimate the effect of CO2. This is in theory possible, but much more subtle than one may think. In general, if a climate model is very sensitive to changes in solar irradiance, it will also be very sensitive to changes in CO2. Therefore, it is difficult to explain the warming in the recent decades by attributing it to solar irradiance (even if it had increased , which is debated). A climate model would react even more strongly to the observed increases in CO2, because the radiative forcing of CO2 has been stronger , independently of the climate model you want to use.

eduardo said...

@ Reiner and Oliver,

Bayesian methods have indeed gained much interest recently. One reason is what Oliver noted: they require lots of computing resources . But I also think that the 'post-normal' situations, by which I mean situations of uncertainty that however require an answer, are the ideal playground for Bayesian methods. If climate was not that relevant for society, I guess the climate scientist would just happily conduct their observations until only one climate model is identified that yields a satisfactory agreement with observations and that is able to predict yet unobserved phenomena: the 'classical approach'. It is the pressure to come up with predictions with imperfect models that makes Bayesian methods more attractive.
Interestingly, very much respected statisticians consider Bayesian methods a contamination for science, and Bayesian statisticians do not have a better opinion of the 'classical group'

wflamme said...

@#32

Eduardo,

so you are not referring to other/alternative models that trivialize CO2 ... like the simple persistence model you and Hans(?) wrote a short paper about some time ago - temperature record series and things like that.
Instead you are talking about the same models?

Ok then, but I still don't understand how the overall rating emerged.

@ReinerGrundmann said...

The comparison between the general term "bayesian" and the co-occurence with "climate change" is as follows

http://tinypic.com/r/29zzpj/5

Before the 1990s, there was barely any interest in Bayesian stats, then there was a linear growth in general use until 2009, but an exponential growth in combination with climate change after 2007. If I am not mistaken this was the year when the Fourth Assessment Report was published.

eduardo said...

@34

Wolfgang,

no, I was referring to the full-fledged atmosphere-ocean models that contain a representation of the physical questions that govern the motion of fluids , condensation, heat exchange, etc.
The statistical model Hans, Thomas Stocker and myself used in the paper you are referring to was rather a possible statistical description of the global mean temperature in the absence of external forcings, i.e. it aimed at describing only the internal temperature variations. We actually had two of those simple 'models' and we varied the free parameters of those models over a wide range, because it is very difficult to pin down a value with such a coarse description of global temperature. The conclusion was indeed that internal variability - as described by these two models- alone cannot explain the recent clustering of high temperatures.

Here we see that the word 'model' is slippery. Statisticians also use the word 'statistical model' and I also use this term sometimes, but I would rather say that a statistical model is a just description of some properties of a data set, but they dont contain a 'theory' that could be applied to 'predict' something that has not happened in the past. A statistical description is usually not valid beyond a certain range of observed data.

The overall rating of climate models - now in the sense of full-fledged models - has not arisen yet. As I wrote some are better than other in simulating temperatures, and yet other are better at simulating precipitation etc. This is the reason to say that all of them are more or less equally wrong. Why is more important, to simulate well temperature or precipitation ? The answer is not clear. There are indeed some models that are bad at almost everything, and yet they are kept within the IPCC suite for reasons that are not totally scientific, but this another story.

Anonymous said...

eduardo, #24
Thanks for your extensive reply.

"However, both are in essence two algorithms that use some input to
produce some output."

OK, you operationalize the definition of theory. That is fine with me,
more understandable than what Cartwright has to say about it (not her
fault of course). Yet, I still think it is a good idea to regard
"scientific theory" and "GCM" as qualitatively different things. The
reason for being obnoxiously insistent here is the communication with
the world outside science. You have a really good idea about the
epistemological role of these models, which my dad, egged on by
"skeptics", has not. It makes a huge difference whether I have to
defend a "scientific theory" in the light of its obvious shortcomings
when compared with reality, or whether I only have to defend the
usefulness and correct construction of GCMs.

I don't find it simple to nail down that difference though. First lets
drop Newton's Law as poster child "scientific theory" for comparison,
it is just too far away from the GCMs, if only because of its
linearity. Doing this I also retract the arguments that were based on
differences in model implementation (between GCMs and Newton's Law),
point taken. Instead, I argue now to draw and explain the line between
numerical weather prediction and climate prediction. The models used
are very similar and even share code. The difference is in the timescale
of predictions and implicitly in the finer spatial scale that feasibly
can be modeled for a couple of days.

I can present weather prediction as "based on proven scientific
theory" because of certain characteristics not available in "climate
prediction": 1) Demonstrated useful level of accuracy, 2)
Reliable assessment of uncertainty, 3) Demonstrated progress
in performance
, 4) Regular demonstration of the correct
representation and relevance of the underlying processes: An
instance of prediction failure will prompt the meteorologists track
down the cause and usually they find it in the form of bad initial
conditions.

In the end it boils down to the degree of trust I have in a
prediction. That is subjective to a certain extent and fits nicely
with your connection to the Bayesian point of view. While we will
never dramatically improve the observation based validation of climate
models (at least in a time horizon that matters), we can and hopefully
will improve very much on the "internal validity" side of things. I
argue for a more systematic approach to increase this trust in GCMs
and to explicitly acknowledge the importance of "internal validity".

"But then validation of a model or theory is just a matter of degree
of believe."

Yes, I firmly believe that it is! One doesn't notice it that much
inside science, because we share to a large extent the belief about
which criteria matter and we have much of them quantified. When it
comes to choosing between two models, which perform similar in history
matching, for a prediction task, two scientist might come to differing
decisions, essentially based on something like "I believe this process
is more important than that one."

"What is the difference between Newton's law and the account of the
Bible ?"

The correctness of the latter is only believed by relatively few
people, compared to the believers in the former. But this is an
extreme example, as gravity is much harder to not believe in than for
example radiocarbon dating (creationists), dose-effect relationships
(people consulting homeopaths) or astronomy (people consulting
astrologists).

-hvw

jaket kulit said...

Thanks , I’ve recently been looking for information about this topic for a while and yours is the best I’ve came upon so far. However, what about the bottom line? Are you positive concerning the source