I think blogs can be used to gather information as well as disseminate it. As such I am asking for the help of readers of this posting. I was asked recently to comment on a proposal for a
sustainability tool-box for decision makers and a roadmap to creating a well
balance socio-eco system. On Google, a
search for “climate change roadmap” produces “about 5,110 results”; "decision
makers toolbox climate change" produces “about 962,000 results”.
People are now writing proposals claiming to
build new and improved roadmaps and toolboxes, new ways to build boats to
navigate the future. My question is,
could anyone tell me if any of the toolboxes full of existing tools or any of
the roadmaps have ever served any real utility?
By real utility, I mean have the tools ever built anything and have the
road maps ever led anywhere. I am not
being facetious; I would simply like to hear of accounts where such research
has been applied, or at least seriously considered. The road maps and toolboxes are, after all, for the benefit of decision makers. But have any ever been found to be useful? Obviously, their use would indicate their utility and perhaps it would be a worthy project to select those that have worked as representative of what a tool box or road map should look like. It might also be that none, or very few, have ever been applied in any sense, in which case why are research resources being used to continually build more of the same? Would this say something about the controlling powers of the purse strings of research?
Posts
25 comments:
I have some experience with tools (maybe not toolboxes), and most are disappointing. I could complain for pages over pages on that, but I think it is more useful to point out positive experiences.
1. The long-term success of tools is supported by a strong commitment of developers of the tool to maintain and run the 'thing' (usually a numerical computer model) for a couple decades, even when there are phases with lower or even no funding. It may sound funny, but the most successful tool I am familiar with is maintained and further developed by a very committed civil servant (Beamter) with a lifetime job. Such arrangements were still possible in the 1990s, but today no one is hired permanently by a University for such a job.
2. The tool was put on a broader base by establishing a Europe-wide network of co-developers. Some of these are employed in academia, others in European Joint Research Centres (JRCs).
3. It was never promised that decision makers themselves can and should use the tool (this is a pipedream in all cases), but always made clear that scientifically trained people are required to run policy simulations. In addition, regular training sessions for policy officers that work for decision makers are offered.
4. Network members working in JRCs serve as the science-policy interface by transmitting needs for policy advice to developers, and by communicating results to decision makers.
With this institutional design this tool has been used for quite some time already to inform policy decisions on agricultural and environmental policy issues in Europe.
Dennis,
maybe you should put it this way:
"People are now writing proposals claiming to build new and improved roadmaps and toolboxes, new ways to build boats to navigate l'ennui."
@Karl Kuhn
Many thanks for your input. However, in my brief look into decision makers tools, most are text based volumes with many schematics but nothing about the projects suggest there is anything that could be 'run' as in an operating model, just a presentation of new labels for old things and a rearrangement of the standard boxes, no strength of relationships among variables, simply an assumption that box a is connected to box b and box c is connected to box a. There is never any instructions as to how the tool might be used and what might be the outcome if it is used or could be used. It simply strikes me odd that such work seems so my in demand by those handing out research funds (look at FP7 for recent calls for such work)
Dennis
the terminology of 'tools' and 'tool boxes' often means 'heuristic', a way of thinking about things. I agree with you in that there seems to be an impression that new tools are needed without showing that existing tools don't work. BTW, funders and researchers seem to tacitly agree on this and this is no surprise as researchers who have obtained previous funding, are often invited to design new research programmes for funding agencies.
This may have led to a situation where we get more and more tools and tool boxes without evaluating their usefulness.
'Roadmaps' seem to be a synonym for strategic plan: where do you want to go and where will you be at which point in time?
It would be worth to do some research on this ;-)
For example, when did the terminology of tools, tool boxes and roadmaps emerge? Who started using it? Is it a EU phenomenon?
@Dennis Bray #3
Oh, just conceptual spaghetti graphs ... most are just terrible, but some are quite useful, provided that they reveal explicit and sometimes even quantitative functional relationsships between the variables involved in the process or model that is to be represented. For a good example check out consideo.de ... I can recommend them from own experience.
Nevertheless, there is definitely an unhealthy proliferation of all that stuff. IMHO this is driven by the desire of funding agencies to support policy-relevant research, combined with the urge to get rid of all that funds to build the Wissensgesellschaft. More input, more output, hahaha.
Dear Dennis,
the International Technology Roadmap of Semiconductors (ITRS) is a success story. The whole world, industry and academia is brought together to work on the technology needed for future generations of semiconductors for the computer industry. Without it, according to my opinion, we would not have succeeded to make the current generations of computer chips, neither CPUs nor DRAM.
It focuses the efforts of a huge variety of companies, small and large, and universities around the world in the pre-competitive phase of technology development to work jointly towards a common goal.
http://www.itrs.net
Best regards
Günter
Dennis,
Scientists rarely seem to claim to provide "roadmaps" for politicians; no wonder not much shows up on Google. "Toolbox" however appears to be a very popular proposal key-word. Not surprising that most outcomes are comparable to a claw-hammer with claws on both sides.
If you really want to find research that successfully contributed to global warming related decision making, why don't you look at performing projects and then find the underlying research? That is much easier.
Is it just me or are you social science people (also referring to Reiner Grundmann here) hang-up about lexicalicies? Why not ask "What are examples for successful applied research?" Whether or not that has the word "Toolbox" somewhere in the proposal seems random to me.
Two immediate thoughts: roadmap reminds me of "roadmap to peace" in the Middle East conflict; toolbox reminds me of Foucault, who suggested that his books are not for readers but toolboxes for users (decision-makers). It is not unusual in the humanities to talk of theory toolboxes, by the way (which might go back to the famous Foucault quote). I'll take up this perspectives from the humanities for a moment:
Your question is based on the assumption that tools create something useful (in measurable terms) and roadmaps lead to some specific place. This assumptions might be true when I want to repair something or to travel somewhere.
But in political reality, where decision makers want to change direction of society for example towards peace or here: sustainability, terms like "toolbox" or "roadmap" gain an additional momentum or significance. They become "performative" or part of "speech acts":
"Performative utterances (or performatives) are defined in the speech acts theory (part of the philosophy of language) as sentences which are not only passively describing a given reality, but they are changing the (social) reality they are describing."
Here two random examples from my field: UNESCO works on a "roadmap"" towards the preservation of digital heritage; in doing so, they link a new part of contemporary reality (internet) with the idea of heritage. It is something that is put into practice in working on and finally proclaiming a roadmap
The IUCN (intern. Union for the Conserv. of nature) created a toolbox for decision makers to improve the management of national parks, called IPAM (Integrated protected area management). Obviously, some people find it pretty useful, as you can (could) read here. I am sure it is useful - as long as you accept the general roadmap.
In order to assess roadmaps / toolboxes for sustainability, for example, it is hardly possible to measure success / usefulness or failure; I guess, what you need to develop is a toolbox of your own, an inventory of tools for critical assessment of discursive strategies. The question you pose (and the way you do it) is already half-way there: talking of "controlling powers of the purse strings of research", you are already in the middle of cultural critique (as opposed to simply measuring 'if it works or not').
By the way, I would recommend to any decision-maker to look for tools in Foucault's books; they would easily identify the use of terminologies like "roadmap" as discursive strategies and thus gain a wider perspective when putting them into practice.
Krauss:
"In order to assess roadmaps / toolboxes for sustainability, for example, it is hardly possible to measure success / usefulness or failure; I guess [...]"
I agree ... all you can do is follow up on whether the 'decisionmakers' (another such terrible technocratic word) use the roadmaps and tools (and find them useful, and for what reason) or not.
@ Reiner
‘The terminology of ‘tools’ and ‘toolboxes’ often means heuristic’, a way of thinking about things’. I think ‘rarely’ should replace ‘often’. They are only heuristic in the context of this discussion, for most people they are a hammer, a saw, etc. They are something that has some function, some utility. Maybe I should ask if anyone can enlighten me concerning the value of a heuristic tool and that those proposing to construct such a thing should be a little more explicit in just what they are proposing. With that in mind, could philosophy be construed as a tool, as it provides a way of thinking about things, could religious scripture? What I am familiar with seeing when I look at a heuristic toolbox is an expensive and convoluted way of presenting common sense and platitudes, and most versions are simply a rearrangement of a previous version. And I also wonder how many decision makers actually read the outcome of this ‘research’. There is much talk of how best to present such information; perhaps it is time to try a more entertaining Where’s Waldo approach. I agree fully, that it is time for some evaluative research on decision makers’ toolboxes and roadmaps. It might also be time for some evaluative research on the designing of new research programmes – perhaps suggesting the involvement of the broader science community; or better yet, just open calls for innovative research.
@Karl Kuhn
Thanks for the suggestion of consideo. I would agree fully that functional relationships in toolboxes and roadmaps are the things most missing. But how could you call it policy relevant research? That might be the intention, but is it really relevant?
@Guenter Hess
I am not at all familiar with the ITRS. Perhaps it could provide a model for such things environmental
@HVW
I think ‘roadmaps’ is a very fashionable term in the sustainability literature of late. Regardless, if we constantly produce claw hammers with claws on both sides, why hasn’t anyone noticed the futility in producing them? Why are more and more funds awarded to build the same?
Looking at performing projects would indeed be good, however, where do I find a performing project? How would one measure its application and success?
Not all social science people are lexicaladdicts. There is some hangover from the days when social scientists strived to produce terms and concepts that no one understood, and then spent an entire career trying to explain them – this was how success was measured. But I think we would have to be a bit more specific than asking for examples of successfully applied research. I think the problem lies in trying to claim pure academic research (good or not) as applied research for the sake of obtaining funding. Maybe, just maybe, the problem lies if the funding schemes, particularly where the success of a proposal is driven as much by having the right key words as having a good idea.
@ Bray:
"I think the problem lies in trying to claim pure academic research (good or not) as applied research for the sake of obtaining funding."
Depends on the donor - the DFG does not require research to 'applied', 'relevant' or 'useful', quite on the contrary, they stress that their purpose is to support fundamental research (Grundlagenforschung). Thus, working with the DFG is quite 'reasonable' given the limited societal impact most academic research can possibly have.
With other donors things look differently. BMBF or the EU pride themselves with their (imagined) ability to nudge researchers towards more societal relevance.
For instance, it was expected that the sophisticated numerical models I used for my economic analyses of African agricultural markets should be used as 'decision support systems' by local politicians and bureaucrats after the end of the project. In contrast to DFG, these 'political' donors virtually can't get enough of 'tools', 'decision support systems', 'modelling with stakeholders', 'roadmaps' and similar glitzy, illusionary drivel, from the laptop right to the smallholder, thereby totally forgetting the long ways that research sometimes has to take in our countries until it becomes policy-relevant information. And yes, they fund research of which they hope to get politically useful results ... and not research that is critical of current or envisioned policies. You will never find anything of the latter in BMBF or EU calls. And DFG will usually reject policy analyses as they are not 'fundamental research'.
@ Werner
First, why does it have to be roadmap, why not peacemap or simply map. Roadmaps to me remind me of something I had to use before I had a navigation system. Toolbox reminds you of Foucault, as in you can Foucault some of the people some of the time but you can’t Foucault all of the people all of the time? He may have suggested that his books are toolboxes for users but is there any evidence that they were ever used? Why has toolboxes been adopted as the terminology? Theory tool boxes – why not books? Most disciplines have theories. Most disciplines do not keep them in toolboxes.
So lets take a step back. Theory is defined as ‘A supposition or a system of ideas intended to explain something, especially one based on general principles independent of the thing to be explained’ (Oxford dictionary). This is precisely what is lacking in the decision makers’ toolboxes. Typically we are presented with a schematic that shows the hip bone is connected to the thigh bone with little or no explanation of why or how, in short, nothing is explained, simply, possible connection are described.
Re my assumption … tools … roadmaps when I want to repair something or lead somewhere. Isn’t this what the ‘roadmap to peace’ that you mention, was intended to do? Lead to peace? What a strange concept that a map is a model of where you intend to go.
The force of performative utterances – here I would like to borrow a quote from Richard Tol “WTF”. Regardless of what you want to call them – re-label them – the crux of the posting here is do, or have, they served any utility? Have the contents changed the social or physical reality? Until you can answer me this, how can I conclude ‘they are changing the social reality they are describing’? And on that note can I assume they are changing the water level that is rising – is this what the toolbox is to aid against?
UNESCO – whatever they want to call it – worked on a means to improve the management of national parks – call it knitting basket if you want. Ok, for the sake of clarity, where did the road map journey take them. We know the start point and the end point, but the journey?
The IPAM seems to provide a catalogue of definitions and categories, but I found no explanation of how this helps make decisions. The roadmap (?) seems to say define and protect. It is a manifesto, not help for a decision maker. It would be akin top defining a region’s vulnerability to environmental impacts at stopping there. It also seems somewhat of a fill in the blanks continuation of the National Parks Movement from the 1800s.
Why is it ‘hardly possible to measure success/ usefulness or failure … ? It is quite simply a matter of obtaining first an inventory of ‘toolbox’ projects and then asking decision makers first, if they are aware of them; second, if they used them; third, if they found them of any valid use. Not so difficult. Just about anything designed to have utility can be evaluated.
And please, I don’t want to be in the middle of cultural anything. I have a Sokal tee-shirt. ‘If it works or not’ is all I want to know.
I am sure all decision makers are all already well versed in Foucault. (They might have simply watched the video though.) You can sleep in peace tonight.
@Karl Kuhn re the DFG
Truly that is comforting to know. Knowledge for knowledge sake is becoming a rarity.
@Dennis,
I think the problem lies in trying to claim pure academic research (good or not) as applied research for the sake of obtaining funding. Maybe, just maybe, the problem lies if the funding schemes, particularly where the success of a proposal is driven as much by having the right key words as having a good idea.
I guess that is exactly the point. The two-clawed hammer need not be useless, but rather might be exactly what the researcher is actually interested in. Just that some disconnect with the keyword-requirements of the proposal had to be accepted ...
I guess Karl Kuhn above paints a realistic picture: A certain type of research funding stake-holders believes research can do what should be their (the decision-makers) work. And that they can get the researchers to deliver it by demanding "glitzy, illusionary drivel, from the laptop right to the smallholder" :). Karl Kuhn's example of software-tools might not completely capture what is meant with "tools" here, but it is exemplary: From my own experience in multiple fields, just to code a perfectly fine research model ('the paper is published') into a proper program that can be used for decision-making (just the basics: validation, user interface, documentation, maintainability, ...) often demands extraordinary commitment, funding trickstery and courage from a tenured professor who can afford some "inefficiency" with respect to her evaluation. Lots of powerful tools are lost for that reason.
My idea would be to promote an interface layer, somehow institutionalized but outside the usual academic structure with its own incentive structure, to sample the smorgasbord of parts and pieces and funny "tools" from the repository of traditional science, and actually build the tools that can be used in the real world. "Werkzeugmacher" is a highly qualified profession in Germany.
From another discussion I have learned recently that the counts given by Google as response to multi-word searches are virtually worthless. A search with the words "decision makers toolbox climate change" (without the quotation marks)resulted in my case in the reported count of 315000 hits, but going through the pages of results ended up with 376 page links.
Dennis,
sounds like a Pawlow reaction to "Foucault" and "performative". Obviously, your sociological education was in the eighties. I am fine serving as a dummy for your umpteenth coming-out as a tough-evidence-based-guy; but let me tell you: when push comes to shove, they will still smell your sociological background. But wtf, as your friend R.T. says. Anyway, I am better off before you have to suffer from another postmodern shock. Good luck on your search for the deeper meaning of roadmaps and tool-boxes!
Hey Werner
That is just the point - I was not searching for 'deeper' meaning. I was just searching for examples of success stories. A Foucaulian interpretation is not necessary.
And I don't know who they are that are doing the smelling but I have never tried to hide the fact that I am a sociologist, I have no problem with that. I don’t find it an embarrassment. But, you are correct, I was trained before the rise and fall of pomo. There are sociology departments that offer more than cultural studies. In fact I think in some cases they have become separate departments.
Pavlov - didn't he conduct well-structured experiments - evidence and data and observations and measurement and all that stuff? Ah, who knows, maybe he just liked dogs and made up stories; truth is, at a deeper level, he might well have been in the pockets of a dog food manufacturer.
'Tough-evidence-based-guy'? I guess I could replace evidence with assertions (see Pavlov above).
Should I guess or maybe collect a little bit of data? Hypothesis: People who work in cultural studies tend to feel insecure about their work when outside the circle of cultural studies? Is it perhaps data envy? Relax, and live a little.
D: "The force of performative utterances ... “WTF”. ... Have the contents changed the social or physical reality? Until you can answer me this, how can I conclude ‘they are changing the social reality they are describing’?
H: Watch out! Your back!
D: What?
H: Poststructuralists! Behind you!
D: So wha ..
Boom! THEY ARE NOT ONLY DESCRIBING BUT ALSO CONSTRUCTING THE SOCIAL REALITY! BlamBlamBlam!
D: Ouch, help, help!
H: Tautology! Use the Tautology Defense!
D: Got it, won't hold long though.
H: crrk, crrk, Almighty, almighty, this is PBR Sokal, we need a Habermas, NOW, crrk, crrk ....
@Bray:
By this last sentence ...
"Would this say something about the controlling powers of the purse strings of research?"
... you may have activated other Pawlovian reflexes ... :-)
Werner,
the toolbox quote from Foucault is intriguing. Here is the original quote:
"I would like my books to be a kind of tool‐box which others can rummage through to find a tool which they can use however they wish in their own area... I would like [my work] to be useful to an
educator, a warden, a magistrate, a conscientious objector. I donʹt write for an audience, I write for users, not readers."
What is the meaning of the term toolbox here? I think it is clear that he means less a specific tool for a well-defined task but rather a place from which to take things away, like from a stone quarry (Steinbruch) or a flee market. People ('users') can use little bits and pieces for their purposes, not reflecting the intended meaning by the author (Foucault in this case). A reader from an 'audience' would have to reflect the context and intention. It is as if he appeals to the lay people, not to the academics, to use his material as they see fit.
The spirit of his quote is quite contrary to much of the climate debate where everyone emphasises the importance of context, or the data which is dependent on theories and models. Cherry picking is a sin. Foucault, at least in this quote, encourages people to cherry pick from his work. This way people get cherries rather than claw hammers.
More on Tools and roadmaps:
Tools normally come with user manuals. They instruct the user how to make the tool work in the intended way. This does not seem to be the case with the tools developed by the kind of research mentioned by Dennis. Here, we have a close community between knowledge producers and consumers, and there is arguably a common tacit understanding about the 'correct' use of the 'tools'. There is a lot of tacit knowledge produced which cannot be made explicit. It probably could be made explicit if there was enough time (and interest!) but there is not, as the rush towards the next set of tools continues. After all, research has to come up with new things. I share the concern expressed by Dennis.
Roadmaps are symbolic representations of a geographical area which can be used to plan a journey from A to B. Nowadays there is a fashion of using it for all kinds of policy and strategy documents, but the roadmap (as we know it) is largely irrelevant. The term is used as synonym of 'plan', 'process model', or 'flow chart', often involving a time dimension (by what time to we get to B?). Yes, the metaphor was used in the Middle East to give the failed 'peace process' a new label, and to establish goals and a timetable. It did not work.
Thanks for the support Reiner
Post a Comment (pop-up window,non-moderated)