Model uncertainty

I am opening a new thread to discuss issues arising in the Grilling Jones post. This has to do with data sharing and the relation between datasets and theoretical models. So please stay on this topic when commenting. The reference is to a sociological study (thanks to Jin W for alerting me!). In case you find it difficult to access this article(free our data!), I reproduce the main findings from the conclusion below.

Young, Cristobal (2009) Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth, AMERICAN SOCIOLOGICAL REVIEW, 2009, VOL. 74 (June:380–397)

...In methodological terms, this article illustrates
how research findings can contain a great
deal of model uncertainty that is not revealed in
conventional significance tests. A point estimate
and its standard error is not a reliable
guide to what the next study is likely to find,
even if it uses the same data. This is true even
if, as in this case, the research is conducted by
a highly respected author and is published in a
top journal. Below, I outline a number of specific
steps that could help improve the transparency
and credibility of statistical research in
sociology.

1. Pay greater attention to model uncertainty.
The more that researchers (and editors and
reviewers) are attuned to the issue of model
uncertainty, it seems likely that more sensitivity
analyses will be reported. Researchers with
results they know are strong will look for ways
to signal that information (i.e., to report estimates
from a wider range of models). Results
that depend on an exact specification, and unravel
with sensible model changes, are not reliable
findings. When this is openly acknowledged, the
extensiveness of sensitivity analysis will, more
and more, augment significance tests as the
measure of a strong finding.

2. Make replication easier. Authors should
submit complete replication packages (dataset
and statistical code) to journals as a condition
of publication, so that skeptical readers can easily
interrogate the results themselves (Freese
2007). This is particularly important for methodologically
complex papers where it can be quite
difficult and time consuming to perform even
basic replications from scratch (Glaeser 2006).
Asking authors for replication materials often
seems confrontational, and authors often do not
respond well to their prospective replicators.

In psychology, an audit study found that only 27
percent of authors complied with data requests
for replication (Wicherts et al. 2006). Barro’s
openness in welcoming this replication—readily
providing the data and even offering encouragement—
seems to be a rare quality. Social
science should not have to rely on strong personal
integrity of this sort to facilitate replication.
The institutional structure that publishes
research should also ensure that any publication
can be subject to critical inspection.16

3. Establish routine random testing of published
results. Pushing the previous point a bit
further, Gerber and Malhotra (2006) suggest
establishing a formal venue within journals for
randomly selected replications of published articles.
The idea is to develop a semiregular section
titled “Replications,” with its own
designated editor, in which several of the statistical
papers each year are announced as randomly
selected for detailed scrutiny, with wide
distribution of the data and code, and the range
of findings reported in brief form (as in Table
1).
Indeed, this could provide ideal applied exercises
for graduate statistics seminars. Even if
only a dozen or so departments across the country
incorporate it into their classes, this alone
would provide a remarkably thorough robustness
check. The degree of model variance would
quickly become transparent. Moreover, the
prospect of such scrutiny would no doubt
encourage researchers to preemptively critique their own findings and report more rigorous
sensitivity analyses.

4. Encourage pre-specification of model
design. One of the problems in statistics today
is that authors have no way to credibly signal
when they have conducted a true (classical)
hypothesis test. Suppose a researcher diligently
plans out her model specifications before
she sees the data and then simply reports those
findings. This researcher would be strategically
better off to conduct specification searches
to improve her results because readers cannot
tell the difference between a true hypothesis
test and a data mining exercise.

The situation
would be greatly improved if there were some
infrastructure to facilitate credible signaling. A
research registry could be a partial solution. In
medical research, clinical trials must be reported
to a registry—giving a detailed account of
how they will conduct the study and analyze the
data—before beginning the trial.17

A social science
registry would similarly allow authors to
specify their models before the data become
available (Nuemark 2001). This is feasible for
established researchers using materials like time
series data or future waves of the major surveys
(e.g., NLSY, PSID, and GSS). This will, for the
subset of work that is registered, bring us back
to a time when model specification had to be
carefully planned out in advance. Authors could
then report the results of their pre-specified
designs (i.e., their true hypothesis tests), as well
as search for alternative, potentially better, specifications
that can be tested again when the next
round of data becomes available.

Because most
data already exist, and authors can only credibly
pre-specify for future data, this would be a
long-term strategy for raising the transparency
of statistical research and reducing the information
asymmetry between analyst and reader.
Thirty years ago, model uncertainty existed
but computational limitations created a “veil
of ignorance”—neither analyst nor reader knew
much about how model specification affected
the results. Today, authors know (or can learn)
much more about the reliability of their estimates—
how much results change from model
to model—than their readers.

As Hoeting and colleagues (1999:399) argue, it seems clear that
in the future, “accounting for model uncertainty
will become an integral part of statistical
modeling.” All of the steps outlined here would
go far, as Leamer (1983) humorously put it, to
“take the con out of econometrics.”
(pp 394-5)