Poles Apart

Over the past few months, scientists studying global warming have been rocked by a series of awkward revelations. In November, someone made public more than 1000 emails, many quite damning, from climate researchers at the University of East Anglia (UEA), in what skeptics successfully branded as "Climategate." December and January saw the authoritative Intergovernmental Panel on Climate Change (IPCC) admitting that their published claim that Himalayan Glaciers would disappear by 2035 was improperly sourced and wrong, even as critics pounced on other instances in which the panel violated their own procedures for insuring that science was properly represented in their influential reports. (The IPCC had shared the 2007 Nobel Peace Prize with Al Gore.)

Not surprisingly, different people responded completely differently to these events. Critics see the revelations as confirming that global warming, and especially its human source, is just a hoax. Others counter that the revelations that scientists are fallible human beings do not in the least affect the overwhelming evidence for man-made climate change. Unsurprisingly, both extremes find confirmation for what they already believed.

They are also both wrong.

First let's examine the hoax theory. Although we don't know who released the emails, they look to have been culled from more than a decade of exchanges and chosen to best incriminate the mainstream climate researchers, both those at UEA and their correspondents. What is striking is that the emails do not
reveal any serious evidence that the researchers are fabricating the global temperature rise.

To be sure, the emails show serious misbehavior, including a request from UEA researcher Phil Jones that others delete emails to avoid a freedom-of-information inquiry. Other emails suggest that the researchers hoped to tweak the IPCC process to exclude legitimate, peer-reviewed papers that they didn't regard as credible.

But if the emails showed evidence of a true hoax, the critics poring over them haven't found it--and not for lack of trying. Instead, they have highlighted examples of ambiguous or unfortunate wording, which are conveniently picked up by the likes of Sarah Palin. But although Palin may not know any better, the critics understand that when Jones referred in 1999 to Penn State researcher Michael Mann's "trick" to "hide the decline," he was describing a technique to de-emphasize the inconvenient truth that tree-ring data don't match measured temperatures, which were clearly rising.

This is a serious matter: if the tree-ring data are not good proxies for temperature in cases where we know the temperature, how can we trust them in cases where we don't know the temperature? But as far as I know, the published papers acknowledge this manipulation, which is an acceptable way to deal with omissions of questionable data. Nothing is being hidden. In fact, this whole issue was addressed by the U.S. National Academy of Sciences in 1996, who confirmed the unprecedented nature of the current warming trend.

The critics know this. Their use of this and similar examples shows that they are less interested in the overall truth than in scoring points and discrediting mainstream climate science.

But the disingenuous actions of the critics do not excuse the behavior of the mainstream scientists.

It is tricky to interpret what must have been seen as private correspondence between colleagues. Nonetheless, the emails seem to show that the East Anglia scientists did not really trust the processes of science, or at least the political decisions based on the science. The researchers in the emails are acutely aware of the political context of their results, and present the data to make their case. For example, UEA tree-ring expert Keith Briffa says "I know there is pressure to present a nice tidy story as regards 'apparent unprecedented warming in a thousand years or more in the proxy data' but in reality the situation is not quite so simple." Briffa, whose data contains the "decline," commendably goes on to argue for a more honest and nuanced description of the data.

It's not unusual for researchers to choose data to tell a particular story. And there really is no evidence that the researchers suspected the story they were telling--of unprecedented, industrially-caused warming--was wrong. But it is clear that the handful of teams around the world who evaluate historical climate trends displayed their data to support a clear end goal. Subsequent revelations show that this mindset also affected the choice of references in the IPCC report, especially with regard to the impacts of climate changes.

One of my correspondents, a biologist who uses complex biological models, likens the climate-science consensus to Lysenkoism--the Soviet era anti-Darwinist biological agenda. I take this person to mean that dissent from the reigning paradigm is not accepted in mainstream scientific circles: anyone questioning the "consensus" is quickly branded a "denier." This is not the way to do science, especially for something as important as climate change. And what it means is that the "overwhelming consensus" is not as convincing as it seemed a few months ago.

But it doesn't mean that the consensus is wrong.

 

Blinded Science

Medical researchers systematically shield their results from their own biases. Other scientists, not so much.

Medical science includes many types of studies, but the universally accepted "gold standard" is the randomized, controlled, double-blind, prospective, clinical trial. In this kind of study, patients are randomly assigned to either receive the treatment being tested or an ineffective substitute and then watched for their response, according to predefined criteria.

"Blind" means that the patients don't know whether they are getting the "real thing" or not. This is important because some will respond pretty well even to a sugar pill, and a truly effective drug must at least do better than that. This feature may not be so important in other fields, assuming that the rats or superconductors they study don't respond to this "placebo effect."

But the "double" part of "double blind" is also critical: in a proper trial, the doctors, nurses, and the researchers themselves don't know which patients are getting the real treatment until after the results are in. Without this provision, experience shows, they might treat the subjects differently, or evaluate their responses differently, and thus skew the conclusions. The researchers have a lot invested in the outcome, they have expectations, and they are human.

So are other scientists, I'm afraid.

It is surprising that most fields don't expect similar protection against self-deception. Sure, it's not always easy. In my Ph.D. work, for example, I made the samples, measured them, and interpreted the results. Being at the center of all aspects of the research helped keep me engaged and excited in spite of the long hours and modest pay, and was good training. But I also felt the pressure to get more compelling results.

These days, many fields already involve multidisciplinary collaborations that leverage the distinct skills of different specialists. Would it be so hard for the person who prepares a sample to withhold its detailed provenance from the person doing the measurement until the measurement is finished? With some effort, people could even hide information from themselves, for example by randomly labeling samples. No doubt it takes some of the immediate gratification out of the measuring process, but the results would be more trustworthy.

As recent events have made clear, trust is especially important in climate change. In October, Seth Borenstein, a journalist with the Associated Press, gave a series of measurements to four statisticians, without telling them what the data represented. In each case the experts found the same thing. When the data were revealed to be temperature measurements, each expert had found a long-term upward trend in global temperatures, with no convincing sign of the supposed cooling of the last ten years.

But why did it take a journalist to ask the question this way? Shouldn't this sort of self-doubt be built into all science at a structural level, not just assumed as an ethical obligation?

Experimental science will always be a human endeavor, I hope, so the results can never be completely divorced from expectations. But there are ways of making it more trustworthy.

Beautiful Data

There's an old joke where a scientist presents "representative data," when everyone knows it's really their best data. Like many jokes, there's a large measure of truth in it. (And like many science jokes, it's not actually funny unless you're a scientist.)

Audiences, whether in person or in print, like a good story. And it's best when data tell a story on their own, without further words of explanation. Scientists can handle tables of numbers better than most people, but, even for them, pictures tell the most compelling stories.

In fact, many experienced researchers begin preparing a new manuscript by deciding what figures to include. In part this is because figures take a lot of space in journals, but in addition many readers will go from the title straight to the figures, bypassing even the short abstract. If the pictures don't tell a good story, readers may just move on.

This is what it means when scientists say data are "beautiful": not that they have some intrinsic aesthetic appeal, but that they tell a good story about what's happening. Ideally, the story is compelling because the experiments have been done very well. But that's not always the reason.

Some of the most beautiful data I ever saw, in this sense, were presented at Bell Labs by Hendrik Schön in early 2002, at a seminar honoring Bob Willett and the other winners of that year's Buckley Prize. In contrast to Willett's painstaking work over many years elucidating the properties of the even-denominator fractional quantum Hall effect, Schön presented one slide after another demonstrating a wide variety of phenomena in high-mobility organic semiconductors.

The problem was that the story the beautiful data were telling was a lie. Schön's rise and fall were expertly described in Eugenie Reich's 2009 book Plastic Fantastic, and I was later on the committee that concluded that he had committed scientific misconduct.

But honest scientists also must be careful when they choose data that supports the story they want to tell. There is an intrinsic conflict of interest between telling the most compelling story and facing honestly what the data are saying. Decisions about which data to omit, and how to process the remainder, must be handled with great care.

As a rule, scientists overestimate their objectivity in selecting and processing data. As individuals, they are more swayed by expectations than they would like to admit. Collaborators can help keep each other honest, but only to a degree.

One thing that keeps science on track in these situations is that other people may make the same sort of measurements. Often different experimenters have a different idea of what's right, and the back-and-forth helps the field as a whole converge toward the truth.

But what happens when a whole field expects the same thing? There's a real danger that the usual checks and balances of scientific competition will break down, and all the slop and subjectivity of experiments will be enlisted in service of the common expectation.

Just because data is beautiful--that is, tells a good story--doesn't mean that story is right.

Monday Morning Quarterbacks

Many news stories simply report the facts; the better ones put the facts in the context and explore their likely impact.

Then there are stories that explain "why."

I have a simple assessment for these explanation stories that saves me a lot of time. This is it:

"What's new?"

More specifically, is there anything about the "explanation" that wasn't known before the event actually happened? If not, click on.

A classic example is election-night analysis. Even before all the votes are counted, pundits materialize to explain what message the voters were trying to send, and which campaign screwed up.

Unfortunately, most of what they say was just as true the day before. Sure, they now have more precise results, and maybe a geographic breakdown and some exit polls. But most of the commentators' facts were already known to everyone. Somehow the surrounding story seems more profound once it aligns with actual events--and once the arguments for the other side have been conveniently forgotten.

Stories about stocks can also be amusing, or pathetic. It's hard to find any report on a market move that doesn't attribute it to concern about Chinese exports, or some such. When the market is truly uncooperative, analysts resort to saying that it "shrugged off" some really dire economic news. Sorry, guys. If you knew how the market would react, you'd be rich.

Unfortunately, the challenge of explanation applies to the real economy as well. I'm a regular reader of Paul Krugman, who warned a year ago that the stimulus package would likely not generate enough jobs. Sure enough, we now have an unemployment rate that once would have been thought unacceptable. So was Krugman right? Or were the conservatives who said the stimulus just wouldn't work?

The sad fact is we hardly know any better now than we did a year ago. We know what happened, of course. But we still don't know what would have happened if we had done something different. Evaluating such past hypothetical situations requires the same kind of modeling, and relies on the same ideological assumptions, as predictions do. Unsurprisingly, the experts mostly see the past the same way they saw the future.

The same problem may apply to climate change. I don't usually think of the existence of global warming as a political question (as opposed the policy response). In a hundred years, after all, liberals and conservatives will both suffer the same heat and drought and see the same sea level rise, or not. But even then, they probably won't agree on what we did, or didn't do, to get them there. We don't have a duplicate world to do control experiments on.

As Yogi Berra is quoted, "Predictions are hard, especially about the future." Sadly, they are almost as hard about the past.

Outliers

One of the most subtle and perilous questions in science is when to omit data.

Sometimes there are really good reasons to leave something out. After all, there are lots of ways to screw up data.

In the old days, people could read instruments wrong, or write or copy measurements incorrectly. Even with data acquired and processed by computers, instruments can overload or malfunction and produce incorrect readings. More frequently, even if a measurement itself is correct, changes in the apparatus or the external context can destroy its apparent significance. And it's almost always possible to save data in the wrong place or with the wrong description.

And it matters. Wrong data can cause a lot of headaches. Many analyses reflect a statistical representation of the complete data set, such as the average value, for example, and curve fitting typically penalizes large deviations even more than small ones. So even a single errant measurement can distract from many good ones.

All this means that scientists have good and powerful reasons to eliminate "outliers" that fall outside the normal range of variation, since there's a good chance they are wrong and could skew the results away from the "real" answer.

The problem is that eliminating points requires a subjective judgment by a human experimenter. Often this person is testing a hypothesis, and so has a working expectation of what the data "should" look like. The experimenter will be strongly motivated to toss out points that "don't look right"--even if that just means they are unexpected. That temptation must be avoided.

Distinguishing truly nonsensical measurements from those that simply don't accord with a researcher's expectations requires a level of objectivity and humility that is rare in most people, and difficult even for well-trained scientists.

But it is one of a scientist's most important tasks.

I once heard someone say that it's OK to throw out one data point out of seven. I think that's ridiculously general, and also dangerous. Human nature being what it is, I think the standards need to be higher.

What I learned in my undergraduate laboratory class is that you should check the data as you go along (plotting it by hand in your ever-present lab notebook, if you must know how old I am), to be ready for any measurement problems that arise. If a measurement looks funny, repeat it. If the repeat is what you originally expected, it may be OK toss out the funny one. The repeat might be an individual point, or an entire series. Even better is to do the new measurement twice, and use the majority rule.

Unfortunately, it's not always possible to repeat the measurement exactly. Another alternative is to make a similar measurement, for example with a similar sample. Whenever possible, replication should be part of normal quality control anyway, so this may not be too hard.

But what about when no repeat is possible at all, as happens in historical sciences? You could just throw out all the measurements as unreliable and find a new line of work. But if you opt to toss some of it and not the rest, you really need a very good argument about why that data has a problem. This is a really slippery slope, if there is no way to double-check your argument. People--including scientists--are notoriously good at coming up with post-hoc "just-so" stories for why things are the way they are.

If you really think the data is wrong, but you can't be sure everyone would agree with your logic, scientific tradition still gives you an option: say what you did. Whenever a data is chosen or processed according to a questionable procedure, proper conduct requires that you declare the procedure, certainly in any journal article.

Unfortunately, I have the feeling that, in the era where hot results are sent to general-interest journals like Scienceandnature, this sort of documentation is relegated to the supplementary material or never stated at all. This is a dangerous trend.

Incidentally, many definitions of scientific misconduct include errors of omission. For example, here is the relevant definition from the National Science Foundation's policy:

Falsification means manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.

In other words, if your deliberate omission distorts the conclusion, you are guilty of fraud. Don't do it.

    

The Honest Broker

In case you hadn't noticed, discussion of global warming has become somewhat polarized. Amid accusations, on the one hand, that industry-financed non-experts deliberately sow confusion, and on the other that a leftist cabal exaggerates the risks and threatens our economy, Roger A. Pielke, Jr. is something of an anomaly.

A professor of environmental studies at the University of Colorado, Pielke is an expert who endorses the broad consensus that humans are causing dangerous changes. But he also criticizes scientists like those on the Intergovernmental Panel on Climate Change for stifling legitimate dissent in the service of narrow policy options. In his 2007 book, The Honest Broker: Making sense of science in policy and politics, Pielke touches on climate change only tangentially as he outlines how scientists can more constructively contribute to contentious policy decisions.

Reading the title, I thought at first that I understood what Pielke meant by an "Honest Broker." As an undergraduate thirty years ago I dabbled in the still-young academic field of Science, Technology, and Society. Books like Advice and Dissent: Scientists in the political arena, by Joel Primack and Frank Von Hippel illustrated how scientists who step outside their specialized knowledge to advocate particular policies risk both their own credibility and that of science. To preserve the authority of expertise, scientists should be careful and clear when they spoke outside of their specialty.

But the intervening years, Pielke says, have shown that the whole notion that science provides objective information that is then handed over to inform policy makers, the so-called linear model, is naïve and unrealistic. Only rarely, when people share goals and the relation between causes and effects is simple, can scientists meaningfully contribute by sticking to their fields of expertise as a "Pure Scientist" or by providing focused answers to policy questions as a "Science Arbiter."

More frequently, people do not share goals and the causal relationships are more complicated. Scientists who wish to contribute to these policy debates are naturally pulled into the role of "Issue Advocate," marshalling the science in support of a narrowed range of politically-supported options. Although this is a useful role, Pielke warns, scientists often drift into it unwittingly. As they deny any political influence on their scientific judgments, these "stealth issue advocates" can damage the authority of science even as they obscure the true nature of the political decision.

To address this problem, Pielke pleads for more scientists to act as "Honest Brokers of Policy Alternatives," to give his complete description. Such scientists, presumably as part of multi-disciplinary committees like the now-defunct Congressional Office of Technology Assessment, would act to expand the available policy alternatives rather than restrict them. Unlike the science arbiter, Pielke's honest broker recognizes an inseparability of policy issues from the corresponding scientific issues, but nonetheless provides a palette of options that are grounded in evidence.

In my technology research, I've had my own complaints about the analogous linear model. I've found that pure research often leads to more research, rather than to the promised applied research and products that make everyone's lives better. But Pielke's criticism of the linear model is more fundamental. He correctly notes that in many complex situations, scientific knowledge does not, on its own, determine a policy outcome. But he then seems to conclude that there is no legitimate role for objectively valid science that can narrow policy options.

In discussing Bjørn Lomborg's The Skeptical Environmentalist, for example, Pielke says "Followers of the linear model would likely argue that it really does matter for policy whether or not the information presented in TSE is 'junk' or 'sound' science." He then shows that for some criticisms of the book, the validity of the science was irrelevant to policy. But many of the standard talking points raised by global-warming skeptics are well within the bounds of science, so clarifying them is a useful narrowing of options, even if it doesn't lead to a single, unanimously correct policy.

Nonetheless, Pielke's short, readable book provides a helpful guide for what we can hope for in policy debates involving science, and how scientists can most productively contribute. What we can't hope for is a single, science-endorsed answer to complex issues that trade off competing interests and conflicting values. For that, we have politics.

Green Computing

Supercomputers run the vast simulations that help us to better predict climate change--but they also contribute to it through their energy consumption.

The lifetime cost of powering supercomputers and data centers is now surpassing the cost of buying the machines in the first place. Computers, small, medium, and large, have become a significant fraction of energy consumption in developed countries. And although the authors of Superfreakonomics may not understand it, the carbon dioxide used to supply this energy will absorb, during its time in the atmosphere, some 100,000 times more heat energy than that.

To draw attention to this issue, for the last two years researchers at Virginia Tech have been reordering the Top500 list of the fastest supercomputers, ranking them according their energy efficiency in the Top Green500 list. I have a wee news story out today on the newest list, released last Thursday, on the web site of the Communications of the Association for Computing Machinery. The top-ranked systems are from the QPACE project in Germany, and are designed for quantum chromodynamics calculations.

Calculating efficiency isn't as straightforward as it sounds. The most obvious metric is the number of operations you get for a certain amount of energy. This is essentially what Green500 measures in its MFLOPS/W, since MFLOPS is millions of floating-point operations per second and watts is joules per second.

As a rule, however, this metric favors smaller systems. It also favors slower operation, which is not what people want from their supercomputers. Some of the performance lost by running slowly can be recovered by doing many operations in parallel, but this requires more hardware. For these reasons, the most efficient systems aren't supercomputers at all. The Green500 list works because they only include the powerhouse machines from the Top500 list, which puts a floor on how slowly the competing machines can go.

Over the years, researchers have explored a family of other metrics, where the energy per operation is multiplied by some power of the delay per operation: EDⁿ. But although these measures may approximately capture the real tradeoffs that systems designers make, none has the compelling simplicity of the MFLOPS/W metric. This measure also leverages the fact that supercomputer makers already measure the computational power to get on the Top500 list, so all they need to do extra is measure the electrical power in a prescribed way.

These systems derive much of their energy efficiency from the processor chips they use. The top systems in the current list all use a special version of IBM's cell processor, for example. I worked on power reduction in integrated circuits more than a decade ago--an eternity in an industry governed by Moore's Law--and some of my work appeared in a talk at the 1995 International Electron Devices Meeting. I also served for several years on the organizing committee of the International Symposium on Low Power Electronics and Design, but I'm sure the issues have advanced a lot since then.

In addition to the chips, the overall system design makes a big difference. The QPACE machine, for example, serves up as about half again as many MFLOPS/W as its closest competitor by using novel water-cooling techniques and fine-tuning the processor voltages, among other things. These improvements aren't driven just by ecological awareness, but by economics.

There's still lots of room for improvement in the energy efficiency of computers. I expect that the techniques developed for these Cadillac systems will end up helping much more common servers to do their job with less energy.

Climate Cover-Up

In their new book, Climate Cover-Up: The Crusade to Deny Global Warming, James Hoggan and Richard Littlemore waste little time wringing their hands about the reality or seriousness of the global warming threat. They dispense with this question quickly, showing that the essential features of carbon-dioxide induce warming have been known for over a century.

Although a devil might lurk in the details, the recent state of the science is captured by Naomi Oreskes' 2004 literature study in Science, which found zero dissenters from the consensus among 928 journal articles referencing "global climate change." Similarly, the 2007 Fourth Report of the Intergovernmental Panel on Climate Change, whose political charter leads it to avoid poorly understood possibilities like collapsing ice sheets, nonetheless states that "most of the observed increase in global average temperatures since the mid-20th century is very likely due to the observed increase in anthropogenic [greenhouse gas] concentrations."

In contrast, a Pew Survey released last week concludes that only 36% of Americans think there is solid evidence that the Earth is warming because of human activity, down from an already low 47% over the past few years. Climate Cover-Up explores how it has come to pass that the public still thinks that this is an open scientific question. Hoggan and Littlemore describe the extensive, organized efforts to make it appear open, largely funded by corporations with much to lose from effective climate actions.

Hoggan is a public-relations professional who says that PR people have a duty to serve the public good. In 2005 he founded DeSmogBlog to highlight just the sorts of systematic distortions that the book catalogs, and Littlemore is editor at the site. Their highly readable book describes these efforts, and the funding behind them, with journalistic precision and documentation.

Their laundry list of deception includes "astroturf" groups that use sanitized corporate funds to present a "grass roots" appearance; "think tanks" that increasingly eschew analysis for promotion of policies that favor their sponsors, and petitions signed by scientists who are not or who have little or no expertise in climate. Hoggan and Littlemore systematically discuss these and other programs to frame the "debate" as one in which huge uncertainties remain--as if that should be a source of comfort.

Sowing doubt is a tried-and-true strategy for delaying government response. David Michaels' excellent 2008 book, Doubt Is Their Product, for example, and Devra Davis' 2007 The Secret War on Cancer related how the tobacco industry perfected this technique to delay serious government action against their product for decades. Some of the same firms are coordinating the skeptical response to climate change, and some of the same scientists, like Frederick Seitz and S. Fred Singer, have played roles in both controversies (despite having expertise in neither cancer nor climate).

Hoggan and Littlemore describe how these omnipresent figures benefit financially from their support of corporate needs, and they reveal the irrelevance of many of the "thousands" of signatories on some highly publicized petitions. But they don't address why other scientists--many intelligent and sincere--sign on to such statements. Why are researchers who have no expertise in climate, as well as members of the public, so willing to question those who do, when they would never presume to second guess articles about cancer treatments or particle theory?

In the end, though, these efforts have achieved their goal: keeping the journalistic treatment of global warming "balanced," unlike the clear trend in expert opinion. This problem was captured in a 2004 study by Maxwell T. Boykoff and Jules M. Boykoff, "Balance as bias: global warming and the US prestige press" which was published in 2004 in the journal Global Environmental Change, and in Chris Mooney's story, "Blinded by science: how 'balanced' coverage lets the scientific fringe hijack reality" in Columbia Journalism Review later that year.

This book is not likely to convince true skeptics of the seriousness of global warming. For those who understand the stakes, however, the book is a powerful inoculation to help recognize the conspiracy-theory talking points, most recently regurgitated by the authors of SuperFreakonomics, for the misinformation it is.

Where there's smoke, there's not always fire. Sometimes, it's a massive corporate-sponsored campaign to blow smoke.

Blogstorm Warning: SuperFreakonomics

Is all publicity good publicity? Stephen Dubner and Steven Levitt, the authors of the bestselling book Freakonomics and the like-named blog, may find out. I, for one, have shelved any inclination to buy their new book, SuperFreakonomics: Global Cooling, Patriotic Prostitutes, and Why Suicide Bombers Should Buy Life Insurance, after seeing the blogosphere's reaction to its climate-change chapter.

The battle was joined in this post by the vocal climate-change activist (and fellow MIT physics PhD) Joe Romm. Romm followed up here, here, here, and here, giving some credence to the idea that he has an axe to grind. But in light of the wide audience the book is likely to get, a preemptive takedown might be justified.

Among Romm's claims is that the Steves misrepresent the views of Ken Caldeira, the Stanford ecologist who has been promoting research into geoengineering approaches to global warming. Romm quotes the book as saying of Caldeira, "his research tells him that carbon dioxide is not the right villain in this fight." Meanwhile, Caldeira's web page prominently features this statement: "Carbon dioxide is the right villain," says Caldeira, "insofar as inanimate objects can be villains." Sounds pretty clear.

Romm may not be entirely clean, though. Dubner claims that Romm goaded Caldeira into disavowing the book's characterization. Roger Pielke Jr., who has had his own battles with Romm, regards him as a liar. It does look as though Romm may have compromised his credibility, although his latest post makes a good defense.

But even if Romm made mistakes, it doesn't make those in SuperFreakonomics any more excusable. It is simply disingenuous to claim, as Dubner does, that the "Global Cooling" in the subtitle is supposed to refer to geoengineering solutions, not to the canard that there was a scientific consensus in the 1970s that climate was in danger of cooling. As painstakingly tabulated by Brad DeLong, this is just one of a whole host of misleading or mistaken statements in the book. Paul Krugman also takes issue with the Steves' take on a particular economics case for early action.

Krugman sums up the problem with Freakonomics' trademark contrarianism:

Clever snark like this can get you a long way in career terms — but the trick is knowing when to stop. It's one thing to do this on relatively inconsequential media or cultural issues. But if you're going to get into issues that are both important and the subject of serious study, like the fate of the planet, you'd better be very careful not to stray over the line between being counterintuitive and being just plain, unforgivably wrong.

The Map and the Territory

The map is not the territory. Alfred Korzybski

I confused things with their names: that is belief. Jean-Paul Sartre

Ceci n'est pas une pipe. René Magritte

In fields ranging from economics to climate to biology, scientists build representations of collections of interacting entities. Everyone knows that the real systems have so many moving parts, influencing each other in poorly known ways, that any representation or model will be flawed. But even though they understand the limitations, experts routinely talk about these systems using words that come from the models, rather than from reality. Climate scientists talk of the "troposphere," economists talk of "recessions," and biologists talk of "pathways." Such concepts help us organize our thinking, but they are not the same as the real thing.

Sometimes the difference between the "map" and the "territory" is manageable. Roads and rivers are not lines on a piece of paper, but they clearly exist. Similarly, the frictionless pulleys and massless ropes of introductory physics have a simplified but clear relationship to their real-world counterparts (at least after you've spent a semester learning the rules). Still, it's easy to get sucked into thinking of these well-behaved theoretical entities as the essence, the Platonic ideal, even as one learns to decorate them with friction and mass and other real-world "corrections."

For many interesting and important problems, however, the conceptual distance between the idealizations and the boots-on-the-ground reality is much larger. You might think that experts would recognize the cartoonish nature of their models and treat them as crude guides or approximations, rather than fundamental principles partially obscured by noisy details. Judging from the never-ending debates in economics, however, the more obscure the reality, the more compelling the abstractions become.

Even in less contentious fields, experts can mistake the models for reality. For example, the fascinating field of systems biology aspires to map networks containing hundreds or thousands of molecules using high-throughput experiments like microarrays and computer analysis. Although one might like to describe all these interactions using coupled partial differential equations, researchers would often be happy simply to list which molecules interact. This information is often represented as a graph--sometimes called a "hairball"--which represents each molecule as a dot or node, and interactions as a line or edge connecting them.

Finding such graphs or networks is a major goal of systems biology. In principle, an exhaustive map is more useful than the traditional painstaking focus on particular pathways, which are presumably a small piece of the entire network. But to yield benefits, researchers need to understand how "accurate" the models are.

A few years ago, a group of systems biologist decided the time was ripe to critically evaluate this accuracy. They established the "Dialogue on Reverse Engineering Assessment and Methods," or DREAM to compare different ways of "inferring" biological networks from experiments. (I covered the organizational meeting, as well as meetings in 2006, 2007, and 2008, under the auspices of the New York Academy of Sciences. A fourth meeting, which like the third will be held in conjunction with the RECOMB satellite meetings on Systems Biology and Regulatory Genomics, is scheduled for December in Cambridge, Massachusetts.) These meetings, including competitions to "reverse engineer" some known networks, have been very productive.

Nonetheless, one thing the DREAM meetings made clear is that "inferring" or "reverse engineering" the "real" networks is simply not a realistic goal. Once the networks get reasonably complicated, it's essentially impossible to take enough measurements to clearly define the network. The ambiguity even applies to networks that actually have been engineered, that is, created by people on computers. The "inferred" networks are a useful computational device, but they are not "the" network. And they never will be.

For these reasons, many researchers think the only proper way to assess the results is by comparing to experiments. If the models are good, they should not only match observed data, but should extrapolate to accurately predict what happens in a novel situation, such as the response to a new drug. Interestingly, the most recent DREAM challenges included tasks of this type. Disappointingly, however, the methods that best predicted the novel responses simply generalized from other responses: they did not include any network representation at all!

It seems reasonable to expect that a model that tries to mimic the internal network, even if it is flawed, would better predict truly novel situations. But it's hard to know what it will take for the system to hit a tipping point where it does something completely different, which was never observed before or included in the modeling. Often, we won't recognize the limitations of our complex models--in biology, climate, or economics--until they break.