The Sherman Kent Center for Intelligence Analysis

Occasional Papers: Volume 3, Number 1, Oct. ‘04

Making Sense of Transnational Threats
Warren Fishbein and Gregory Treverton
Sherman Kent Center

The Kent Center is pleased to publish as part of its Occasional Papers series this “think piece” on how to understand and address the unique analytical challenges posed by complex and fast-moving transnational threats such as terrorism, WMD proliferation, and organized crime. Co-authored by Warren Fishbein of the Kent Center’s Global Futures Partnership and Gregory Treverton of the RAND Corporation, the paper proposes some practical ideas for adapting the organizational culture and processes in which analysis of these issues is done to improve understanding and warning.

The authors use as a springboard for their discussion the ideas generated by a series of unclassified, multidisciplinary workshops with outside experts convened by GFP and RAND during 2003 to explore “Developing Alternative Analysis for Transnational Issues.” (Reports of these workshops are published separately by RAND Corporation in report CF-200.) In this paper, workshop insights are coupled with findings from further research on concepts such as intuitive thinking, sense-making, and mindfulness to suggest an approach for applying what the authors call “alternative sense-making” to complex transnational issues.

The ideas suggested here, however, are less a prescription for analytical practice than an invitation to dialogue, debate, and further research that will help advance the doctrine of analysis for transnational threats. The Kent Center welcomes this contribution to the literature on intelligence analysis and looks forward to continued exploration of the arguments presented here and in an abridged version, “Rethinking ‘Alternative Analysis’ to Address Transnational Threats,” published in Kent Center Occasional Papers, Volume 3, Number 2.

The Director
Sherman Kent Center

Table of Contents

Summary

Introduction

The Logic of Alternative Analysis

The Nature and Role of Alternative Analysis

Text Box: Selected Alternative Analysis Techniques

Looking at Transnational Issues: How Are They Different?

Beyond State and Transnational: Seeking a Better Taxonomy

Table 1: Transnational vs. Traditional Targets

Text Box: Puzzles and Mysteries

Understanding Complex Issues: Intuitive Thinking and The Concept of “Sense-Making”

“Sense-Making” in High-Reliability Organizations

Applying Alternative “Sense-Making” to Transnational Issues

Table 2: Key Practical Ideas

Conclusion

Summary

Intelligence Community analytic organizations need to institutionalize processes to consider whether and how they might “have gotten it wrong” to enhance their abilities to anticipate potential threats in highly complex, fast-moving transnational issues, such as terrorism and weapons proliferation. Such processes would involve sustained, collaborative efforts by analysts to question their judgments and underlying assumptions, employing both critical and creative modes of thought. For such processes to be effective, significant changes in the cultures and business processes of analytic organizations will be required.

These are the key conclusions arising from a project undertaken by the Kent School’s Global Futures Partnership and the RAND Corporation to rethink “alternative analysis”—tools designed to help analysts and decisionmakers employ rigorous self-review, question judgments, and explore alternative outcomes—to better address threats in the increasingly important realm of transnational issues. In a series of unclassified workshops, Intelligence Community analysts and analytic managers came together on a nonattribution basis with outside thinkers in a broad range of fields relevant to the analytic process, including cognitive psychology, psychiatry, organizational behavior, artificial intelligence, knowledge management, intelligence studies, and the foreign policy process. Through presentations and discussions among participants, the workshops sought to generate broad concepts about adapting alternative analysis to enhance warning of out-of-the-ordinary actions undertaken by nonstate actors, epitomized in the September 11 attacks. What follows in this report are some of the more intriguing ideas that surfaced at the workshops, arrayed and developed by the project leaders into a systematic argument.

If traditional intelligence analysis generates forecasts or explanations based on logical processing of available evidence, alternative analysis seeks to help analysts and policymakers stretch their thinking through structured techniques that challenge ongoing assumptions and broaden the range of possible outcomes considered. Properly applied, it serves as a hedge against the natural tendencies of analysts—like all human beings—to perceive information selectively through the lens of preconceptions, to search too narrowly for facts that would confirm rather than discredit existing hypotheses, and to be unduly influenced by premature consensus within analytic groups close at hand. In theory, use of alternative analysis techniques can help to reduce the likelihood of “intelligence failures,” which historically have stemmed in part from such mental errors (e.g. the ingrained belief that the Japanese could not mount a successful attack against Pearl Harbor). In reality, however, alternative analysis has not been particularly effective within the Intelligence Community. It has been employed only sporadically at best, and more often than not as a “nice-to-have” supplement tacked on to traditional analysis rather than integrated at the outset as an essential component of the analytic enterprise in a world of uncertainty and deception.

But an even more significant problem with alternative analysis, looking ahead, is that it may be less effective at enhancing the warning process for threats in the transnational realm than it could be, at least theoretically, in the more traditional state-to-state realm. This is so because alternative analysis, like analysis of any kind, involves the application of structured argumentation to address discrete questions—what if Ruritania acquires “the Bomb?” Yet, many issues in the transnational realm may not lend themselves as readily to such formal approaches as do more traditional types of intelligence problems.

Analysis involves breaking down problems into constituent parts, such as causes and effects, and using logical operations to identify and test hypotheses for the purposes of explanation and prediction. Analysis is relevant for bounded problems in which there is a restricted range of outcomes or hypotheses to be evaluated. Many classic state-to-state intelligence issues in which there are well-established patterns fit into this category, including the status of a rival’s military capabilities or foreign policy objectives. These issues are either, to use familiar terminology in intelligence studies, “puzzles” (theoretically soluble if denied information were obtained) or “mysteries” (which are future and contingent and thus not soluble, but for which several possible outcomes can be identified and ranked in likelihood). Traditional analysis can be effectively deployed against bounded problems, and alternative analysis tools can ensure that more than one conclusion or outcome is seriously considered.

However, intelligence problems in the transnational realm are, generally speaking, less bounded than are those in the state-to-state realm. Transnational groups, like terrorist cells or proliferation networks, are likely to be smaller and more numerous than states, less constrained by rules and historic precedent, and more affected by tactical and situational circumstances, as opposed to deeper internal drivers. They are thus more difficult to understand and more capable of engaging in a wide array of unprecedented behaviors.

To be sure, the distinction between transnational and traditional intelligence problems should not be overstated: there are some state-to-state problems, such as battlefield situations or crisis diplomacy, where situationally driven interactions among a large number of players also can produce a wide variety of outcomes. For either transnational or traditional intelligence problems that are subject to high levels of uncertainty—a third category that can be called “complexities”—analysis is unlikely to prove very effective beyond a very general level. There are simply too many possible causes and effects and too little reliable data to be able to disprove hypotheses.

Such issues can be comprehended, however, through a process of “sense-making.” This concept, developed by management scientists to help business and public- sector organizations cope with uncertainty, also has clear applications to intelligence. Sense-making is a continuous, iterative, largely intuitive effort to paint a picture of what is going on in the environment of an organization. It is accomplished by comparing new events to past patterns, or in the case of anomalies, by developing stories to account for them. A process somewhat akin to sense-making is done everyday in the Intelligence Community in current intelligence, which is a continuous, largely informal effort to update the story line on an issue. It also underlies the key warning concept of “connecting the dots,” which implies holistic recognition or discovery of patterns of behavior.

Reducing the impact of cognitive or organizational biases on judgment in the sense-making process requires an approach different from that of alternative analysis. The aim would not be to rigorously examine alternative assumptions or outcomes but rather to prompt analysts to be continually on the lookout for different types of patterns. It would be, to employ another concept used by organizational decisionmaking experts, to promote “mindfulness” within the analytic intelligence organization. This intellectual orientation, favoring continuous evaluation of expectations and assumptions, is found in many organizations that successfully deal with high levels of complexity and uncertainty–such as aircraft carriers and nuclear power plants. Such organizations do very effective sense-making of their environments, as is indicated by exceptionally low rates of accident (their version of “intelligence failure”). High levels of mindfulness are associated with, among other things, a preoccupation with failure, both past and potential, and a “learning culture” in which it is safe and even valued for members of the organization to admit error and raise doubts.

For intelligence, enhancing mindfulness would be a process, not a tool. That process would be:

• Continual, not discrete or “one-off” efforts. The objective would be to regularly explore different possible outcomes and debate assumptions, linked to all incoming information that could possibly relate to the issue under consideration.
  
• Creative, freewheeling, in place of the more formal alternative analytic process with its strong emphasis on logical argument to come to clear conclusions. It would consciously mix mental biases–for instance, building teams, akin to the practice of some Wall Street firms, known as “barbelling,” which involves pairing young financial professionals with those over 50 to take advantage of both adventurousness and caution born of experience. And it would provide increased time for reflection and conversation, for novel ideas often “pop out” of slow-moving, largely unconscious, contemplative modes of thought, rather than out of more conscious, purposeful, and analytic modes.
  
• Collaborative, because transnational issues inevitably involve teams or even networks of analysts given that these issues cross both national and functional boundaries. Indeed, sense-making might be “out loud”—orally reviewing assumptions and alternative outcomes in a collective forum.
  
• Counterintuitive, seeking disconfirming evidence, not confirming evidence, for instance, by featuring regular—if brief and informal—exercises in which analysts focus on how they could be wrong.
  
• Consumer-friendly, an enormous challenge since sustaining “mindfulness” among time-pressed consumers is even more difficult than getting them to read alternative analysis papers on occasion. It requires new and engaging, experiential intelligence “products” such as RapiSims—sophisticated spreadsheet-based programs that allow consumers to manipulate variables to explore alternative outcomes—and enhanced relationships between analysts and consumers to facilitate informal dialogue about assumptions and outcomes.

Some ways in which these processes could be implemented are summarized in the table below.

Key Practical Ideas

In sum, alternative analysis–or, more accurately, alternative “sense-making”–needs to be conceived of as ongoing organizational processes aimed at promoting sustained mindfulness, rather than as a set of tools that analysts are encouraged to employ. These processes would require creating an organizational culture that, through exhortation, training, and example, values continuous, collective introspection—often difficult to achieve in the existing high-demand, understaffed situations in which analysts find themselves. Implementing such a cultural shift would, therefore, require that it be a high priority of senior intelligence managers, reinforced by changes in reward structures, production schedules and staffing requirements.

Could mindfulness-focused organizational processes really enhance warning of emerging transnational threats? No one can confidently answer yes, but reflecting on past surprises in “complex” situations suggests that even modest improvements could make a significant difference in preparedness. We could posit, for example, that if the concerns of the Phoenix FBI office about flight training before September 11 had been shared broadly within the government and integrated into a mindfulness-focused inter-agency process—featuring collaborative sense-making, web-log type forums, computer-generated references to extant scenarios for crashing airplanes into prominent targets—they might have garnered far broader attention than they did.

Introduction

Heading off surprise has driven US intelligence since the attack on Pearl Harbor, which itself was responsible for the basic shape of the modern American Intelligence Community. September 11 was a powerful impetus to the debate on how to do better; intelligence produced shriller and shriller strategic warning of terrorism over the summer of 2001 but could not follow the specific trail of hijackers in and out of the country, nor provide tactical warning of specific threats at home.

Most of the criticism after September 11 focused on the ragged sharing of information between intelligence and law enforcement, especially the CIA and the FBI, and the way the FBI conceived its business. Yet the attacks also directed attention to how intelligence analysis is done, as was reflected in the frequent refrain about the Intelligence Community’s failure to “connect the dots.” [1] In particular, analysts were alleged to have suffered some version of what foreign affairs columnist Thomas Friedman called a “failure of imagination.”[2] That is, they were so fixed on the risk of bombs on planes that the possibility that terrorists might use a passenger airplane as a bomb had receded from view. More generally, the possibility of a devastating attack taking place on American soil, while acknowledged, was not fully grasped. For some, like Friedman himself, this was an almost excusable outcome of the collective American inability to grasp the nature of “evil.” For others, this was a less pardonable failure to forecast a “predictable surprise”—predictable in that there had been previous attempts or plots to commandeer aircraft for use as weapons that, at least in hindsight, should clearly have shaped analysis. [3]

Over the years, the Intelligence Community, as well as other public and private-sector organizations that have to cope with uncertain futures, have developed tools for rigorous self-review to help avert “failures of imagination.” These tools have collectively become known as “alternative analysis” to distinguish them from more conventional forms of analysis that generate forecasts or explanations based on logical, typically linear processing of available evidence. Alternative analysis techniques seek to help analysts and policymakers stretch their thinking by broadening the array of outcomes considered—examining “alternatives”—or by challenging underlying assumptions that may constrain thinking. Alternative analysis is designed to hedge against the natural tendency of analysts—like all human beings—to focus on information that confirms rather than discredits existing hypotheses, or to be unduly influenced by premature consensus within analytic groups close at hand. Although many of the techniques of alternative analysis have been around in some form for many years, only in the 1990s did the Intelligence Community deem these tools an essential component of the overall analytic effort.

In 2003 the Global Futures Partnership (GFP) in the Directorate of Intelligence’s Sherman Kent School for Intelligence Analysis and the RAND Corporation embarked upon a project to reconsider alternative analysis in light of the growing importance of transnational issues, primarily terrorism, but also including organized crime and weapons proliferation, among others. The starting assumption was that transnational issues presented a different set of analytic challenges than did more traditional intelligence topics targeted primarily on nation states.

The project focused more on how to effectively integrate alternative analysis into the overall analytic and policymaking process for transnational issues than on evaluating specific tools or developing new ones. This emphasis reflected the view that whether for traditional or transnational issues, alternative analysis typically is:

• Used only episodically in the analytic process;
  
• Often viewed, at best, as a “nice to have “ supplement tacked on to conventional analysis rather than as an essential component of the overall analytic process; and
  
• Not particularly effective in influencing the policy process.

The classic example of Pearl Harbor provides vivid testimony in support of the last point. As CIA University President Frans Bax has pointed out, senior US Navy officers produced an alternative analysis-like secret report in early 1941 that quite presciently explored the then somewhat outlandish possibility of an aerial torpedo attack launched from aircraft carriers. [4] The report was initially read with interest by senior officials but eventually ended up gathering dust on a classified bookshelf. It did not appear to have influenced thinking in the immediate runup to December 7th when even modest increases in readiness or alertness could have made a significant difference. [5] It is with the goal of finding ways to keep alternative analysis from similarly “gathering dust” in the present era—so rife with potential discontinuity that it has already been dubbed the “age of surprise”—that this project was undertaken. [6]

To this end, GFP and RAND convened a series of unclassified one-day workshops on better integrating alternative analysis into the analytic process. [7] It examined that question from several different viewpoints—that of the individual analyst, the intelligence organization, and the policymaker. The workshops brought together analysts from the CIA’s Directorate of Intelligence and from other agencies focused on transnational issues, along with a distinguished group of over 30 nongovernmental experts. These experts came from a variety of disciplines relevant to thinking about the analytic process: cognitive psychology, psychiatry, group dynamics, information technology, organizational studies, knowledge management, artificial intelligence, diplomatic history, technology studies, strategic studies, and journalism, along with experts in specific transnational domains such as terrorism and proliferation. The aim of the workshops—which featured both formal presentations and break-out group discussions—was to blend the widely varied perspectives of the participants with the aim of generating new ideas that could ultimately yield more concrete proposals. What follows here is a synthesis of key findings about applying alternative analysis to transnational issues, coupled with the results of further research undertaken by the project leaders in response to ideas raised in the workshops.

The Logic of Alternative Analysis

The underlying rationale for the incorporation of alternative analysis techniques into the analytic process has to do both with the complexity of the subject matter of intelligence and with the limitations of the human mind. Intelligence addresses, almost by definition, issues characterized by high uncertainty, such as the intentions of foreign actors, their capabilities when they are determined to keep those hidden, and the outcomes of complex, interactive systems such as economies and the international system as a whole. [8]

The information base available to analyze these and other types of issues—both openly available and clandestinely acquired—is huge, and it is a great challenge to pick out diagnostic information from the torrent of irrelevant data (the “signals” from the “noise”). Moreover, unlike other intellectual endeavors, intelligence analysis often confronts efforts to shroud key information in secrecy or to actively mislead by planting deceptive information. For any intelligence issue, prediction is at least very difficult, and thus it is useful to systematically consider alternative explanations and outcomes.

But perhaps an even more important reason to consider alternatives involves the fact that analysts do not process information entirely “objectively” because of biases in human perception and judgment. As Richards Heuer and others have argued, all individuals assimilate and evaluate information through the medium of “mental models” (sometimes also called “frames” or “mind-sets”). [9] These are experience-based constructs of assumptions and expectations both about the world in general and more specific domains. These constructs strongly influence what we “take on board,” with information that it is in accordance with our models more likely to be perceived and remembered than information that is at variance with them. They are all the more powerful when analysts or leaders are fatigued and thus lose the ability to think freshly, locking themselves into routine behavior patterns. [10]

Mental models serve a critical function in thinking as they allow individuals to give meaning to, and thus efficiently process, what otherwise would be a morass of data. But they can cause us to overlook, reject, or forget important incoming information that is not in accord with our assumptions and expectations. Mental models strongly influence our “search” processes through what has been called “confirmation bias,” leading us to seek information that supports what we believe to be the case and thus having a major impact on research that underpins intelligence analysis. [11]

Mental models are resistant to change, even in the face of changing external circumstances. A cognitive psychologist observed (at one of the project workshops) that in analyzing complex problems, individuals rely on “what has worked before” and rarely update frameworks (mental models) even when they can no longer explain new data. This is because humans cannot change their way of thinking very quickly without experiencing a disquieting sense of losing control. Experts in their fields, whose highly developed models allow them to make quick and accurate judgments most of the time, are often the most likely to cling to longstanding interpretations in the face of anomalous information.

This is the “paradox of expertise” that has led to astonishing misjudgments and misguided forecasts by bona fide experts in science, technology, and business, among other fields of endeavor. One celebrated example is the head of the US patent office at the end of the 19th century who declared confidently that virtually everything that could be invented had already been invented. [12]

Mental models are not the only factors that can inappropriately influence thinking. Others include “judgmental biases,” such as a common tendency to judge the probability of an event by the “availability” of examples of similar types of event rather than by its mathematical probability; cultural biases, such as a belief that individuals in other cultures will act or react similarly to the way we do; and motivational influences on thinking, such as a desire to avoid addressing unpleasant issues. [13] Because such deviations from purely rational thinking are commonplace, remedial steps such as the use of alternative analysis techniques are essential to avoid misjudgments. [14]

Obstacles to objective thinking affect not only the individual analyst but larger groups as well. Students of decisionmaking have documented the phenomenon of “groupthink” in both public and private organizations, in which the desire for cohesion within small task groups whose members work closely together over time imposes conformity in thinking. [15] Even more insidious is what a leading thinker on organizations has called “organizational lock-in.” A mode of thinking can become so dominant in an organization’s culture that “confirmation bias” becomes embedded in its decisionmaking system. In this case, the organization as a whole, not just individual analysts or decisionmakers, primarily searches for, and inevitably finds, information consonant with prevailing ideas. [16]

To what extent in practice have mental models or other types of bias negatively affected intelligence analysis? In fact, for just about any failure in warning, whether American or foreign, the contribution of cognitive factors is apparent.

• Prior to Pearl Harbor, for instance, US policymakers held assumptions and expectations—that it would be extremely difficult, if not impossible, for Japan to attack a well defended and distant naval base—that contributed to the lack of warning and preparedness.
  
• The Jeremiah Commission blamed the alleged failure of US intelligence to forecast the 1997 Indian nuclear test on cultural “mirror imaging,” which led analysts erroneously to believe that Indian leaders, like many of their Western counterparts, would demur on following through with politically risky campaign pledges. [17]

• In the 1973 Yom Kippur War, to take a non-US example, Israeli officials held tenaciously to their mental model that Egypt and Syria would not attack until they had regained the potential to defeat Israel, blinding them to the danger of an attack with more limited political objectives. [18]
  
• In the private sector, one of the most dramatic cases was IBM’s failure in the mid-1980s to appreciate the implications of then emerging personal computer technology (a field which it could have easily dominated given its tremendous financial and scientific resources), reflecting assumptions about the long-term dominance of mainframe technology. [19]

A distant but still instructive warning failure discussed at length at one workshop was that of French intelligence in 1940, which was unable to comprehend the unorthodox German tactic of invading through the Ardennes Forest, leading to the defeat of the French Army. [20] French intelligence, probably the best in the world at that time, undertook sophisticated analyses of German blitzkrieg tactics in early 1940, all of which supported the belief that the brunt of an attack would come through the plains of Belgium. This increasingly entrenched view actually helped to blind the French general staff to mounting evidence of an Ardennes offensive, such as aerial photography of German pontoon bridge construction in the area and the pattern of German reconnaissance flights, which closely mirrored the later invasion route. This example demonstrates that even “hard information”—often touted as the key to improved intelligence—may be of little use unless it is received by “open minds.”

The Nature and Role of Alternative Analysis

“Alternative analysis” encompasses several techniques geared toward broadening an analyst’s thinking. At its most basic, alternative analysis can take the form of a simple “key assumptions check” by which an analyst explicitly states the underlying beliefs that have been guiding analysis of an issue and then seeks to evaluate their continued relevance. More sophisticated forms include “contrarian” techniques in which prevailing analytic lines and their underlying assumptions undergo direct challenge, and “contingent” techniques that broaden the range of outcomes considered by varying underlying assumptions (see text box). [21] These techniques may be carried out by individual intelligence analysts, groups of analysts, or outside thinkers—alone or in combination with intelligence analysts—who may be better equipped than insiders to bring contrary or divergent perspectives into the analytic process. [22]

As currently applied by the Intelligence Community, alternative analysis techniques address discrete issues and yield “finished intelligence.” Issues are specifically selected because they are subject to doubt or because they are deemed too important to “get it wrong.” A fairly intensive, though time limited, effort is made to challenge assumptions or to identify alternative outcomes, depending on the technique employed, culminating in a written product delivered to relevant policymakers. This approach is, on the one hand, a strength of alternative analysis because nothing can be as persuasive as a well researched, well-argued written product or briefing delivered at the right time to a policymaker. But for amorphous, continuous threats, for which there may be no “right time,” the specificity and product-driven focus of alternative analysis can be more problematic.

The degree to which alternative analysis has been effective in addressing the challenges of uncertainty and mental biases in the intelligence sector is unclear, particularly given the spotty application of these techniques to date. In the business sector, where the technique of scenario planning has been used more systematically over a sustained period, there appear to have been some successes. Royal Dutch Shell, for example, has used scenario planning for some 30 years to anticipate and react to major discontinuities in energy markets (such as the oil embargoes of the 1970s), transforming itself in the process into one of the larger of the major global energy companies. [23]

Shell is far and away the most often mentioned success story, but scenarios are used at least periodically by an estimated one third of US companies—presumptive evidence that bottom-line focused private-sector executives find them helpful. [24]

The potential gains from employing alternative analysis come at some cost, as raised in workshop discussions.

• Alternative analysis efforts sometimes take analysts off-line, risking slip-ups in current coverage.
  
• Contrarian techniques can undermine group cohesion and, on controversial issues, where alternative views are associated with policymaker positions, can present the appearance that analysis has been “politicized”—shaped to produce a particular answer.
  
• Alternative analysis directs attention to outcomes that, while possible, are almost by definition improbable, potentially diverting policy attention and resources away from more likely threats. Harried policymakers looking for “answers” may not understand or appreciate the subtle arguments that emerge from the alternative analysis process, and may be tempted to consider it a form of analytic self-defense.

These downsides, while important to keep in mind, do not outweigh the gains possible from utilizing these techniques, however, particularly given the potential catastrophic consequences of intelligence misjudgments.

Looking at Transnational Issues: How Are They Different?

A key issue examined as part of the project was the extent to which transnational issues differ, as an intelligence challenge, from more traditional state-to-state issues. The prevailing view among project participants—who included analysts with broad experience dealing with both types of issues—was that significant differences do exist. [25] This judgment, it must be noted, appeared to be predicated on an implicit comparison of the dominant threats of recent years—the Soviet Union and Al-Qa‘ida—rather than a more systematic consideration of the full range of transnational and traditional issues. Moreover, some participants argued that there was considerable overlap between state-to-state and transnational issues, suggesting the need to consider a crosscutting taxonomy of intelligence issues (as will be explored later on in this section). That said, we present below participants’ impressions of the key differences because they offer insights into the types of special challenges faced in important segments of the transnational realm.

In comparison to their state-to-state counterparts, transnational intelligence issues were said to:

• Be less “bounded.”
  
• Exhibit a closer action-reaction relationship—or “observer-actor”linkage—between key players.
  
• Unfold more quickly and suddenly.
  
• Present a broader, but lower-quality information base.

Resulting in:

• Requiring more collaborative and multidisciplinary analysis.
  
• Having to support a broader range of consumers.

Less “bounded”: States are, for the most part, clearly “delineated,” having known borders and capitals, and doing much that is openly observable. Moreover, states act within the context of formal and customary rules, such as established military doctrines, international law, and historical precedents, giving predictability to many of their actions. Transnational actors are amorphous, fluid, and hidden, presenting intelligence analysts with major challenges simply in describing their structures and boundaries. And because such actors are also far less constrained by formal rules than their state counterparts, they can engage in a wider variety of tactics on a regular basis, adding immensely to the challenge of forecasting their behavior.

Closer observer-actor linkage: The former Soviet Union was a relatively independent actor, with much of its behavior driven by internal factors rather than by the international system. (Recall the famous line attributed to a US defense secretary about the nuclear relationship between the United States and the Soviet Union: when we build up, they build up; when we slow down, they build up.) By contrast, transnational actors have a more intense relationship with the dominant actor in the international system, the United States. Their tactics are often predicated upon our policies and defensive measures, making their behavior less determinate and predictable. Our understanding of transnational actors’ intentions will lead us to take actions that will—to a greater extent than would be the case with more structured, internally driven state actors—prompt adaptive behavior on their part. This will turn predictions into “self-negating prophecies.”

Events unfold more quickly and suddenly: Intelligence analysts had a decade to explore the impact of Gorbachev’s accession on the Soviet system, according to workshop participants, but in the case of Al Qa‘ida, for example, events have unfolded at a stunning pace since the August 1998 attacks on US Embassies in Kenya and Tanzania. Generally speaking, as flexible and adaptive entities, transnational actors are more capable than most governmental actors of engaging in sudden shifts of behavior. Moreover, tightly linked transnational networks are subject to what network scientists refer to as “cascades,” which are fast, domino-like sequences of events that can result from small perturbations (as in a cascading electric power grid failure). To take a vivid example from the economic domain, a relatively small crisis involving the Thai currency quickly enveloped much of Asia in 1997, as transnationally-linked financial markets suffered cascading withdrawals of funds and confidence. [26]

Broader and lower-quality information base: Because of the unbounded and high-profile nature of transnational threats, analysts must wade through a sea of information that contrasts sharply with the much more limited information available on closed societies such as the Soviet Union. And much of the information is, at best, of uncertain reliability. Moreover, as compared with a state with a long history, much less contextual information is available that can be used to evaluate the reliability of new information. For these reasons, the problem of separating “signals” from “noise” is more acute in the transnational domain, as is, arguably, the problem of detecting deception.

More collaborative and interdisciplinary: While a country political or economic analyst can often work in relative isolation from analysts with other specializations, transnational issues require combinations of regional and functional expertise. Weapons proliferation analysis, for instance, draws upon specialists in science and technology, illicit transfers, money laundering, politics, and network behavior, to name but a few, to track and comprehend the activities of weapons networks. To a much greater extent thanin traditional areas, transnational analysis is a team or even networked activity (as specialists will be located in many agencies). This has both potential benefits in terms of avoiding mental biases (mixing different perspectives) but also potential risks in the form of groupthink and “lock-in,” to say nothing of the inevitable practical difficulties of ensuring robust communication and information-sharing among diverse parties across organizational boundaries.

Consumer base is different: For state-to-state issues, key consumers are relatively few, most of them located at the apex of the national security decisionmaking establishments. Intelligence and its consumers also share a common “story” in dealing with states. Both know that states, even very different states, are territorial and usually hierarchical, with histories, traditions and standard operating procedures. All that is much less true for transnational issues and actors, in part because they are relatively new phenomena. We have, at best, a provisional story about them, or, more likely, tentative, competing stories about them based on often short histories. For this reason, too, the challenge of “connecting” substantively—conceptually and contextually—with the consumer is greater in the transnational than in the state-to-state arena. For transnational issues, the number and variety of consumers is also much greater, extending to a variety of agencies outside the traditional national security establishment. Moreover, it is far less clear who is the “key” consumer. An airport security officer or a public health doctor may have a more urgent “need to know” about a threat than the President because he or she may be in a more immediate position to thwart it. At the same time, most new transnational consumers have little familiarity with intelligence and how to interpret it.

Beyond State and Transnational: Seeking a Better Taxonomy

Some workshop participants, as noted before, argued that the domain of transnational issues could not easily be separated from state-to-state issues. Some transnational issues, such as weapons of mass destruction (WMD) proliferation, feature significant involvement of state as well as transnational actors. Moreover, the transnational arena contains entities that are as bounded and stable as nation-states, such as organized crime syndicates. At the same time, within the state-to-state arena, there are key intelligence issues that, like many transnational issues, are unbounded, fast moving, characterized by adaptive behavior, and obscured by voluminous information, such as crisis diplomacy, failed states, insurgencies or the battlefield environment. The challenges facing analysts in comprehending al Qa‘ida may not, in fact, be that much different from those confronting analysts in the runup to the Battle of France or Pearl Harbor—even as they probably are different from those associated with tracking fairly consistent Soviet behavior during the latter stages of the Cold War.

Table 1: Traditional vs. Transnational Targets

Given overlap between transnational and traditional (or state) intelligence issues, workshop discussions focused on identifying an underlying category displaying the problematic characteristics cited above—unbounded, fast moving, and the like—but which can include both transnational and state issues. One promising, cross-cutting framework was the division of problems into “ordered” and “disordered” categories, as developed to address business decisionmaking issues by the management scientist David Snowden. [27] Ordered problems follow established patterns and thus can be understood through rational means of inquiry; disordered problems do not follow such patterns and thus are not amenable to reductionist techniques. Snowden further divides ordered space into “known” and “knowable” problems, and disordered space into “complex” and “chaotic” problems.

• Known problems are those for which there is a unique deterministic relationship between causes and effects; the challenge is to correctly categorize the problem, obtain the necessary data to solve it, and apply accepted formulas. An example is a process-engineering issue for which there are “best-practice” solutions.
  
• Knowable problems involve contingent relationships between a limited set of causes and effects. In this realm it is possible to use analytic reductionist techniques to predict outcomes, at least probabilistically. For example, an investment decision in an existing business area lies within this realm as one can usually identify the limited set of factors (market demand, strategies of the competition, and so on) that will influence the set of expected outcomes.

• Complex problems involve a wide array of causes and effects that can interact in a variety of contingent ways. This is the realm of interactive systems such as battlefields, ecosystems, and competitive markets in which large numbers of relatively small actors respond to a shifting set of situational factors. Because interactions reflect unique circumstances, they do not necessarily repeat in any established pattern and are not amenable to predictive analysis. The best that can be done in the case of such problems is to “sense” emerging patterns with an eye to reinforcing, disrupting, or taking advantage of them.
  
• Finally, there are chaotic problems where there is no discernible relationship between causes and effects. Such problems (e.g. the aftermath of a natural disaster) require immediate action to bring them to a more ordered state where reflective thought processes, including those used in intelligence, can be effective.

The Snowden framework suggests that there is a class of problems beyond the well-established division of intelligence issues into “puzzles” and “mysteries” (see text box) that, for want of a better term, can be called “complexities.” [28] These problems involve some combination of the following factors: large numbers of small sized actors, fluidity of rules governing behavior, and the large influence of situational as opposed to internal factors in shaping behavior. Due to these characteristics, these problems can yield a wide range of sui generis outcomes that defy probabilistic prediction.

Complex problems are prevalent in the transnational realm because actors are small, numerous, and relatively unbounded by rules, and processes are highly interactive. From the perspective of the intelligence analyst, the presence of overwhelming amounts of information of questionable reliability adds to this complexity because it is difficult even to ascertain current realities.

To be sure, complex problems also describe some state-to-state intelligence issues, such as crisis diplomacy and battlefield intelligence where high interactivity and information overload (e.g. the “fog of war”) are present. One major difference, however, is that crisis or battlefield conditions are usually time-limited or exceptional in the state-to-state realm, whereas equivalent conditions are an ongoing fact of life in the transnational realm.

Understanding Complex Issues: Intuitive Thinking and the Concept of “Sense-Making”

If “complexities” are resistant to reductionism and prediction, then it is important to explore how such problems are or can be addressed by intelligence in order to identify an appropriate “alternative” process for tackling mental biases. To do so, it is useful to begin by reconsidering the concept of “intelligence analysis.” [29] Upon close inspection, this concept actually encompasses not only “analysis,” strictly defined, but also other mental processes that are actually more relevant to the comprehension of complex problems encountered in intelligence issues.

Analysis, according to one reasonable definition, involves breaking down a problem into its constituent parts and then using logical processes to derive useful conclusions, whether this is for the purposes of explanation or prediction. Those logical processes include generating a range of plausible hypotheses and then rigorously evaluating the hypotheses in light of available evidence according to accepted criteria. [30] According to Heuer, such criteria may be:

• Situational (sui generis factors intrinsic to the situation under study, as is used in narrative history);

• Deductive (generalizations about human behavior drawn from social scientific investigation);
  
• Or by analogy (comparing the present situation to historically similar situations with an eye to comparing the validity of cause-effect relationships).

Heuer further argues that good analysis should satisfy an even more stringent requirement. As in scientific investigation, it should focus more on disconfirming rather than confirming evidence, because a hypothesis that is supported by evidence may, in fact, be one of several in that category, in which case the evidence is not diagnostic, while a hypothesis that is disconfirmed can be categorically eliminated from consideration.

Heuer maintains, based on his own experience and observations, that intelligence analysis does not, in practice, really fit any stringent criteria of analysis. It tends to be a relatively informal process of reaching judgments based on confirmatory evidence in which the aim is to “satisfice”—to present the minimally best case that will satisfy consumers or superiors. Rob Johnston, in a very recent unclassified survey of analytic practice throughout the Intelligence Community, came to broadly similar conclusions. [31] He argues that most of what passes for intelligence analysis is reportorial—”CNN plus secrets”—coupled with general reflection and discussion that uses the prevailing line as a starting point. Formal analytic tools (Johnston counts about 160, most of which, however, have been developed outside the intelligence realm) are used sparingly and haphazardly with no common doctrine about their use within, much less across, intelligence agencies. [32] Johnston attributes this state of affairs to the unfamiliarity of subject matter experts with methodological tools, and cites the need to bring analytic methodologists into the intelligence process to promote the employment of formal analytic techniques.

Heuer’s and Johnston’s observations suggest that intelligence analysts engage in mental processes other than analysis, strictly conceived. To define these residual processes, it is useful to draw upon several constructs developed by management scientists to categorize less structured forms of perception and decisionmaking that take place within business organizations.

• One of these is the concept of intuitive decisionmaking, particularly as developed by such thinkers as management expert Gary Klein and cognitive psychologist Guy Claxton. [33]
  
• A second concept is that of organizational “sense-making,”—essentially intuition at the organizational level—as developed by the noted organization theorist, Karl Weick. [34]

Klein argues, based on studies of real world decisionmaking among businessmen, firefighters, and military officers, that most people base decisions on intuition or “gut instinct” rather than upon formal analysis. This process (more formally called response-primed decisionmaking) involves the relatively unstructured application of subject expertise and overall experience to detect patterns of opportunity and threat, which in turn shape action.

Intuitive decisionmaking works in areas marked by well-defined and repetitive problems (such as medical diagnosis) as well as in more uncertain areas where the “gut” provides useful guidance. As one popular business writer has pointed out, commodity traders, who develop well honed gut instincts to survive in a fast moving, highly uncertain market environment, have been able to beat more analytically focused professionals such as US Marines in simulated competitions, even in the latter’s own game. [35] Indeed, clever defense lawyers seek to discredit ballistics experts by requiring them to describe the analytic steps through which they reached their conclusions. They do this knowing that, like chess masters, those experts reach their conclusions not through a strictly analytic process but rather through a difficult to articulate process of patterns as much sensed as analyzed—the results of having seen thousands of previous cases.

Traditional analysis, according to Klein, complements intuition where there is a structured problem to be addressed and time for serious study. But attempts to substitute analysis for intuition in situations marked by high uncertainty can be counterproductive, as the reductionism needed to fit such problems into models or highly structured analytic schemes may produce misleading results.

In another take on intuition, psychologist Guy Claxton lays out two processes of thinking—a “deliberative” mode that corresponds to conscious analytic thinking and a more “contemplative” mode that corresponds to intuitive, subconscious thinking. [36]

• Deliberative thinking is focused on answers, solutions, and logical explanations; is purposeful, effortful, and fast; is reliant upon literal and explicit language; and works well when tackling problems that can be treated as an “assemblage of namable parts.”
  
• Contemplative thinking is focused more on questions and observations, is playful, metaphoric, imprecise, and slow moving, and works best on problems that must be grasped holistically.

Claxton’s concept of intuitive thinking differs in some key respects from Klein’s—it is slow instead of fast moving—but his conclusions about its effectiveness in addressing issues marked by high uncertainty are similar. He writes that “recent scientific evidence shows convincingly that the more patient, less deliberative modes of mind are particularly suited to making sense of situations that are intricate, shadowy or ill defined.” Such situations would seem to correspond to the “complexities” of transnational and other issues.

Weick’s concept of sense-making involves the process through which organizations—vice individuals—comprehend the complex environment with which they must contend. It is a continuous, iterative, largely informal effort to understand, or “make sense” of what is going on in the external environment that is relevant to the organization’s goals and needs. In essence, it is the collective intuition of an organization. Through conversations at all levels, organizations construct ongoing interpretations of reality by comparing new events to past patterns, or in the case of anomalies, by developing stories to account for them. Weick argues that the fluid sense-making process has clear advantages as a framework for organizational action over “decisionmaking” because the latter often locks the organization into polishing and defending formal decisions that may no longer be appropriate in fast changing situations.

Both of these concepts—intuitive decisionmaking and sense-making—can be combined into a framework for categorizing the residual thought processes of intelligence analysts, which, broadly following Weick, can be called intelligence sense-making. This process involves the application of expertise, imagination, and conversation—and the benefit of intuition—within intelligence analytic organizations to identify changes in existing patterns or the emergence of new patterns, without systematic, consideration of alternative hypotheses.

Compared to analysis, intelligence sense-making is continuous rather than discrete, informal rather than formal, and focused more on unbounded rather than on bounded issues. Intelligence sense-making loosely describes the more thoughtful aspects of “current intelligence,” in which newsworthy developments are evaluated in the context of past patterns usually without the benefit of much formal analysis. [37] The concept of “connecting the dots” also bears more of an affinity to sense-making than it does to formal analysis in that it involves holistic pattern recognition, rather than the testing of alternative hypotheses.

Intelligence sense-making offers a way to comprehend “complexities,” whose combination of speed, uncertainty, and interactivity defy traditional analytic approaches. One workshop presenter, a noted expert on intelligence, made this point in observing that, in contrast to traditional Soviet-threat type issues, the transnational arena (terrorism in particular) benefits far more from lots of “pairs of eyes” looking at data for emerging signs of threat than from detailed analyses of narrowly drawn issues. While lots of eyes would, of course, be beneficial in addressing traditional issues, they arguably are crucial to success in transnational issues, where having multiple intuitions wrestling with enormous information flows should significantly improve the odds in favor of identifying threat patterns.

“Sense-Making” in High-Reliability Organizations

Like traditional analysis, intelligence sense-making suffers from the effects of mental biases, and may be even more vulnerable to those biases to the extent that it applies highly developed “mental models” to incoming information. The process of challenging assumptions in this domain—”alternative sense-making”—will be different, however, from that used in the more familiar analytic domain where alternative analysis techniques strive to produce discrete products using formal tools to explore a limited range of outcomes.

A useful starting point in seeking to identify an “alternative analysis” approach for complex environments is Weick’s work with Kathleen Sutcliffe. [38] The authors examined a number of organizations, such as nuclear power plants and aircraft carriers, which face challenges in addressing uncertainty akin to those faced by intelligence organizations. Those organizations confront on a daily basis continually shifting conditions that can give rise to unexpected outcomes, with the potential for catastrophic consequences. An aircraft carrier deck, they note, is the “most dangerous four and one-half acres in the world” with a volatile mixture of jet fuel, weaponry, “controlled crash” landings, shifting sea and weather conditions, noise, and young and often inexperienced personnel. Nonetheless, carriers experience exceptionally low rates of error and accident in comparison to other fields that also continually confront uncertainty, such as medicine. The same can be said of nuclear power plants where, according to another writer, “there is no ‘regular’ functioning of the plant—the status of the plant is continually changing and therefore . . . anomalies are very difficult to track.” [39]

In studying these types of “high- reliability organizations” (HROs), Weick and Sutcliffe have identified lessons that can be applied generally to addressing uncertainty. The unifying trait of HROs is that they exhibit the quality of “mindfulness,” defined as:

“ . . . the combination of ongoing scrutiny of existing expectations, continuous refinement and differentiation of expectations based on new experiences, willingness and capability to invent new expectations that make sense of unprecedented events, . . . and identification of new dimensions of context that improve foresight and current functioning.” [40]

The critical words in the above definition are “ongoing” and “continuous.” Mindfulness is the result of a never-ending effort to challenge expectations and to consider alternative possibilities. Such eternal vigilance helps eliminate “blind spots” that result in organizations missing early warning signs of unexpected and unwanted change. A mindful orientation “redirects attention from the expected to the irrelevant, from the confirming to the disconfirming, . . . from the more certain to the less certain, . . . and from the consensual to the contested.”

Weick and Sutcliffe identify several attributes of organizational culture that contribute to mindfulness. Two that appear to be critical to anticipating uncertainty, as opposed to reacting to it, are most relevant to intelligence organizations:

• A preoccupation with failure, both past and potential.
  
• A “refusal to simplify.”

High-reliability organizations continually seek information on and then examine their own errors, with an eye to improving operations. “They tend to view any failure, no matter how small, as a window on the system as a whole.” This preoccupation also extends to thinking about future downsides: HROs tend to be “skeptical, wary, suspicious of quiet periods,” because they know that “not all failure modes have . . . been experienced or exhaustively deduced.” Underpinning these attitudes is a “learning culture” in which it is safe and even valued for members of the organization to admit error and raise doubts.

The refusal of HROs to simplify involves a refusal to take things for granted or to rely on standard interpretations. Personnel in HROs are relentless in their efforts to try to understand the complexities of the situations they face rather than lumping problems into broad categories such as “make or buy, friend or enemy, profit or loss.” As an example, mechanics in nuclear power plants almost never rely on the simplifications of blueprints when they have to intervene in the system, but rather they personally “walk the system” to assure that no subsequent changes have been made that could affect the outcome. Particularly relevant to the intelligence environment, different departments constantly interact when confronted with a problem, generating hypotheses about “what is going on, what can be done, and what the long-term, system-wide consequences of the proposed action might be.” Such interdepartmental interaction brings different perspectives to the table, building a progressively more complex vision of the problem at hand.

The sense of workshop participants and the project leaders was that, compared to “high reliability organizations,” the Intelligence Community generally:

• Does not undertake the continuous, relentless, critical self-examination that is commonplace in HROs, as is evidenced by, among other things, the generally spotty application of alternative analysis techniques;
  
• Has neither regularly explored its own failings with an eye toward self-improvement nor effectively implemented and internalized findings when studies of failure have been done.
  
• Contains analytic production systems that promote simplification by placing a premium on a clear “bottom line” in intelligence products, with much of the complexity either eliminated or shunted to the rear, and by emphasizing a consistent line on a particular issue in order to avoid “confusing” consumers
  
• Has relatively less interaction among analysts very early in the process, than is the case in HROs, with most of the interaction taking the form of formal coordination after basic assessments have been developed.

Applying Alternative “Sense-Making” to Transnational Issues

The concept of mindfulness, as practiced by HROs, provides a construct for designing processes to improve understanding and warning for complex, transnational issues. For threats that can suddenly emerge at any time, anywhere, and in a variety of forms, analysts need to think more in terms of a broad mental readiness to perceive early warning signs of threat than in terms of challenging specific assumptions or identifying specific alternative outcomes.

The following principles can guide the development of an alternative process for intelligence sense-making. Such a process must be:

• Continual
  
• Creative
  
• Collaborative
  
• Counterintuitive
  
• Consumer-friendly

Continual: The conventional model for employing alternative analysis—identify an issue too important to “afford getting it wrong” and then challenge assumptions and identify alternative outcomes—is not really suitable for ongoing complexities. There are too many outcomes to be considered, too much potential for sudden change, and too many contingent interactions for any “one-off” effort to be particularly useful. Moreover, cognitive research suggests that such efforts may not make a sufficient imprint on thinking to affect ongoing analysis. This is because information that is inconsistent with expectations is less likely to be remembered than information that is consistent. Since alternative thinking goes against the grain of established thought, its ability to have a sustained impact on understanding, individual or collective, is always open to question, all the more so if, as is often the case, analytic cadres turn over fairly rapidly. [41]

A continuous, sustained program of small to medium-sized efforts, however, would regularly explore different possible outcomes and debate assumptions, all linked to incoming information about the issue under consideration. This probably is best thought of as an ongoing conversation (both face-to-face and electronic) among interested parties, structured to encourage divergent thinking. Larger efforts, such as multiple scenario workshops or multiplayer games, would aim at feeding results into the ongoing dialogue, not simply publishing and moving on. Information technology to capture and automatically recall both previous judgments and alternatives, perhaps cued by keywords, would be essential to supplement human memory and to further stimulate debate.

Creative: Traditional alternative analysis is a fairly formal process with some elements of creativity but with a strong emphasis on logical argument to come to clear conclusions. Alternative “sense-making” for complex issues would, by contrast, be more freewheeling and creative. In part, firm conclusions are not desirable given the higher levels of uncertainty inherent in these issues. In addition, the objective is to stimulate pattern recognition—to connect the dots—a creative process in itself. Cognitive research suggests that judgments about the likelihood of events often reflect the “availability” and vividness of memories about similar types of events. [42] To the extent that alternative sense-making can help to suspend premature judgment and make an array of possibilities come alive, it may stimulate exploration of alternative “dot” arrangements.

Increasing creativity within the intelligence field to enhance intuitive judgment can be accomplished in a number of ways. Press reports indicate that the CIA has worked, for example, with the film industry to create audio-visual games to help analysts “think like terrorists.” Such major efforts would have a more lasting impact if they were supplemented by smaller, more frequent efforts geared toward cultivating creativity at both the individual and organizational levels. Systematic use of the wide array of divergent thinking techniques as a precursor to the preparation of intelligence products would help to put a wide range of possibilities on the table before provisional intellectual closure was reached. Johnston’s proposal to complement subject matter specialists with analytic methodologists might be structured to ensure that these professionals possess or acquire expertise in the application of such divergent thinking techniques. [43]

A conscious mixing of mental biases would also help to assure that different types of “eyes” are searching for threatening patterns. The specific suggestion raised in workshop discussions involved the practice—known as “barbelling”—used by some Wall Street firms, which involves pairing young financial professionals with those over 50 to marry adventurousness with caution born of experience. Firms with such an age distribution, according to a noted presenter from the financial community, tend to perform better than those whose professionals cluster in the thirty to fifty age range. Recruiting and assigning of analysts could also be designed to produce teams that consciously mix biases, such as perceptual and judgmental thinking styles (as identified in Meyers-Briggs– type tests).

Providing analysts with much greater time and freedom to think about problems than is normally allowed by hectic intelligence production schedule could also foster creativity, as is suggested by Claxton’s findings. Ideas most often “pop out” of the slow moving, largely unconscious, contemplative mode of thought rather than from the more conscious, purposeful, and analytic mode. Moreover, a state of mental relaxation is also conducive to the playful workings of the contemplative mode, something hard to achieve in the midst of a grueling production schedule. Other options include enabling analysts to pursue independent projects, contacts, and conversations seemingly peripheral to the main tasks at hand, as well as occasional working retreats—for physical setting is related to creativity as well. [44]

Collaborative: Although many forms of traditional alternative analysis are collaborative, an individual analyst can do others, like devil’s advocacy or “what-if” analysis. Such individual efforts are possible because the aim is to develop or to challenge logical arguments, something that the individual mind does well. However, examining alternatives on a sustained basis for transnational issues (and for comparable traditional issues) involves a team effort embracing a variety of disciplines and regional specialties—far beyond the capabilities of a single analyst. Moreover, introspection is extremely difficult for most individuals, as a leading cognitive psychologist noted at one workshop, and is amply documented in psychological literature. To promote sustained awareness of possible flaws in one’s thinking requires a continuous dialogue among individuals with different perspectives.

One collaborative technique discussed at length at the workshops was “out loud sense-making”—a structured process for orally reviewing assumptions and alternatives. Such a process is used by surgical teams at a New England hospital, in response to recurrent errors by anesthesiologists in diagnosing respiratory reactions—often fatal—to anesthesia. Now, when confronted with breathing problems, the team—including nonphysicians—systematically reviews possible causes out loud before arriving at a conclusion, dramatically reducing the error rate. Such processes, enhanced by skilled custom design and facilitation, could be regularly employed in the intelligence arena whenever anomalous data emerged.

Another collaborative technique identified at the workshops involves the greater use of Web-logs (or “blogs”) as a platform for intelligence production. US embassies already use this art form—which allows for continuous posting of information—as their primary means for conveying information. Unlike published papers, intelligence Web-logs would be a form of “unfinished” production in which both intuitions and more formal arguments could be posted, and then challenged by those with alternative opinions. Indeed, Web-logs could be the forum for a facilitated virtual dialogue—the electronic equivalent of out loud sense-making.

Counterintuitive: Intelligence analysts need to focus on patterns that are different, even contradictory to those they expect, to lessen the degree to which their mental models inhibit their ability to perceive new information. And they must do so fairly regularly to promote continuous awareness of possibilities. This is not easy given the natural human tendency—especially under time pressure—to revert to established ways of viewing the world.

One possible approach is to institutionalize brief, informal exercises in which analysts regularly focus on how they could be wrong, along the lines of what Klein calls a “premortem.” [45] In this group exercise, individuals imagine fiascoes relating to their areas of responsibility—in the case of intelligence analysis, a warning failure—and brainstorm about how they might come about and how they could be headed off. The very pointed emphasis on the negative, Klein argues, helps to shake up complacency that arises from common overconfidence in judgments. Such an exercise would be a functional equivalent of the “low-probability/high-impact” exercise of traditional alternative analysis, with the difference that its informal nature lends it to more frequent use.

Workshop participants also discussed changing the culture within intelligence organizations to one that is more conducive to self-questioning. Assessment of analytic performance often is done by outside bodies with negative findings exploited by media and external critics, thus prompting a defensive stance among intelligence professionals. There has been only modest emphasis internally on looking at failures—and even less on examining successes—with an eye to drawing lessons for self-improvement, notwithstanding Sherman Kent’s exhortation for such introspection more than 40 years ago.

The systematic and non-punitive approach of the nuclear power industry to collecting information on, and learning from, errors—associated with its low failure rate—provides one approach worth considering. Another, perhaps more relevant, example is provided by the US Army’s Center for Army Lessons Learned (CALL), which continuously looks at operational problems in order to draw implications for improving performance. A dedicated effort to continually take a friendly look at getting it both right and wrong through periodic “After Analysis Reviews”—modeled on the Army’s After Action Review—might help to create a more introspective environment. [46] Allowing analysts the time for such exercises—or mandating them—would powerfully demonstrate organizational commitment to learning.

Consumer-friendly: One of the greatest challenges of any “alternative” effort is to effectively communicate the message to those who occupy decision-making roles. Decision-makers are buried by both information and tasks. Motivating them to spend time reading sophisticated analysis in general, let alone analysis that queries existing analytic lines, is a considerable challenge. Several presenters at the workshops who had been senior officials in the terrorism area stressed the extent to which information overload had grown in the post-September 11 environment. Moreover, in the transnational domain, many potential key consumers are in middle and lower operational levels, or outside the government, and thus have even fewer contexts for understanding intelligence information. And, of course, in order to carry out “alternative sense-making,” alternative points of view would need to be put before this wide array of harried individuals on a fairly regular basis.

One way to accomplish this is to rethink the concept of the intelligence “product.” Intelligence organizations continue to insist upon written prose and formal briefing as the “gold standard” for disseminating information even though adults rarely retain more than ten percent of what they are “told” either orally or in written form. Instead, more experiential, interactive formats, as discussed at the workshops, might better capture the attention and imagination of intended audiences and strengthen retention of insights.

• Use of Web-logs would give consumers—particularly nonsenior consumers without formal feedback processes—the opportunity to tap in from time to time on debates within the analytic community and to pose questions themselves.
  
• RapiSims—i.e. Rapid Simulation using, increasingly sophisticated spreadsheet-based programs—would allow consumers to manipulate variables to generate alternative outcomes. Decision-makers could quickly and easily explore a range of possibilities in a way that is more likely to be retained than if presented in a long and dry formal tome.
  
• Half day “gaming” sessions—intentionally kept brief to allow even the most harried to participate on occasion—could help decision-makers, at a minimum, experience the uncertainties surrounding an issue.

Another avenue would be to try to strengthen personal relationships, such as through increased face-to-face contacts, between individual analysts and consumers in order to facilitate informal exchanges on alternative outcomes.

One presenter, a specialist in corporate innovation, argued that the real product of the analytic organization is the “analyst rather than the analysis,” just as professionals in other knowledge industries—management consulting, law, etc.—are the key selling points for their organizations rather than specific products. [47] A professional known by and enjoying the trust of consumers is in a far better position to influence their thinking than is any specific report or briefing. Such a professional, if inculcated in alternative sense-making values and processes, also would be in a position to periodically draw consumers’ attention to the results of internal exercises on assumptions and outcomes.

Some of the practical ideas stemming from workshop discussions are summarized in Table 2.

Table 2: Key Practical Ideas

Conclusion

Understanding the complexities facing analysts today requires an alternative analysis approach that is more an ongoing organizational process aimed at promoting continuous “mindfulness” than it is a set of tools that analysts are encouraged to employ occasionally. The latter approach lays the responsibility for alternative analysis too heavily on the shoulders of individual analysts who may not have the incentives or authority to question their own or their colleagues’ lines of thinking. And in the case of highly uncertain, rapidly changing, boundary-crossing threats such as those in the transnational arena, such questioning can really take place only as part of an ongoing, organizationally-supported collaborative effort, given the wide range of disciplines and institutions that contribute to shaping the ultimate analytic product.

With the global landscape likely to feature a growing array of hard-to-track threats in the years ahead—as a result of the increasing availability of technical know-how and soaring interconnectivity, among other factors—the need for a process to produce sustained mindfulness will only grow. Even in the case of more traditional intelligence issues, developing and enhancing organizational processes designed to promote alternative analysis might encourage more systematic use of these tools.

Thinking of alternative analysis as an organizational process has important implications for how the intelligence analytic organization is managed. It means that senior intelligence officials must make challenging assumptions and considering alternative possibilities a high, indeed critical, priority. It calls for reward systems that encourage analysts to think about how they could be or have been wrong. Production schedules and staffing requirements need to be adjusted to allow time for analytic reflection. Throughout the organization there must be a “culture,” one inculcated by exhortation, training, and example, that values continuous, collective introspection.

Rethinking the concept of alternative analysis may even mean rethinking how the intelligence field defines itself. For instance, the CIA’s Directorate of Intelligence has self-consciously modeled itself after the academic realm and to a lesser extent the press, where questions of accuracy—getting the data and facts right—are of paramount importance. But should it not think of itself as more equivalent to an aircraft carrier or nuclear power plant, where questions of failure drive every aspect of operations?

Would increased mindfulness reduce the risk of surprise? Alternative “sense-making” is but one component of the warning system, which includes collection, information sharing, and dissemination. Inadequacies in any of these areas will undermine even the most mindful analytic efforts. Moreover, the environment that intelligence seeks to comprehend is, arguably, considerably more complicated than that facing an aircraft carrier, with uncertainties of every type—coupled with active attempts to deceive—compounding the analytic challenge.

Increased mindfulness is not a panacea. Nonetheless, a look at some past surprises in “complex” situations suggests that even modest improvements from incorporating mindfulness could make a significant difference in preparedness.

• In the case of Pearl Harbor, as Cohen and Gooch have pointed out, small enhancements in military readiness, like deploying barrage balloons and torpedo nets in the harbor, could have resulted in a significant reduction of losses. [48] Would these steps have been more likely taken had the findings of the alternative analysis-type study of a surprise carrier attack noted earlier been integrated into an ongoing process of reexamination, instead of being consigned to a dusty file shelf?
  
• In the case of the runup to the Battle of France, examining the pattern of German reconnaissance flights would have highlighted a likely invasion route, perhaps resulting in redeployment of French forces to their weakest sector—the Ardennes. Had there been an ongoing process of reexamining assumptions in light of incoming information—some of which suggested the Ardennes possibility —might not that critical analysis have taken place? Interestingly, the then head of French military intelligence wrote years later that—evidently still not comprehending an important contributor to the intelligence debacle—”whoever works in intelligence…must avoid being led by intuition or excessive imagination”. [49]
  
• On September 11th, about half of the hijackers had been flagged for scrutiny at the gate before boarding the ill-fated flights. Had the concerns of the Phoenix FBI office about flight training not only been shared broadly within the government but also integrated into a mindfulness-focused inter-agency process—featuring out loud sense-making, Web-log type forums, computer-generated references to extant scenarios for crashing airplanes into prominent targets—might at least some of the detentions been prolonged, disrupting the plan?

Counterfactual questions cannot, of course, yield firm answers, but if they pique curiosity or create unease, it is a signal that further exploration of the issue at hand is warranted. This is the case for alternative analysis of transnational threats.

[1] The findings of the joint House-Senate investigation of September 11 outlines the basic story. It is Final Report, Part I, The Joint Inquiry, The Context, Part I, Findings and Conclusions, 10December 2002. A fuller account is contained in Senator Richard Shelby’s long supplementary document, September 11 and the Imperative of Reform in the Intelligence Community, Additional Views, 10 December 2002. Both are available at www.fas.org/irp/congress/2002_rpt/index.html (last visited 5 December 2003). See, in particular, Shelby’s report, p. 15ff. The National Commission on Terrorist Attacks Upon the United States also lays out the story, and it adds some fresh details. See its report, The 9/11 Commission Report, (Washington, 2004), available at http://www.9-11commission.gov/ (last visited 28 July 2004).

[2] Thomas Friedman, “A Failure to Imagine, “ New York Times, 14 May 2002.

[3] It is not as though planes as bombs had not been considered. The concern runs back at least as far as the late 1980s. See, for instance, Brian M. Jenkins, The Terrorist Threat to Commercial Aviation, P-7450, (Santa Monica: RAND, March 1989), p. 10: “The nightmare of governments is that suicidal terrorists will hijack a commercial airliner and, by killing or replacing its crew, crash into a city or some vital facility.” Rather, the concern receded from view as attention shifted to other kinds of threats.

[4] Frans R. Bax, “Intelligence Lessons From Pearl Harbor,” Studies in Intelligence, November 2002, pps. 1-9. The classic work on the intelligence failure at Pearl Harbor is Roberta Wohlstetter, Pearl Harbor: Warning and Decision, (Stanford: Stanford University Press, 1962).

[5] Eliot Cohen and John Gooch, Military Misfortune: The Anatomy of Failure in War, (New York: Vintage, 1991), pp. 50-51

[6] Elliot A. Cohen, “A Tale of Two Secretaries,” Foreign Affairs, May/June 2002, pp.33-46.

[7] Reports of the not-for-attribution workshops are published in a separate RAND report, CF-200. As might be expected from a series of gatherings of distinguished and diverse minds, discussions dealt with the issue at hand but also ranged well beyond to include general issues of improving the intelligence process. The individual workshop reports are thus well worth reading.

[8] Challenges posed to analysis by uncertainty are examined in greater depth later because there is a case to be made (or at least further explored) that transnational issues involve a significantly higher level of uncertainty than more traditional intelligence topics and thus requires a different approach to alternative analysis.

[9] Richards S. Heuer, Jr., The Psychology of Intelligence Analysis, (Washington: Center for the Study of Intelligence, 1999).

[10] Fred I. Greenstein and Michael Lerner, eds., A Source Book for the Study of Personality and Politics (Chicago: Markham, 1971).

[11] J. Edward Russo and Paul J.H. Schoemaker, Winning Decisions: Getting It Right the First Time, (New York: Doubleday, 2002), pp. 84-85.

[12] Found on Media Futures Archive, http://www.hfac.uh.edu/MediaFutures/home.html. (last visited 23 April 2004.

[13] An exploration of these factors and their implications for intelligence lies beyond the scope of this paper, although we will address a few later on in the context of discussing transnational alternative analysis

[14] For a general discussion of cognitive biases see Ziva Kunda, Social Cognition: Making Sense of People, (Cambridge: MIT Press, 2001), pp. 53-110.

[15] The classic work is Irving L. Janis, Victims of Groupthink: Psychological Study of Foreign-Policy Decisions and Fiascoes (Boston: Houghton Mifflin, 1972).

[16] Kees van der Heijden, The Sixth Sense: Accelerating Organizational Learning with Scenarios, (New York, John Wiley, 2002), pp. 50-51.

[17] The post mortem was chaired by former vice chairman of the Joint Chiefs of Staff, Adm. David Jeremiah. The report was never made public, but for reportage on it, see the Washington Post, June 3, 1998, p. A18 and the New York Times, same date. For the transcript of Jeremiah’s briefing on the report, see http://www.fas.org/irp/cia/product/jeremiah.html (last visited 16 December 2003).

[18] On the 1973 war, see, for instance, William B. Quandt, Peace Process: American Diplomacy and the Arab-Israeli Conflict Since 1967, (Washington, Brookings Institution Press, 1990).

[19] Peter Schwartz, The Art of the Long View, (New York, Doubleday, 1991), p. 220.

[20] Ernest R. May, Strange Victory: Hitler’s Conquest of France, (New York: Farrar, Straus and Giroux, 2000).

[21] We should note that in addition to the above, what are called “serious-play” techniques—military and political gaming, computer simulation—are sometimes included within the alternative analysis rubric. These techniques test assumptions interactively by having humans or intelligent agents play out situations implied by those assumptions. However, “serious play” stands apart from the alternative analysis mainstream in that it involves a much higher level of methodological sophistication and resource commitment than is usually the case with contrarian or contingent techniques. Although a few interesting ideas emerged related to gaming and simulation during the course of workshop discussions (and will be reported later in this paper), we did not explore this very specialized area at any length.

[22] For an overview of alternative analysis techniques, see Roger George, “Fixing the Problem of Analytical Mind-Sets: Alternative Analysis,” International Journal of Intelligence and Counter-Intelligence, Vol. 17, No. 3, Fall 2004, pp. 385-405.

[23] For a brief description of Shell’s scenario process, see P. Schoemaker and K. van der Heijden, “Integrating Scenarios into Planning at Royal Dutch/Shell,” Planning Review, 20, 3 (1992).

[24] Kris Frieswick, “The Difficulties of Thinking Ahead,” in CFO.com, February 1, 2002, http://www.cfo.com, last viewed 4 May 2004.

[25] The project focused on three transnational issues—terrorism, weapons proliferation, and organized crime—with experts in these fields offering opening presentations at the first workshop. In practice, terrorism tended to drive workshop discussions, being uppermost in the minds of most participants.

[26] For a readable discussion of networked phenomena see Albert-Laszlo Barabasi, Linked: How Everything is Connected to Everything Else and What It Means, (New York: Plume, 2003).

[27] David Snowden, “Complex Acts of Knowing: Paradox and Descriptive Self-Awareness,” Journal of Knowledge Management, Special Issue, September 2002, available at http://www.kwork.org/Resources/snowden.pdf, (last visited December 17, 2003). Snowden’s framework derives from complexity science, which examines inter-relationships in systems whose units are interconnected and where the entire system exhibits properties different from the sum of its parts. For a general discussion of the implications of complexity for international systems, see Robert Jervis, System Effects: Complexity in Political and Social Life, (Princeton: Princeton University Press, 1997).

[28] On the distinction between puzzles and mysteries, see Gregory F. Treverton, “Estimating Beyond the Cold War,” Defense Intelligence Journal, 3, 2 (Fall 1994); and Joseph S. Nye, Jr. , “Peering into the Future,” Foreign Affairs, 77, 4 July/August 1994, 82-93.

[29] The modern father of intelligenceanalysis is Sherman Kent. See his Strategic Intelligence for American World Policy, (Princeton, NJ: Princeton University Press, 1949). For a wonderful review of Kent’s book, one that particularly focused on two shortcomings in traditional foreign intelligence analysis, see Willmoore Kendall, “The Function of Intelligence,” World Politics, 1, 6 (July 1949), 542-52. Given the bright white line between intelligence and policy, intelligence usually analyzed foreign government in isolation (or ignorance) of what ours was doing, and, related, most intelligence was limited to “foreign” developments and so found it difficult (and perhaps illegal) to assess the impact of, say, American banks on Asia’s finances.

[30] This definition is a composite of several definitions of analysis discussed in Rob Johnston, “Developing a Taxonomy of Intelligence Analysis Variables,” Studies in Intelligence, Vol. 47, No. 3, 2003 found at http://www.cia.gov, last viewed May 4, 2004.

[31] Johnston is an anthropologist by training. See Rob Johnston, The Culture of Analytic Tradecraft: An Ethnography of the Intelligence Community (Washington: Center for the Study of Intelligence, Central Intelligence Agency), in press.

[32] Such ideas raised heated discussions at one of the GFP/RAND workshops. Some participants argued that analytic offices in the Intelligence Community were employing increasingly sophisticated methodologies to address complex questions, although others argued that this is not the same thing as regularly applying structured techniques to day-to-day issues.

[33] Gary Klein, Intuition at Work: Why Developing Your Gut Instinct Will Make You Better at What You Do, (New York: Doubleday, 2002).

[34] See his Sensemaking in Organizations, (London: Sage Publications, 1995).

[35] Thomas Stewart, “How To Think with Your Gut,” in Business 2.0, November 2002, http://www.business2.0.com. According to Stewart, the Marines moved to a more intuitive approach to decisionmaking training partly in response to this experience.

[36] Guy Claxton, Hare Brain, Tortoise Mind: How Intelligence Increases When You Think Less, (New York: Harper Collins, 2000), pp. 3-11.

[37] The current intelligence process, as traditionally organized, is less collective and more formal than the sense-making process described by Weick. It is largely up to individual analysts to make sense of changes on their “accounts.” Analysts do participate in conversations with colleagues and supervisors about their subjects, but also engage in formal, sometimes combative “coordination” of their written products with counterparts in other offices and agencies.

[38] See Managing the Unexpected: Assuring High Performance in and Age of Uncertainty, (San Francisco: Jossey-Bass, 2001).

[39] Klein, pp. 132-134.

[40] This and the following quotes from Weick and Sutcliffe are found on pages 25-62.

[41] See Kunda, pp. 161-210.

[42] Kunda, pp. 89-101.

[43] For a discussion of creativity in an organizational context see John Kao, Jamming: Art and Discipline of Corporate Creativity, (New York: Harper Business, 1997).

[44] Claxton, pp. 201-226. Also see Judith H. Heerwagen, “Creativity” at http://www.science.doe.gov (last visited July 13, 2004).

[45] Cited above, pp. 88-91.

[46] For a discussion of the Army’s after action process, see Nancy M. Dixon, Common Knowledge: How Companies Thrive By Showing What They Know, (Boston, Harvard Business School Press, 2000), pp.37-38.

[47] Former CIA Deputy Director for Intelligence Winston Wiley often said that the Directorate of Intelligence produces two products: intelligence analysis and intelligence analysts.

[48] Cohen and Gooch, pp.50-51

[49] May, p.365