Somehow, I started to do naturalistic studies of decision-making without appreciating what I was getting myself into.
Naturalistic research means exploring the phenomenon you want to understand in the setting where it happens. If you want to know how people make tough decisions, you find a way to watch them or interview them, or review subsequent accounts of what they did. You concentrate your energies on the phenomenon, instead of passively adhering to the methodology. As I write this, I realize that it seems pretty obvious.
But it’s not obvious to all the experimenters who work in controlled laboratory settings, presenting artificial tasks to naïve college students. And that was my background. Coming out of graduate school, that’s the way I was trained.
Then, in 1985, I got a contract from the Army Research Institute to study decision-making. The Army wanted to know how people make challenging decisions under time pressure and uncertainty. I expect that most of the proposals that were submitted were to do laboratory-based experiments, varying time pressure and uncertainty. However, my colleagues and I were different. We really wanted to know how this might be possible. We knew that the ideal way to make decisions was to lay out the options and score each on a standard set of evaluation dimensions. We could not imagine how soldiers, in the heat of battle, could go through such an exercise. What were they doing instead? We had some ideas, but we were really curious, so our proposal was to study people who actually do make life-and-death decisions under time pressure and uncertainty. We chose to study firefighters. And we won the contract.
Immediately, I had to face the reality that none of my training had prepared me to observe or interview skilled decision-makers. So I added a communications specialist to our team, Anne Clinton-Cirocco. She was not a psychologist, but she had a gift for connecting with people and encouraging them to tell their stories.
And they told us wonderful stories, blowing up our preconceptions and letting us formulate the Recognition-Primed Decision (RPD) model (Klein, Calderwood, and Clinton-Cirocco, 1986; Klein, 1998), which explains how people actually make decisions in their work and in their lives. The RPD model has been replicated a number of times and is still going strong after almost 40 years, but that’s another story.
I was excited about the RPD model and submitted a paper to one of the most prestigious organizations around, the Judgment and Decision Making Society, for their next conference, but it got rejected. Very disappointing. As the conference drew closer, I looked at the program and noticed that there was a session on “Naturalistic Decision Making.” That’s the session my paper should have been in. I attended the conference in New Orleans (I think it was 1988 or 1989) to sit in on this session.
The conference room was packed. There were three papers in the session. One was by an economist who reviewed some data sets to infer decision strategies. That made sense. A second paper was about medical decision-making; the researcher had conducted interviews with seven physicians. Yes, this seemed directly relevant to me, until the researcher described the task—“Disease 1 has symptoms A, B, and C. Disease 2 has symptoms B, C, and D. The probability of a patient having Disease 1 …” and so forth. No specific diseases or symptoms were mentioned. After describing his paradigm, the researcher told the audience that each of the seven physicians had the same reaction. They said that this was not medicine. It was statistics. And each walked out of the cubicle. With that, the audience in the conference room burst into laughter, and so did I. Yet I realized that the audience was laughing at the physicians for being so dense, whereas I was laughing at the researcher for being so dense.
The third paper, by one of the prominent figures in the field, was about firefighting, even more relevant to me. It was about wildland firefighting. This researcher, however, did not study actual firefighters. He studied college students. His task centered on a map grid (not even a topological map) with a “fire” that spread from one cell to the next with a certain probability. Two of my colleagues had just come back from Idaho, where they did observations and interviews during an actual forest fire, and this academic exercise had no relationship to the actual phenomenon.
If this was what the Judgment and Decision Making Society considered Naturalistic Decision Making, I saw little reason to attend any more of their meetings. The experience did, however, sharpen my appreciation for naturalistic research and its importance.
Another experience further advanced my views on naturalistic research. I heard of a discussion between a naturalistic researcher, my late wife Helen Klein, and a colleague, an organizational psychologist who was in the middle of a research project. Penny (not her real name) was investigating how people react to personnel decisions. During this casual conversation, Penny mentioned to Helen that almost half of the people she interviewed had cried. Helen asked, “Why did they cry?” Penny answered that she didn’t know. The study was about halfway done. Helen suggested that Penny probe any more participants who might cry to learn what was distressing them. “I can’t do that,” Penny replied. “It’s not in the protocol.” Helen politely suggested that Penny just ask after completing all her interview questions. But Penny wouldn’t be persuaded. Asking that question wasn’t part of the original protocol, plus she hadn’t asked the earlier participants.
Simple curiosity wasn’t enough for Penny. Rigid adherence to protocol came first. That illustrates the difference between a naturalistic approach seeking to learn and discover and a methodological straitjacket that rules out curiosity.
I adopted a naturalistic approach because it was the best way to study decision-making in context, and incidents like these have only deepened my appreciation of naturalistic research.
