Methods Used by Psychologists to Adequately Explain Human Behavior Analysis

psychology_as_the_scienc

CLICK HERE TO ORDER CUSTOM PAPERS

CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE

Research in the Psychological Laboratory: Truth or Triviality? Craig A. Anderson,1 James J. Lindsay, and Brad J. Bushman Department of Psychology, University of Missouri—Columbia, Columbia, Missouri (C.A.A., J.J.L.), and Department of Psychology, Iowa State University, Ames, Iowa (B.J.B.) Abstract This article examines the truism that studies from psychological laboratories are low in external validity. Past rational and empirical explorations of this truism found little support for it. A broader empirical approach was taken for the study reported here; correspondence between lab and field was compared across a broad range of domains, including aggression, helping, leadership style, social loafing, self-efficacy, depression, and memory, among others. Correspondence between lab- and field-based effect sizes of conceptually similar independent and dependent variables was considerable. Methods Used by Psychologists to Adequately Explain Human Behavior Analysis

In brief, the psychological laboratory has generally produced psychological truths, rather than trivialities. These same data suggest that a companion truism about field studies in psychology—that they are generally low on internal validity—is also false. Keywords external validity; metaanalysis; philosophy of science Are you happy with current rates of violent crime in the United States? How about the U.S. ranking in achievement test scores in science, compared with other industrialized nations? Do recent increases in smoking rates among U.S. teens bother you? Do the continuing problems of high rates of “unsafe sex” practices and the resulting incidence of AIDS seem to cry out for a solution? Constant attention to the problems of modern U.S. society by the mass media, politicians, and concerned citizens may seem to indicate a generally pessimistic worldview. Paradoxically, though, this focus on problems actually reflects a fundamentally optimistic view that as a society we can and should solve these problems. This same optimism drives modern psychology as well. The whole point of the science of psychology is to learn so that we can improve “things.”

THE DEBATE ABOUT EXTERNAL VALIDITY

This functionalist view gives rise to a long-running and frequently counterproductive debate about the value of theory-oriented laboratory research versus applicationoriented field research. This article addresses a very specific question from this debate: Does the psychological laboratory yield truths or trivialities? Since Campbell (1957) clearly distinguished between internal and external validity, a common truism has been that laboratory studies are typically high on internal validity but low on external validity.2 That is, laboratory studies are good at telling whether or not some manipulation of an independent variable causes changes in the dependent variable, but many scholars assume that these results do not generalize to the “real world.” Hence, application- Copyright © 1999 American Psychological Society 3 oriented scholars sometimes deride the psychological laboratory as a place where only trivial facts are to be found.3 In essence, the charge is that (some, most, or all) research from the psychological laboratory is externally invalid, and therefore pointless. One domain where this debate periodically arises concerns aggression (Anderson & Bushman, 1997). Consider obvious differences in surface characteristics between real-world versus laboratory aggression. Assault typically involves two or more people who know each other, arises from an escalating cycle of provocation, and results in serious physical injury.

Aggression in the lab, however, typically involves one person (who only thinks that he or she is interacting with an unknown person via computer, or notes, or message boards of one kind or another), in a session that may last only 50 min, and involves the attempted delivery of noxious stimuli such as electric shock or blasts of noise. But the charge that psychological laboratory research lacks external validity is not unique to the study of aggression. Recent years have seen similar debates in the study of personnel selection (e.g., Schmitt, 1996), leadership (e.g., Wolfe & Roberts, 1993), management (e.g., Griffin & Kacmar, 1991), and human memory (e.g., Banaji & Crowder, 1989; Neisser, 1978). For instance, in one early laboratory study of context-dependent memory, Dallett and Wilcox (1968) asked participants in one condition to study word lists while standing with their heads inside an oddly shaped box that contained flashing lights of several different colors; other participants studied the word lists without putting their heads inside the box. Participants later recalled the lists either with or without the box, and were most successful when the study and recall conditions were the same. As is obvious, these condi- 4 VOLUME 8, NUMBER 1, FEBRUARY 1999 tions have no counterpart in the “real” world, thus inviting complaints about external validity. It is easy to see why nonexperts frequently charge that lab studies are trivial, artificial, and pointless, and easy to ignore such complaints as reflections of ignorance. But when the charge comes from experts—other psychological researchers who presumedly share goals, training, and perspective—a thoughtful response is required. Such responses have also been forthcoming.

RESPONSES TO LABORATORY CRITICISMS

Embracing Invalidity One particularly elegant response, by Mook (1983), celebrates external invalidity. He described four cases in which the artificial lab setting is not only acceptable but actually preferred to the real-world setting: First, we may be asking whether something can happen, rather than whether it typically does happen. Second, our prediction may . . . specify something that ought to happen in the lab. . . . Third, we may demonstrate the power of a phenomenon by showing that it happens even under unnatural conditions that ought to preclude it. Finally, we may use the lab to produce conditions that have no counterpart in real life at all . . . . (p. 382) Mook’s main point is that the goal of most laboratory research is to discover theoretical relations among conceptual variables that are never sufficiently isolated in the real world to allow precise examination. What Is Supposed to Generalize? A second (and related) response is to note that usually researchers are interested in generalization of theoretical relations among concep- tual independent and dependent variables, not the specific instantiations of them. The same scandalous joke will mean something different in church than it does in the men’s locker room. In one case it may create embarrassment, whereas in the other it may create humor. If one were interested in the effects of humor on thought processes, one would be foolish to use the same joke as an experimental manipulation of “humor” in both settings. The lack of a manipulation’s generalizability constitutes an external validity problem only if one intends specific instantiations of conceptual variables to generalize across radically different contexts, but most laboratory research is concerned only with generalizability of the conceptual variables. Methods Used by Psychologists to Adequately Explain Human Behavior Analysis

The General Problem With Generalization

A third response begins with the observation that generalization is, generally, risky business. Inductive reasoning, at least in absolute terms, is never wholly justified. Even though every time you have dropped a hammer it has fallen, you cannot know for certain that it will fall the next time. Perhaps the laws of nature will change, or perhaps you will enter a location where your understanding of the laws of nature is revealed to be incomplete. Thus, generalizing from one situation to another, or from one participant population to another, is as problematic for field research as it is for lab research. So, the argument goes, why single out lab research for criticism? At least the lab makes it somewhat easier to satisfy concerns about internal validity, and without internal validity there is nothing to generalize anyway. But, people do generalize from specific instances to general concepts, then from these general concepts to new situations involving different instances of the general Published by Blackwell Publishers, Inc. concepts. A justification for such generalization can be readily found both at the level of species survival and at the level of scientific and technological advances: In both cases, generalization works much of the time.

Past Empirical Approaches

If what psychologists expect (hope?) to generalize are systematic relations among conceptual variables (i.e., theories), and if we grant that attempting to make generalizations is acceptable as long as it occasionally works, then another response to the external validity challenge becomes feasible. The challenge becomes an empirical question. Three different empirical approaches have been used: singlestudy tests, single-phenomenon tests, and single-domain tests. Single-study tests examine a specific laboratory finding in other contexts or with other populations. For example, Godden and Baddeley (1975) successfully generalized the context-dependent memory effect using scuba divers as subjects. Word lists that had been studied underwater were better recalled underwater, whereas lists studied on dry land were better recalled on dry land. Such singlestudy tests of external validity abound in psychology. Many “work,” though of course some do not. Though these tests answer the generalization question for a particular case, they do not adequately answer the broader question concerning the external validity of a given laboratory phenomenon.

Single-phenomenon tests examine the external validity of a whole empirical phenomenon rather than one specific laboratory finding. For example, do laboratory-based effects of anonymity on aggression generalize to field settings? Questions like this can be investigated by using meta-analytic techniques. That is, one could statisti- CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE cally average all of the research results from lab and field studies that have tested the relation between anonymity and aggression. We performed such a meta-analysis (Anderson & Bushman, 1997) and found comparable anonymity effects in lab and field settings. Similar tests of the generalizability of a specific laboratory phenomenon can be found in numerous additional areas of psychology. Many of these single-phenomenon tests show comparable effects for the psychological laboratory and field studies. But failures to generalize also occur. Single-domain tests further broaden the generalizability question to a whole research domain.

For example, do most aggression findings from the psychological laboratory generalize to field studies? In other words, do the effects of key independent variables—such as alcohol, anonymity, and media violence—have the same effects in lab and field studies of aggression? If laboratory studies from a given domain are inherently invalid and those from field studies are valid, then lab and field studies in that domain should fail to show any correspondence. We (Anderson & Bushman, 1997; Bushman & Anderson, 1998) used meta-analytic techniques to ask this broad question in the aggression domain, and found considerable correspondence between lab and field.

CLICK HERE TO ORDER CUSTOM PAPERS

A CROSS-DOMAIN EMPIRICAL APPROACH

We extend the single-domain approach by examining the comparability of findings from lab and field across several domains. Basically, we asked whether the effects of the same conceptual independent variables on the same conceptual dependent variables tended to be consistent in lab and field settings across several psychological domains. Method Using the PsycINFO database, we conducted a literature search for the following journals: Psychological Bulletin, Journal of Applied Psychology, Journal of Personality and Social Psychology, and Personality and Social Psychology Bulletin. We searched with the keyword phrases “meta-analysis” and “quantitative review,” and with the combined keyword phrases “meta-analysis” with “field studies” and “metaanalysis” with “laboratory.” This selection of journals was intentionally biased toward social psychology because many of the most vociferous criticisms of the psychological laboratory have focused on the social psychological lab. Our search yielded 288 articles. Many articles were subsequently eliminated because they were methodological in nature, did not include separate tabulations for lab and field settings, or overlapped with a more recent meta-analytic review.4 The final data set represents 38 pairs of lab and field effects. Methods Used by Psychologists to Adequately Explain Human Behavior Analysis

Results and Discussion We used the standardized mean difference, denoted by d, as the indicator of effect size. This index shows the size of the difference between two groups, and does so in terms of the standard deviation. For instance, if Group 1 has a mean of 6 and Group 2 has a mean of 5, and the standard deviation is 2, the effect size d would be (6 – 5)/2 = 0.5. According to Cohen (1988), a “large” d is 0.8, a “medium” d is 0.5, and a “small” d is 0.2. Effect sizes for correlations can easily be converted to ds, which we did to allow direct comparisons across different types of studies (Hedges & Olkin, 1985). Effect-size averages were weighted by sample size and used Copyright © 1999 American Psychological Society 5 pooled standard deviations in most cases, but in a few cases we were unable to determine the weights from the original reports. Table 1 and Figure 1 summarize the results.5 Figure 1 plots the value of d for the lab and field studies for each domain studied. The figure reveals considerable consistency between laboratory and field effects. That is, across domains, the ds for lab and field studies tended to be similar.

The correlation, r = .73, is considerably higher than the gloomy picture that sometimes emerges from the external validity debate. Some readers might wonder whether the disproportionate number of the data points coming from comparisons of gender effects (6 out of 38) biased the results. However, the plot and correlation look much the same with these 6 data points eliminated (r = .75). Similarly, one might wonder about the possible nonindependence of some of the attributional-style data points (the results from Sweeney, Anderson, & Bailey, 1986). Dropping all but the two overall “attribution and depression” effects for positive and negative outcomes again yields essentially the same correlation (r = .73). All three of these correlations are considerably larger than Cohen’s (1988) conventional value for a large correlation (r = .5). Furthermore, an r of .73 is equivalent to a d of 2.14, a huge effect. Two complementary questions arise from these results, one asking why the correspondence is so high, the other asking why it is not higher.

First, consider some limitations on correspondence between lab and field results. One limitation concerns internal validity problems of field studies. Sometimes field studies “discover” relations between independent and dependent variables that are false, and at other times they fail to discover true relations. Both types of internal invalidity reduce correspondence 6 VOLUME 8, NUMBER 1, FEBRUARY 1999 Table 1.

Mean effect sizes and confidence intervals for topics studied in the lab and field Source, independent and dependent variables, and setting Ambady and Rosenthal (1992) Observation time and outcome ratings—lab Observation time and outcome ratings—field Anderson and Bushman (1997) Gender and physical aggression—lab Gender and physical aggression—field Gender and verbal aggression—lab Gender and verbal aggression—field Bushman and Anderson (1998) Anonymity and aggression—lab Anonymity and aggression—field Trait aggressiveness and aggression—lab Trait aggressiveness and aggression—field Type A personality and aggression—lab Type A personality and aggression—field Carlson, Marcus-Newhall, and Miller (1990) Weapons and aggression—lab Weapons and aggression—field Eagly and Crowley (1986) Gender and helping—lab Gender and helping—field (on and off campus) Eagly and Johnson (1990) Gender and leadership style—lab Gender and leadership style—organizations Eagly and Karau (1991) Gender and leader emergence—lab Gender and leader emergence—natural settings Eagly, Karau, and Makhijani (1995) Gender and leader effectiveness—lab Gender and leader effectiveness—organizations Gordon (1996) Ingratiation and evaluations—university (lab) Ingratiation and evaluations—field Karau and Williams (1993) Social loafing—lab Social loafing—field Kraiger and Ford (1985) Race of ratee and performance ratings—lab Race of ratee and performance ratings—field Kubeck, Delp, Haslett, and McDaniel (1996) Age (continuous) and job-training mastery—lab Age (continuous) and job-training mastery—field Age and time to finish training—lab Age and time to finish training—field Age (dichotomous) and job-training mastery—lab Age (dichotomous) and job-training mastery—field Lundeberg and Fox (1991) Expectancies and recall–essay tests—lab Expectancies and recall–essay tests—class Expectancies and recognition tests—lab Expectancies and recognition tests—class Mento, Steel, and Karren (1987) Goal difficulty and performance—lab Goal difficulty and performance—field Mullen and Hu (1989) Group membership and similarity of group members— artificially created groups Group membership and similarity of group members— real groups Number of samples Effect size 95% confidence interval

21 17 0.87 0.98 — — 37 6 18 3 0.31 0.40 0.13 0.03 0.23–0.38 0.25–0.55 0.03–0.24 –0.15–0.22 18 4 13 16 9 3 0.57 0.44 0.49 0.93 0.34 0.97 0.45–0.69 0.25–0.63 0.18–0.29 0.38–0.47 0.18–0.49 0.71–1.23 16 5 0.21 0.17 0.01–0.41 –0.05–0.39 16 36 + 47 = 83 –0.18 0.27 –0.28–0.09 — 17 269 0.22 –0.00 — — 50 24 0.45 0.10 0.40–0.51 0.02–0.17 20 56 0.07 –0.03 –0.06–0.20 –0.06–0.01 54 15 0.38 –0.07 0.33–0.43 –0.13– –0.00 140 23 0.47 0.25 0.43–0.51 0.16–0.35 10 64 0.07 0.39 –0.41–0.56 0.06–0.75 17 31 3 2 9 2 –0.61 –0.52 0.70 1.35 –0.96 –0.38 –1.67–0.12 –1.19–0.02 0.08–1.58 1.35–1.35 –1.44– –0.47 –0.38– –0.38 41 11 41 14 0.60 0.33 –0.07 0.28 0.53–0.67 0.17–0.49 –0.13– –0.01 0.14–0.42 47 23 0.62 0.44 — — 2 0.43 0.04–0.90 2 0.47 –0.14–1.16 Methods Used by Psychologists to Adequately Explain Human Behavior Analysis

Published by Blackwell Publishers, Inc. 7 CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE Table 1. continued Source, independent and dependent variables, and setting Number of samples Narby, Cutler, and Moran (1993) Authoritarianism and trial verdict—video, written, audio trials Authoritarianism and trial verdict—live trials Paik and Comstock (1994) Media violence and aggression—lab Media violence and aggression—field Peters, Hartke, and Pohlman (1985) Leadership style and performance, negative octants—lab Leadership style and performance, negative octants—field Leadership style and performance, positive octants—lab Leadership style and performance, positive octants—field Sagie (1994) Decision-making participation and productivity—lab Decision-making participation and productivity—field Sweeney, Anderson, and Bailey (1986) Attribution and depression, negative outcomes—lab Attribution and depression, negative outcomes—hospital Attribution and depression, positive outcomes—lab Attribution and depression, positive outcomes—hospital Ability and depression, negative outcomes—lab Ability and depression, negative outcomes—hospital Ability and depression, positive outcomes—lab Ability and depression, positive outcomes—hospital Effort and depression, negative outcomes—lab Effort and … Methods Used by Psychologists to Adequately Explain Human Behavior Analysis