Sampling and Data Collection in Research Paper

CHAPTER5

A crisis counselor working with a mental health agency receives a call from the county jail. The deputy there is concerned about an inmate he describes as severely depressed. The counselor responds by asking a number of questions, attempting to make an initial assessment of the severity of the inmate’s depression. Has the inmate been eating his meals? Is he sleeping too much or too little? Is his affect flat when he responds to questions? Has he made any remarks about committing suicide? Later, the counselor may interview the inmate directly, request psychological testing, or refer him to a psychiatrist for further evaluation. Such assessments are analogous to a process in research called measurement. Just as the clinician used a variety of observations by the deputy as indicators of the inmate’s condition, researchers use various observations as indicators of the concepts of interest in a research project. Measurement refers to the process of describing abstract concepts in terms of specific indicators by assigning numbers or other symbols to these indicants in accordance with rules. At the very minimum, one must have some means of determining whether a variable is either present or absent, just as the counselor needs to know whether the inmate is eating or not. In many cases, however, measurement is more complex and involves assessing how much, or to what degree, a variable is present. An example of this is the counselor’s question about how much the inmate is sleeping, “amount of sleep” being a variable that can take on many values. Sampling and Data Collection in Research Paper

Measurement is a part of the process of moving from the abstract or theoretical level to the concrete. Recall from Chapter 2 that scientific concepts have two types of definitions: nominal and operational. Before research can proceed, researchers must translate nominal definitions into operational ones. The operational definitions indicate the exact procedures, or operations, that the researchers will use to measure the concepts. Measurement is essentially the process of operationalizing concepts. Figure 5.1 illustrates the place of measurement in the research process.

Figure 5.1 The Measurement Process

 

In this chapter, we discuss the general issues that relate to all measurement, beginning with some of the different ways in which we can make measurements. We then analyze how measurements that are made at different levels affect the mathematical operations that can be performed on them. Finally, we present ways of evaluating measures and determining the errors that can occur in the measurement process.

Ways of Measuring

From Concepts to Indicators

Normally, we cannot observe directly the concepts and variables that are the focus of both research and practice. We cannot see such things as poverty, social class, mental retardation, and the like; we can only infer them from something else. Take something as seemingly obvious as child abuse. Can you directly observe child abuse? Not really. What you directly observe is a bruise on a child’s back, an infant’s broken leg, or a father slapping his daughter. And even the slap may not relate to child abuse, because parents sometimes slap their children without its being a case of child abuse. However, all these things—the bruise, the broken leg, the slap—may be used as indicators of child abuse. In research and in practice, an indicator is an observation that we assume is evidence of the attributes or properties of some phenomenon. What we observe are the indicators of a variable, not the actual properties of the variable itself. Emergency room personnel may assume that a child’s broken leg is an indicator of child abuse even though they have not observed the actual abuse. Sampling and Data Collection in Research Paper

Child abuse represents a good illustration of the difficulties of moving from nominal to operational definitions with variables involving social and psychological events. At the nominal level, we might define child abuse as an occurrence in which a parent or caretaker injures a child not by accident but in anger or with deliberate intent (Gelles 1987; Korbin 1987). What indicators, however, would we use to operationalize this definition? Some things would obviously seem to indicate child abuse, such as a cigarette burn on a child’s buttock, but what about a bruise on the arm? Some subcultures in our own society view hitting children, even to the point of bruising, as an appropriate way to train or discipline them. Furthermore, some people would argue that a serious psychological disorder a child suffers is an indicator of child abuse, because it shows the parents did not provide the proper love and affection for stable development. In short, one of the problems in operationalizing child abuse, as with many other variables in human service research, is that its definition is culture-bound and involves subjective judgments. This illustrates the importance of good conceptual development and precise nominal definitions for research. It also shows how the theoretical and research levels can mutually influence one another: As we shape nominal definitions into operational ones, the difficulties that arise often lead to a reconceptualization, or a change, in the nominal definition at the theoretical level (see Figure 5.1).

The example of child abuse also illustrates another point about measurement—namely, that more than one indicator of a variable may exist. The term item is used to refer to a single indicator of a variable. Items can take numerous forms, such as an answer to a question or an observation of a behavior or characteristic. Asking a person her age or noting her sex, for example, would both produce items of measurement. In many cases, however, the process of operationalizing variables involves combining a number of items into a composite score called an index or a scale. (Although scales involve more rigor in their construction than indexes do, we can use the terms interchangeably at this point; Chapter 13 will present some distinctions between them.) Attitude scales, for example, commonly involve asking people a series of questions, or items, and then summarizing their responses into a single score that represents their attitude on an issue. A major reason for using scales or indexes rather than single items is that scales enable us to measure variables in a more precise and, usually, more accurate fashion. To illustrate the value of scales over items, consider your grade in this course. In all likelihood, your final grade will be an index, or a composite score, of your answers to many questions on many tests throughout the semester. Would you prefer that your final grade be determined by a one-item measure, such as a single multiple-choice or essay question? Probably not, because that item would not measure the full range of what you learned. Furthermore, an error on that item would indicate that you had not learned much in the course, even if the error were the result of ill health or personal problems on the day of the exam. For these reasons, then, researchers usually prefer multiple-item measures to single-item indicators.

We began this discussion by noting that because variables are abstract, we normally cannot observe them directly. Variables differ in their degree of abstraction, however, and this affects the ease with which we can accomplish measurement. In general, the more abstract the variable, the more difficult it is to measure. For example, a study of child abuse might include the variable “number of children in family,” on the theoretical presumption that large families create more stress for parents and, therefore, are more likely to precipitate abusive attacks on children. This is a rather easy variable to measure, because the concepts of “children” and “family” have readily identifiable, empirical referents and are relatively easy and unambiguous to observe and count. Suppose, however, that the child abuse study also included as a dependent variable “positiveness of child’s self-concept.” Because it can take many different forms, “self-concept” is a difficult notion to measure. Although we have narrowed it to the “positive—negative” dimension, it is still more difficult to measure than “number of children in family,” because we could ask a whole variety of questions to explore how positively people feel about themselves. We also can measure self-concept by behaviors, on the theoretical presumption that people who feel positively about themselves behave differently from those who do not. The point is that highly abstract concepts usually have no single empirical indicator that is clearly and obviously preferable to others as a measure of the concept.

We have emphasized the point that measurement involves transition from the abstract and conceptual level to the concrete and observable level, and this is what most typically occurs in research. Exploratory studies, however, can involve measurement in the opposite direction: First, we observe empirical indicators and then formulate theoretical concepts that those indicators presumably represent. In Chapter 2, we called this inductive reasoning. In a sense, you might think of Sigmund Freud or Jean Piaget as having done this when they developed their theories of personality and cognitive development, respectively. Piaget, for example, observed the behavior of children for many years as he gradually developed his theory about the stages of cognitive development, including concepts like egocentrism, object permanence, and reversibility (Ginsburg and Opper 1988). Piaget recognized that what he observed could be understood only if placed in a more abstract, theoretical context. In a sense, he measured something before he knew what it was he had measured. Once his theories began to develop, he then developed new concepts and hypotheses, and he formulated different measuring devices to test them deductively. The point is that whether one shifts from the abstract to the concrete, or vice versa, the logic is the same, involving the relationship between theoretical concepts and empirical indicators.

Techniques of Measuring

We will discuss specific techniques for measuring variables in other chapters in this book, but we find that discussing these techniques briefly at this point helps make clear the issues surrounding measurement. Measurement techniques in the social sciences and human services vary widely, because the concepts we measure are so diverse. These techniques, however, mostly fall into one of three categories (see Figure 5.2).

Figure 5.2 The Major Strategies Used by Social Scientists to Measure Variables

 

1. Verbal reports. This is undoubtedly the most common measurement technique in social research. It involves people answering questions, being interviewed, or responding to verbal statements (see Chapters 7 and 9). For example, research on people’s attitudes typically uses this technique by asking people how they feel about commercial products, political candidates, or social policies. In a study of school performance, to mention another example, we could measure how well students do in school by asking them what their grades are or how much they know about a particular subject. Sampling and Data Collection in Research Paper

2. Observation. Social researchers also measure concepts by making direct observations of some phenomena (see Chapter 9). We watch people at school or at work and make notes of what they say and do. We may even make an audio or video recording as a way of preserving the observations. In a study of school performance, we could measure how well students do in school by directly observing their behavior in the classroom and noting how often they answer questions posed by teachers, how often their answers are correct, and how they get along with teachers and students.

3. Archival records. Researchers also use a variety of available recorded information to measure variables (see Chapter 8). These records might take the form of statistical records, governmental or organizational documents, personal letters and diaries, newspapers and magazines, or movies and musical lyrics. All these archival records are the products of human social behavior and can serve as indicators of concepts in the social sciences. In the study of school performance, for example, a researcher could use school records to locate students’ grades, performance on exams, attendance records, and disciplinary problems as measures of how well they are doing in school.

These are the major ways that social scientists measure concepts. Researchers must specify exactly what aspects of verbal reports, observations, or available data will serve as indicators of the concepts they want to measure. In addition, researchers use some key criteria to help them decide whether a particular indicator is a good measure of some concept. These criteria will be discussed later in this chapter.

Positivist and Nonpositivist Views of Measurement

Much of the foundation for measurement and operationalization in the social sciences derives from the work of statisticians, mathematicians, philosophers, and scientists in a field called classical test theory or measurement theory (Bohrnstedt 1983; Stevens 1951), which provides the logical foundation for issues discussed in this chapter and derives largely from the positivist view of science discussed in Chapter 2. The logic of measurement can be described by the following formula:

X = T + E

In this formula, X represents our observation or measurement of some phenomenon; it is our indicator. It might be the grade on an exam in a social research class, for example, or a response to a self-esteem scale (see Table 5.1). Also in this formula, T represents the true, actual phenomenon that we are attempting to measure with X; it would be what a student actually learned in a social research class or what his or her true self-esteem is. The third symbol in the formula, E, represents any measurement error that occurs, or anything that influences X other than T. It might be the heat and humidity in the classroom on the day of the social research exam, which made it difficult to concentrate, or it could reflect the fact that a subject incorrectly marked a choice on the self-esteem scale, inadvertently circling a response that indicated higher or lower self-esteem than he or she actually possessed. Sampling and Data Collection in Research Paper

Table 5.1 Elements in the Process of Measurement

X	=	T	+	E
Observation	=	True Phenomenon	+	Error
Reading on a weight scale	=	Your actual weight	+	Clothing you are wearing; Heavy object in your pocket
Grade on an examination in social research class	=	Actual knowledge you acquired in social research class	+	Heat and humidity in test room; Distraction due to fight with partner
Score on a scale measuring self-esteem	=	Your actual level of self-esteem	+	Incorrectly marking a self-esteem scale; Questions on self-esteem scale that are difficult to understand

The formula is very simple—but also very profound and important: Our measurement of any phenomenon is a product of the characteristics or qualities of the phenomenon itself and any errors that occur in the measurement process. What we strive for is measurement with no error:

E = 0

and, therefore,

X = T

The ideal to strive for is a measurement of the phenomenon that is determined only by the true state of the phenomenon itself. Scientists recognize, however, that they normally cannot achieve this ideal state in its entirety. In reality, we attempt to reduce E as much as possible. Later in this chapter, we will complicate this measurement formula a bit, but for now, it can stand as a shorthand way of understanding the process of measurement. Sampling and Data Collection in Research Paper

Before going deeper into the process of measurement, it is important to consider the nonpositivists’ critique of classical measurement theory. Many nonpositivists argue that we haven’t examined a huge assumption in this at all, one that may render the entire topic somewhat problematic. The assumption is that the phenomenon being measured (T) exists objectively in the world and that our measurement device is merely discovering it and its properties. Some things do exist in the world independently of our perceptions and our judgments about them. The computer monitor on which these words are being written, for example, has a screen that is nine inches tall—we just measured it with a ruler. Our measurement of it was a discovery of its properties, and the measurement process did not create or change those properties. Now, however, think about a social science concept, such as self-esteem. We measure it by asking subjects to agree or disagree with a series of statements. We score a “strongly agree” response as “4” and a “strongly disagree” response as “1”; then, we sum up those responses to all the separate items in the scale and give a self-esteem score that ranges from 10 to 40. What, however, is the objective reality behind this measurement? If a subject receives a score of 32 on our measurement device, what does that 32 correspond to in his or her subjective world, or mind, or consciousness? The 32 is the X in our measurement formula, but what is the T that it corresponds to? Is the link between the measurement of a computer screen and its actual length as direct as the link between the score of 32 on the self-esteem measure and the actual subjective experience of self?

The nonpositivists argue that many social science concepts do not have such clear and objective referents in the world. Our concepts are based on an intuitive and theoretical understanding of what parts of the world are like. In other words, we are constructing the world, not just discovering it. We believe that something like self-esteem exists, but it is our construction of it that we measure with the self-esteem scale, not the thing itself (if the thing itself even exists). This does not make measurement theory useless, but it does suggest that the whole process is more complicated—and not nearly as objective—as the positivists suggest. Nonetheless, many nonpositivists agree that some social science measurement can follow the model of measurement theory. Some social phenomena, such as age and sex, do have some objective existence in the world. A person’s age has something to do with how many times the earth has circled the sun since his or her birth, and sex has something to do with a person’s physical genitalia. The social significance of these characteristics is another matter, of course, but in many cases, the measurements of age and sex can follow classical measurement theory. Research in Practice 5.1 addresses some of these measurement issues in regard to the significant social issue of domestic violence.

A major problem in most measurement has to do with which indicators to use in a particular research project. This depends in part, of course, on theoretical concerns, but there are other matters to consider as well. One such matter has to do with whether a particular measure permits one to perform mathematical operations on it; we turn to this issue next. Sampling and Data Collection in Research Paper