III. STUDY DESIGN
A. General References in Research
Design and Data Analysis
- Ager, J.W. Discussion: Statistical analysis in treatment
and prevention program evaluation. In: Kilbey, M.M., & Asghar,
K., eds. Methodological Issues in Epidemiological, Prevention,
and Treatment Research on Drug-Exposed Women and Their Children. National
Institute on Drug Abuse Research Monograph 117. DHHS Pub. No.
(ADM)92-1881. Washington, DC: Supt. of Docs., U.S. Govt. Print. Off.,
1992. pp. 31-40.
The chapter focuses on problems of statistical analysis
in the context of evaluations of substance abuse treatment and prevention
programs. The statistical and design issues discussed includes the
following: (1) types of designs and their associated statistical analyses;
(2) covariance and other adjustment techniques in analyses of quasi-experimental
design data; (3) modeling change; and (4) meta-analysis.
- Anderson, V.L., & McLean, R.A. Design of Experiments:
A Realistic Approach. New York: Marcel Dekker, Inc., 1974.
This book on design of experiments is arranged so that
the reader may go from the simple to the complex designs and grasp
the appropriate analyses from the resulting data. Chapters 1 to 3
cover basic concepts in experimental designs. Chapters 4 through 7
cover various experimental designs and illustrate the usage of restriction
error concept. Chapter 8 expresses a view on Latin square type designs,
and chapters 9 and 10 provide descriptions on the completely randomized
factorials. Chapters 11 and 12 deal with three-level factorial experiments,
mixed factorial experiments, and incomplete block designs. The last
chapter of the book describes response surface exploration.
- Bryant, K.J., Windle, M., & West, S.G., eds. The
Science of Prevention: Methodological Advances from Alcohol and Substance
Abuse Research. Washington, DC: American Psychological Association,
1997.
The volume describes latest developments of methodological
methods in the field of substance abuse research. Most of the contributors
to the book are active researchers in the field of substance abuse
prevention who are also methodological experts. It aims at promoting
critical thinking among new and established investigators about how
to design research and analyze research findings. Although the substantive
focus of many chapters is on applications to the prevention of alcohol
and substance abuse, nearly all of the methodological principles and
statistical models are general and have potential application to the
full range of areas in which prevention research takes place.
- Collins, L.M., & Millsap, R. Innovative methods
for prevention research. Special issue of Multivariate Behavioral
Research. Mahwah, NJ: Erlbaum, 1998.
The special issue includes six examples of innovative
methodlogical procedure useful in prevention research. Hedeker and
Mermelstein present a model for multilevel logistic regression with
ordinal outcomes. Reboussin et al. discuss a method for including
continuous predictors in latent transition models. Boker and Graham
provides a conceptual introduction to dynamical systems analysis.
Sayer and Willet describe cross-domain analysis in latent growth curve
analysis. Schafer and Olsen present the technique of multiple imputation
and its use to solve missing data problems. Bacik et al. discuss a
type of missing data that is common in prevention research: the participant
who is absent for one data collection occasion and then returns to
the study. They explore the researcher's options in performing survival
analysis on this kind of data.
- Coursey, R.D., ed. Program Evaluation for Mental
Health: Methods, Strategies, Participants. New York: Grune &
Stratton, Inc., 1977.
The book focuses on setting up and running evaluation
programs for real-life mental health delivery systems. It summarizes
literature and provides both broad conceptualizations about the structure
of evaluation activity as well as practical knowledge. The book emphasizes
the human dimensions of evaluation, such as staff's need for economic
survival and esteem. It also emphasize the role of program evaluation
in staff education and development. The author also take the perspective
that program evaluation is one component of the overall process of
program development and program planning. The book is divided into
four sections: (1) methods and techniques of evaluation, (2) strategies
for implementing those methods, (3) the participants in program evaluation,
and (4) resources available to mental health workers.
- Daniel, W.W. Biostatistics: A Foundation for Analysis
in the Health Sciences, 7th ed. NY: John Wiley & Sons, 1999.
The text is written for health professionals in need
of a reference book on statistical methodology. It covers basic concepts
in statistics such as probability, sampling distribution, estimation
and hypothesis testing. It describes common statistical techniques
such as analysis of variance, simple and multiple regression, logistic
regression, chi-square test and nonparametric and distribution-free
statistics.
- Dowdy, S., & Wearden, S. Statistics for Research.
New York: Wiley, 1983.
This is an introductory textbook to statistics and it
is intended for students who have no prior background in statistical
methods. The author try to provide both an understanding of the concepts
of statistical inference as well as the methodology for the most commonly
used analytical procedures. The book covers basic concepts in statistics,
such as probability distributions, sampling distribution of averages
and paired variables. It then describes commonly used statistical
techniques, such as ANOVA, ANCOVA, and multiple regression.
- Flay, B.R., & Petraitis, J. Methodological issues
in drug use prevention research: Theoretical foundations. In: Leukefeld,
C.G., & Bukoski, W.J., eds. Drug Abuse Prevention Intervention
Research: Methodological Issues. NIDA Research Monograph 107.
DHHS Pub. No. (ADM) 91-1761. Washington, DC: Supt. of Docs., U.S.
Govt. Print. Off., 1991. pp. 81-109.
The chapter discusses the theoretical foundation of
drug use prevention program development and research. The authors
reviewed theories of drug use onset and behavior change and then focus
on the functions and roles of theory and their methodological applications.
Issues like implementation quality, external validity, construct validity,
and special method—theory relationships are discussed in later
parts of the chapter.
- Jason, L., Thompson, D., & Rose, T. Methodological
issues in prevention. In: Edelstein, B., & Michelson, L., eds.
Handbook of Prevention. New York: Plenum Press, 1986, pp. 1-19.
The chapter reviews methodological issues that need
to be considered in designing and implementing preventive-oriented
interventions. The authors first review theoretical notions underlying
preventive programs. Then they discuss specific methodological issues
like goal selection, assessment and screening, experimental and quasi-experimental
designs, generalization and maintenance, cost-benefit analyses, meta-analyses,
social validation, and monitoring the integrity of preventive interventions.
The final section summarizes some of the obstacles to designing well-planned
preventive interventions, and speculates on the types of new directions.
- Judd, C.G., & Kenny, D.A. Estimating the Effects
of Social Intervention. Cambridge: Cambridge University Press,
1981.
The authors present a discussion of the various research
designs used to evaluate social interventions. Designs described in
the book include randomized experiments, regression discontinuity
design, nonequivalent control group design, interrupted time-series
design, post-only correlational design. Judd and Kenny describe each
design and the usual statistical analysis procedures to analyze the
design. They then discuss possible complications in the analysis and
suggest practical solutions to the complications.
- Maxwell, S.H., & Delaney, H.D. Designing Experiments
and Analyzing Data: A Model Comparison Perspective. Belmont, CA:
Wadsworth, 1990.
The book is written to serve as either a textbook or
a reference book on the topic of designing experiments and analyzing
experimental data. The authors proposed a model comparison approach,
which allows the use of a few basic formulas that can be applied to
every experimental design. Such an approach also allows for further
extension to more complex data-analytic methodologies such as structural
equation modeling. Part I of the book explains the logic of experimental
design and the role of randomization in behavioral research. Part
II deals with model comparison for between-subject designs, starting
with a discussion of the general linear model. Part III discusses
model comparisons for designs involving within-subject factors. Part
IV covers alternative analysis strategies such as robust ANOVA and
ANCOVA, repeated measures designs.
- Montgomery, D.C. Design and Analysis of Experiments,
3rd ed. New York: John Wiley & Sons, 1991.
This is an introductory textbook dealing with the design
and analysis of experiments. The book is intended for readers who
have some background in elementary statistics and some familiarity
with matrix algebra is required in portions of chapters 15 and 16.
The third edition is a major revision of the book. While maintaining
a balance between design and analysis topics, new materials and examples
are added. The reorganization of the material on factorial and fractional
factorial designs in chapters 9, 10, and 11 provide more in-depth
treatment of these topics. Chapter 12 covers the Taguchi approach
to parameter design along with a critique of the method. Chapter 16
describes response surface design and introduce the reader to the
mixture problem.
- Rose, J.S., Chassin, L., Presson, C.C., & Sherman,
S.J. Multivariate Applications in Substance Use Research: New Methods
for New Questions. Mahwah, NJ: Lawrence Erlbaum Associates, 2000.
This edited volume introduces the latest advances in
quantitative methods and illustrates ways to apply these methods to
important questions in substance use research. Reflecting current
research trends, the book examines the use of longitudinal techniques
to measure processes of change over time. Researchers faced with the
task of studying the causes, course, treatment, and prevention of
substance use and abuse will find this volume helpful for applying
these techniques to make optimal use of their data.
Back to Top
B. Different Types of Study Designs
1. Case Studies
- Bromley, D.B. The Case-Study Method in Psychology
and Related Disciplines. Chichester: John Wiley & Sons, 1986.
This book deals with both theory and practice of individual
case-studies. It describes in detail how to conduct psychological
case-studies, how to evaluate them, and how to use diagrams and decision
analysis in case-studies.
- Gottman, J.M. N-of-one and N-of-two research in psychotherapy.
Psychol Bull 80:93-105, 1973.
Gottman's paper suggests that time-series methods are
useful in N-of-one research and we can use it to draw weak and strong
causal inferences. The author discusses the use of the method in process
research, outcome research, and measurement design. The paper also
pay special attention to the use of interrupted time series designs.
According to the author, time-series designs have the following advantage:
(1) It permits the study of the single subject and the use of subject-as-his-own-control
research; (2) It allows the study of the form of the effect of the
intervention over time; and (3) It allows one to use information over
time as feedback for making decisions.
- Hersen, M., & Barlow, D. Single Case Experimental
Designs: Strategies for Studying Behavior Change. New York: Pergamon
Press, 1976.
The book discusses general issues in single-case approach,
such as behavior trends and intrasubject averaging and the problem
of generality of findings. It also describes general procedures in
such research. The authors then write about different types of single-case
designs. Topics covered include basic A-B-A withdrawal designs and
their extensions, multiple baseline designs, alternative treatment
design. The last two chapters of the book introduce readers to some
appropriate statistical analyses that can be used to analyse data
from single-case designs and various ways to replicate findings from
these experiments.
- Hoyle, R.H. Statistical Strategies for Small Sample
Research. Thousand Oaks, CA: Sage Publications, 1999.
The book describes and illustrates statistical strategies
that are appropriate for analyzing data from small samples of fewer
than 150 cases. It covers such topics as the use of multiple imputation
software to deal with missing data in small data sets; ways to increase
power when sample size cannot be increased; strategies for computing
effect sizes and combining effect sizes across studies; how to hypothesis
test using bootstrapping; methods for pooling effect size indicators
from single-case studies; frameworks for drawing inferences from cross-tabulated
data; how to determine whether a correlation or covariance matrix
warrants structure analysis; under which conditions latent variable
modeling is a viable approach to correct for unreliability in the
mediator; the use of dynamic factor analysis to model temporal processes
by analyzing multivariate, time-series data; techniques for coping
with estimation problems in confirmatory factor analysis in small
samples; how the state space model can be used with small samples;
and the use of partial least squares as an alternative to SEM when
n is small and/or the number of variables in a model is large.
- Kratochwill, T.R., & Levin, J.R. Single-Case
Research Design and Analysis: New Directions for Psychology and Education.,
Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers, 1992.
The book summarizes the latest development in the field
of single-case research design and analysis. The edited volume consists
of contributions from researchers from various fields who utilize
or analyze data from such designs. The introductory and overview chapter
of the book also have a list of past textbooks that covered the topic
of single subject research design.
- Richards, S.B., Taylor, R.L., Ramasamy, R., & Richards,
R.Y. Single Subject Research: Applications in Educational and Clinical
Settings. San Diego: Singular Publishing Group, Inc., 1999.
The textbook provides background knowledge, basic concepts,
and understanding of relevant issues related to applied behavior analysis
and specifically to single subject research designs. It presents summaries
of the use of the designs and it outlines the major features of such
procedures. It also provides a review of the single subject research
literature as well as descriptions on how to actually implement these
designs. The designs that are covered includes withdrawal designs,
multiple baseline designs, alternative treatment designs. The last
chapter in the book also touched on the various kinds of analyses
that are useful for analyzing single subject design data.
Back to Top
2. Observational Studies
- Ahlbom, A. Biostatistics for Epidemiologists.
Boca Raton: Lewis Publishers, 1993.
The author presents biostatistical methods used in the
analysis of epidemiological studies. It examines the theoretical background
of the methods described and discussed general principles that apply
to the analysis of epidemiological data. Specific topics addressed
include statistical interference in epidemiological research, important
methods used for analyzing epidemiological data, multivariate models,
dose-response analysis, analysis of the interaction between causes
of disease, meta-analysis, and computer programs.
- Anderson, S., Auquier, A., Hauck, W.W., Oakes, D.,
Vandaele, W., & Weisberg, H.I. Statistical Methods for Comparative
Studies: Techniques for Bias Reduction. New York: John
Wiley & Sons, 1980.
Anderson et al. present various techniques for the design
and analysis of comparative studies, which are often used when randomization
is not feasible. The book covered main conceptual issues in the design
and analysis of comparative studies and possible bias incurred. The
book provide concise and useful discussion of the techniques of matching,
standardization and stratification, analysis of covariance, logit
analysis and log-linear analysis, survival analysis. The later part
of the book discuss the comparative effectiveness of the techniques
in reducing bias and other practical issues that must be faced before
drawing causal inferences from comparative studies.
- Clayton, D., & Hills, M. Statistical Models
in Epidemiology. Oxford: Oxford University Press, 1993.
The book is intended for students enrolled for a masters
degree in epidemiology, clinical epidemiology, or biostatistics. In
showing how to use probability models in epidemiology the authors
have chosen to emphasize the role of likelihood. Such an approach
to statistics is both simple and intuitively satisfying, and has the
additional advantage that it requires the model and its parameters
to be made explicit, even in the simplest situations. More complex
problems can then be tackled by natural extensions of simple methods
and do not require a whole new way of looking at things. The book
also covers in depth topics in regression models.
- Elwood, J.M. Causal Relationships in Medicine: A
Practical System for Critical Appraisal. Oxford: Oxford University
Press, 1988.
The book focuses on issues related to etiological research,
early diagnosis and screening, randomized and non-randomized clinical
trials, prognostic studies, health service issues, and the evaluation
of health education and promotion. The book start with a discussion
of the concept of causation, and then the author discuss the types
of study design that can be used to demonstrate causation and the
way in which their key results can be expressed precisely and simply.
Chapters 4 to 8 deal with issues pertaining to subject selection,
observation bias, confounding, and chance variation and various forms
of validity. Chapters 9, 10, and 11 give examples of the application
of the scheme to three study designs: cohort study, case-control study,
and randomized clinical trial.
- Feinstein, A.R. Clinical Epidemiology: The Architecture
of Clinical Research. Philadelphia: W.B. Saunders Co., 1985.
The content of this book can act as a "primer"
for non-clinical readers, and it explains the scientific approach
used by clinical investigators to quantitative challenges in group
data of diagnosis, prognosis, therapy, etiology, and other medical
topics. The text also cover topics and methods that are not included
in text of public health epidemiology. The first six chapters provide
an outlined overview of the field. The next four chapters deal with
statistics. Chapters 11 to 17 offer a single standard and model for
studies of cause-effect relationships in both therapeutic agents and
etiologic agents. Chapters 18 to 24 concern nonrandomized designs,
such as various forms of case-control studies. Chapters 25 to 27 are
devoted to process evaluation. The last three chapters contain some
special topics: the definition of "normal"; the contribution
and limitation of randomized trials and outline of other types of
epidemiology not mentioned in other parts of the book.
- Friedman, G.D. Primer of Epidemiology. New York:
McGraw-Hill Inc., 1994.
The book was originally written to be a brief, simple,
clear introduction to epidemiology for health care professionals.
In subsequent editions of the book, there were elaboration and updating
of some methodological concepts and factual information, as well as
problems to be solved in areas in epidemiology. In this fourth edition,
some new concepts and methods were introduced: the proportional mortality
rate and its use in occupational studies, the coefficient of variation,
the kappa coefficient and intra-class correlation coefficient, open
versus closed cohorts, Kaplan-Meier survival analysis, quality-adjusted
life years (QALYs), some principles concerning confounding variables,
Poisson regression, receiver operating characteristic (ROC) curves.
The chapter on case-control studies was considerably revised in this
new edition too.
- Kahn, H.A., & Sempos, C.T. Statistical Methods
in Epidemiology. New York: Oxford University Press, 1989.
This book is a major revision of H.A. Kahn’s An
Introduction to Epidemiologic Methods, which is meant to be a
book about statistical methods for chronic disease epidemiology that
could be read and understood by non-statisticians. Chapters 1 and
2 cover concisely selected elementary statistics as well as various
survey sampling designs and statistics. Chapters 3 and 4 describe
in detail the concepts of relative risk, odd ratios, and attributable
risk. Chapters 5 and 6 deal with different kinds of adjustment methods
that can be applied to epidemiologic data. Chapters 7 and 8 describe
followup studies, the use of life tables and person-year data. Later
chapters compare results for various methods of adjustment and lay
out in detail the data collection process in epidemiologic studies.
- Kelsey, J.L., Thompson, W.D., & Evans, A.S. Methods
in Observational Epidemiology. New York: Oxford University Press,
1986. (Also see new edition in 1996.)
This text is written for researchers who have background
for elementary epidemiology and biostatistics. The book mainly focuses
on observational epidemiologic studies but it also covers procedures
and concepts in experimental epidemiologic studies. Chapters 4 to
8 in the book describe major types of study designs used in epidemiology,
including the specific situations in which each is useful, the important
issues to be considered in carrying them out, and methods of statistical
analysis. Chapter 9 discusses techniques of epidemic investigation,
and chapter 10 describes common methods of sampling in epidemiologic
studies and presents methods and tables of investigation.
- Selvin, S. Statistical Analysis of Epidemiologic
Data. New York: Oxford University Press, 1995.
The aim of this book is to develop a clear understanding
of issues important to epidemiologic data analysis without depending
on sophisticated mathematics or advanced statistical theory. The level
of this text is beyond introductory but short of advanced. It draws
materials from the fields of statistics, biostatistics, vital statistics,
and epidemiology. A number of statistical methods are surveyed in
a way that should be useful to researchers concerned with the application
of statistics to epidemiologic data. Additionally, these methods are
chosen to illustrate general principles. For example, "jackknife"
estimation (chapter 5) is an excellent way to estimate specific parameters
from collected data but, at the same time, illustrates the application
of a "computer-intensive" estimation method.
- Weiss, N.S. Clinical Epidemiology: The Study of
the Outcome of Illness, 2nd ed. New York: Oxford University Press,
1996.
This book intends to pull together a number of areas
of research that are devoted to measuring and determining the factors
that affect the outcome of illness. It gives these areas of research
a collective label: clinical epidemiology. The author assumes that
the readers have background in introductory epidemiology or biostatistics.
The book begins with a description of the clinical context into which
the research findings ought to fit, hence the discussion of decision
analysis. Next, there are chapters on the evaluation of diagnostic
tests with respect to both their accuracy and their measurable contribution
to illness outcome. The book deals with both the experimental and
nonexperimental approaches. The concluding chapter of the book concentrates
on the role of studies that measure the natural history of illness.
An appendix presents selected statistical methods commonly used in
planning and analyzing data from clinical epidemiological data.
- Woodward, M. Epidemiology: Study Design and Data
Analysis. London: Chapman & Hall/CRC, 1999.
This book is about the quantitative aspects of epidemiological
research. The authors assume that readers have some basic knowledge
of statistics. The text goes through analytical methods for general
and specific epidemiological study designs, leading to a discussion
of statistical modeling in epidemiology in the final three chapters.
Chapter 1 includes a broad introduction to study design, and later
chapters are dedicated to particular types of design (cohort, case-control,
and intervention studies). Chapter 8 concerns the problem of determining
the appropriate size for a study.
Back to Top
3. Randomized Studies
- Fleiss, J.L. Design and Analysis of Clinical Experiments.
New York: John Wiley & Sons, 1986.
The book focuses on experimental designs that are most
relevant to clinical investigators. It also serves as a reference
for biostatisticians who work in clinical settings. Compared with
other texts on clinical trials, the book concentrates more on technical
aspects of design and statistical analysis. In particular, it deals
with the distinction between blocking and stratification, the use
of tables of random permutations to carry out randomized assignments,
the handling of unequal sample sizes, study of change estimation and
interpretation of factorial effects, and how to analyze data from
multicenter studies and crossover studies.
- Kirk, R.E. Experimental Design: Procedures for the
Behavioral Sciences. 2nd ed. Belmont, CA: Brooks/Cole, 1982.
This volume on experimental design is meant to be both
a text and a reference for researchers in the behavioral science.
It covers different kinds of experimental designs, starting from the
simplest completely randomized design to more complicated designs
like the split-plot factorial designs and the fractional factorial
designs. Compared to its first edition, the book also add in in-depth
coverage for multiple comparison procedures, the circularity assumptions
associated with block design, the partition of interactions into interpretable
contrast-contrast interactions, and the analysis of factorial designs
with unequal sample sizes and missing observations. It also included
a chapter on general linear model approach.
- Miettinen, O.S. The need for randomization in the study
of intended effects. Statistics in Med 2:267-271.
The author discusses the need for randomization as a
means of controlling confounders that is accentuated in the study
of intended effects (efficacy) as compared with unintended effects
(toxicity). The indication for intervention is itself a confounder
in the study of efficacy but not of toxicity. On the other hand, contraindications
represent only a minor confounder even in toxicity research. Control
of the indication in nonexperimental terms is commonly infeasible.
The author proposes that the solution to these problems is the randomized
clinical trial.
- Piantadosi, S. Clinical Trials: A Methodologic Perspective.
New York: John Wiley & Sons, 1997.
This book attempts to acquaint investigators with ideas
of design methodology that are also helpful in conducting, analyzing,
and assessing clinical trials. The discussion in the book pertains
to all types of trials: developmental, safety, comparative, and large-scale
studies, although there is an emphasis on comparative designs. The
author guides readers through the process of planning an experiment,
putting together a study cohort, assessing data, and reporting results,
and addresses the problems that are likely to confront any such study.
Back to Top
4. Quasi-Experiments
- Campbell, D.T., & Stanley, J.C. Experimental
and Quasi-Experimental Design for Research. Chicago: Rand McNally,
1963.
The book is the reprint of a chapter in the Gage’s
Handbook of Research on Teaching. It discusses alternatives
in the arrangement or design of experiments, with particular regard
to the problems of control of extraneous variables and threats to
validity. The authors then discuss various kinds of threats to internal
and external validity in different kinds of experimental designs.
The authors classify study designs into pre-experimental, experimental,
and quasi-experimental. Throughout the book, the authors try to illustrate
how one can utilize idiosyncratic features of any specific research
situation in designing unique tests of causal hypotheses.
- Cook, T.D., & Campbell, D.T. Quasi-Experimentation:
Design & Analysis Issues for Field Settings. Boston: Houghton
Mifflin Company, 1979.
The classic text on quasi-experiment designs starts
with a good discussion of plausible alternative interpretations of
findings from field research. The core of the book described various
quasi-experimental designs, such as research with nonequivalents groups
and interrupted time-series experiments. The authors raise issues
about analyzing results from such designs and how such analyses should
be carried out. The book also includes chapters that deal with causal
inferences from observational studies and the use of randomized experiments.
- Salzberg, A.J. Removable selection bias in quasi-experiments.
American Statistician 53(2):103-107.
Quasi-experiments are prone to selection bias, where
the effect of the treatment is confounded with pre-existing differences
in the treated and control sequence groups. Some quasi-experimental
designs are immune to certain specific selection biases. The article
shows that immunity to selection bias is not well characterized in
terms of selection-by-time interaction, and in particular some good
designs can be immune to certain types of selection bias even in the
presence of such interaction.
- Trochim, W.M.K. Research Design for Program Evaluation:
The Regression-Discontinuity Approach. Beverly Hills, CA: Sage
Publications, 1984.
The volume tries to make a case that we should move
beyond the traditional thinking on quasi-experiments as a collection
of specific designs and threats to validity toward a more integrated,
synthetic view of quasi-experimentation as part of a general logical
and epistemological framework for research. The papers in the edited
volume cover topics like the role of judgment in quasi-experimental
designs; the use of tailored designs; the crucial role of theory;
the attention to program implementation; the importance of quality
control; the advantages of multiple perspectives; evolution of the
concept of validity; and the development of increasingly complex realistic
analytic models.
Back to Top
5. Survey Research
- Rossi, P.H., Wright, J.D., & Anderson, A.A., eds.
Handbook of Survey Research. New York: Academic Press, 1983.
The handbook is an introduction to current theory and
practice of sample survey research. It address both the student who
desires to master these topics and the practicing survey researcher
who needs a source that codifies, rationalizes, and presents existing
theory and practice. Part I of the book sets forth the basic theoretical
issues involved in sampling, measurement, and the management of survey
organizations. Part II deals mainly with hands-on, how-to-do-it issues,
such as how to draw theoretically acceptable samples, and how to write
questionnaires. Part III considers the analysis of survey data with
separate chapters for each of the three major multivariate analyses
most in use, and one chapter on the uses of surveys in monitoring
overtime trends.
Back to Top
6. Program Evaluation
- Boruch, R.F., & Gomez, H. Sensitivity, bias, and
theory in impact evaluations. Professional Psychol 8:411-434,
1977.
Ordinary design of experiment technology invites underpowered
(insensitive) experiments because the measurement facts of life go
unrecognized. The purpose of this article is to identify these measurement
lapses and contribute to the development of more rigorous and socially
beneficial program evaluations. Measurement should concern itself
not only with reliability of dependent variable but, more importantly,
with validity of measurement and with measurement (systematic observation)
of the treatment variable. A technical appendix outlines a theory
of measurement in field evaluation.
- Chen, H.T. Theory-driven evaluation. In: Advances
in Educational Productivity 7:15-34, 1998.
This chapter introduces the basic concepts, rationale,
and methodology of designing and conducting theory-driven evaluations
with an emphasis on application so that this evaluation approach could
be widely applied in areas such as education.
- Crits-Christoph, P., & Mintz, J. Implications of
therapist effects for the design and analysis of comparative studies
of psychotherapies. J Consult Clin Psychol 59(1):20-26, 1991.
The authors present technical reasons why therapists
should be included as a random design factor in the nested analysis
of (co)variance (AN[C]OVA) design commonly used in psychotherapy research.
Incorrect specification of the ANOVA design can, under some circumstances,
result in incorrect estimation of the error term, overly liberal F
ratios, and an unacceptably high risk of type I errors. Both results
from simulation studies and a reanalysis of data from 10 psychotherapy
outcome studies are discussed, and implications of these results for
future research designs are examined.
- Pentz, M.A., Trebow, E.A., Hansen, W.B., MacKinnon,
D.P., Dwyer, J.H., Johnson, C.A., Flay, B.R., Daniels, S., & Connack,
C. Effects of program implementation on adolescent drug use behavior:
The Midwestern Prevention Project (MPP). Eval Rev 14:264-289,
1990.
The study evaluates the relationship between level of
program implementation and change in adolescent drug use behavior
in the Midwestern Prevention Project (MPP), a school- and community-based
program for drug abuse prevention. Trained teachers implement the
program with transition year students. Implementation was measured
by teacher self-report and validated by research staff reports. Adolescent
drug use was measured by student self-report; an expired air measure
of smoking was used to increase the accuracy of self-reported drug
use. Regression analyses were used to evaluate adherence; exposure,
or amount of implementation; and reinvention. Results showed that
all schools assigned to the program condition adhered to the research
by implementing the program. Exposure had a significant effect on
minimizing the increase in drug use from baseline to 1 year. Exposure
also had a larger magnitude of intervention effect than experimental
group assignment. Reinvention did not affect drug use. Results are
discussed in terms of research assumptions about quality of program
implementation, and possible school-level predictors of implementation.
- Hawkins, J.D., Abbott, R., Catalano, R.F., & Gillmore,
M.R. Assessing effectiveness of drug abuse prevention: Implementation
issues relevant to long-term effects and replication. In: Leukefeld,
C.G., & Bukoski, W.J., eds. Drug Abuse Prevention Intervention
Research: Methodological Issues. National Institute on Drug
Abuse Research Monograph 107. DHHS Pub. No. (ADM)91-1761. Washington,
DC: Supt. of Docs., U.S. Govt. Print. Off., 1991, pp. 213-234.
The chapter outlines a strategy for assessing the long-term
effects of drug abuse prevention interventions in replicable studies.
It consists of a theory-driven data collection and analysis approach
that implies the need to link proximal intervention outputs to more
distal outcomes desired. The approach also requires prospective longitudinal
followup studies in which complete panels of subjects who vary with
respect to the levels of key predictor constructs are followed up
through the period of their highest risk for drug use.
- Leithwood, K.A., & Montgomery, D.J. Evaluating
program implementation. Eval Rev 4:193-214,1980.
The authors describe a methodology for evaluating program
implementation. Requirements for such a methodology are derived from
an analysis of the functions to be performed by implementation evaluation,
the nature of the program being implemented, and characteristics of
the implementation process. Central features of the methodology involve
procedures for the development of a multidimensional profile of the
program as it evolves in practice from non- to full implementation.
The profile then serves as the basis for instrument development; data
collected through the instruments locate program user behavior in
relation to the dimensions and levels of use described by the profile.
Uses of resulting data to serve program management goals are outlined.
- Rich, K.C. Discussion: Research environment and use
of multicenter studies in perinatal substance abuse research. In:
Kilbey, M.M., & Asghar, K., eds. Methodological Issues in Epidemiological,
Prevention, and Treatment Research on Drug-Exposed Women and Their
Children. National Institute on Drug Abuse Research Monograph 117.
DHHS Pub. No. (ADM)92-1881. Washington, DC: Supt. of Docs., U.S. Govt.
Print. Off., 1992, pp. 293-304.
The chapter discusses the challenges presented by research
environment. Often investigators have to contend with political, social,
environmental, and medical factors that could determine the type and
content of studies that can be performed and the likelihood of their
success. The latter half of the chapter discusses the advantages and
challenges of multicenter collaborative studies.
- Wortman, P.M. Evaluation research: A methodological
perspective. Am Rev Psychol 34:223-260, 1983.
The review chapter first took a historical perspective
and examines the predominant methodological school of thought from
the "early days" in the development of evaluation research.
It then considers more recent development and concerns in the field.
The core of the chapter consists of the examination of three evaluative
methods that have become increasingly important in the field: (1)
social experimentation; (2) meta-analysis; and (3) cost-effectiveness
analysis.
Back to Top
|
