U.S. General Accounting Office, Program Evaluation and Methodology Division. Designing
Evaluations. Washington, DC; 1991. pp. 19-24.
Considering the Evaluation's Constraints
Time is a constraint. It shapes the scope of the evaluation question and the range of
activities that can be undertaken to answer it. It demands trade-offs and establishes
boundaries to what can be accomplished. It continually forces the evaluator to think in
terms of what can be done versus what might be desirable. Because time is finite (and
there is never enough of it), the evaluator has to plan the study in "real time"
with its inevitable constraints on what question can be posed, what data can be collected,
and what analysis can be undertaken.
A rule of thumb is that the time for a study and the scope of the question being addressed ought to be directly related. Tightly structured and narrow investigations are more appropriate when time is short. Any increase in the scope of a study should be accompanied by a commensurate increase in the amount of time that is available for it. The failure to recognize and plan for this link between time and scope is the Achilles heel of evaluation.
Linking scope and time in the study design is important because the scope is determined by the difficulty of the evaluation, the importance of the subject, and the needs of the user, and these are also determinants of time. Though it may be self evident to say so, difficult evaluations, important evaluations, and evaluations in which there is a great deal of interest have different demands with respect to time than other evaluations. No project is "too long" or "too short" within this context.
The need of the study's audience as a time constraint merits additional comment. Evaluations are requested and conducted because someone perceives a need for information. Producing that information without a sensitivity to the user's timetable diminishes its usefulness. For example, a report to the Congress may answer the questions correctly but will be of little or no use if it is delivered after the legislative hearings for which it is needed or after the preparation of a new budget for the program.
Cost is a constraint. The financial resources available for conducting a study partly determine the limits of the study. Having very few resources means that the evaluator will have to consider tight limitations on the questions, the modes of data collection, the numbers of sites and respondents, and the extent and elegance of the analysis. As the resources expand, the constraints on the study become less confining. Having more funds might mean, for example, either longer time in the field or the opportunity to have multiple interviews with respondents or to visit more sites or choose larger samples for sites. Each of these items has a price tag. What the evaluator is able to purchase depends on what funds are available.
It should be stressed that regardless of what funds are available, design alternatives should be considered. Cost is simply an important constraint within which the design work has to proceed. If only a stipulated sum is available, the evaluator has to determine what can be done with that sum in order to provide information that is relevant to the questions. The same resources might allow three or four quite distinct approaches to an evaluation. The challenge is to consider the strengths and weaknesses of the various approaches. Like the constraint of time, cost does not determine the design. It helps establish the range of options that can be realistically examined. Even when resources can be expanded, cost is still a constraint. However, the design problem then becomes one of cost-effectiveness, or getting value for the dollar, rather than one of what can be done within a stipulated sum.
One other point: the quality of an evaluation does not depend on its cost. A $500,000 evaluation is not necessarily five times more worthy than a $100,000 evaluation. An expensive study poorly designed and executed is, in the end, worth less than one that costs less but addresses a significant question, is tightly reasoned, and is carefully executed. A study should be costly only when the questions and the means of answering them necessitate a large expenditure. As with the constraint of time, there is a direct correlation between the scope of a study and the money available for conducting it.
Staff expertise is a constraint. The design for an evaluation ought not to be more intricate or complex than what the staff can successfully execute. Developing highly sophisticated computer simulations or econometric models as part of an evaluation when the skills for using them are not available to the evaluation team is simply a gross mismatch of resources. The skills of the staff have to be taken into account when the design is developed.
It is perhaps too negative to consider staff expertise as only a constraint. In the
alternative view, the design accounts for the range of available staff expertise and plans
a study that uses that expertise to the maximum. It is just as much a mismatch to plan a
design that is pedantic, low in power, and completely unsophisticated when the staff are
capable of much more and the questions demand more as it is to create a design that is too
complex for the expertise available. In either instance, of
course, a design is determined not by expertise but by the nature of the questions.
A realistic understanding of the skills of the staff can play an important role in the kinds of design options that can be considered. An option that requires skills that the staff do not have will fail, no matter how appropriate the option may be to the evaluation questions. A staff with a high degree of technical training in a variety of evaluation strategies is a tremendous asset and greatly expands the options.
Some designs demand a level of expertise that is not available. When this happens, consultants can be brought into the study or the staff can be given short intensive courses or complex and difficult portions of the design can be isolated and performed under contract by evaluators specializing in the appropriate type of study. In other words, the stress is on considering the options available. Preference should be given to building the capability of current staff. When this cannot be done, or time and cost do not allow it, expertise can be procured from outside in order to fulfill the demands of the design.
Location and facilities are secondary constraints in comparison to the others we have discussed, but they do impinge on the design process and influence the options. Location has to be considered from several aspects. One is the location of the evaluator visa-vis where the evaluation is to be conducted. Location is less critical for a national study, since most areas can be reached by air within a few hours, but it increases in importance if the study examines only a few individual projects. The accessibility and continuity of data collection may be jeopardized if the evaluator is on the east coast and the sites are in the South, in the Midwest, and on the west coast. A situation such as this may have to incorporate local persons as members of the evaluation team and may increase the utility of a mail questionnaire or telephone interviews compared to face-to-face interviews.
Another aspect of location has to do with the social and cultural mores of the area where the evaluation is to be conducted. For example, to gain valid and insightful data on attitudes toward rural mental health clinics, it may be wise not to send interviewers from urban areas. Good interviewing necessitates empathy between the persons involved, and it may be hard to generate between an interviewer and a respondent whose backgrounds are very different.
A third aspect of location is the stability of the population being studied. A neighborhood where residence is transient may necessitate a different strategy from a neighborhood where most people have lived in the same house for 40 years and have no intention of moving.
Finally, the evaluator must consider whether a trip to a site is justified at all. For example, if it costs $3,000 to travel to a remote town to ascertain whether a school there is using a $1,500-computer provided by a U.S. Department of Education grant, the choice of not going is defensible.
The constraint of facilities on the design options also has more than one aspect. One
has to do with data collection and data processing. For example, if the study involves
entering large aggregates of data into a computer, the equipment to do so must be
available, or the money must be available for contracting the work. Similarly, if the
design calls for data analysis at computer terminals with phone connections to the main
computer, the equipment is a must. The absence of such facilities limits both the kind and
the extent of the data one can collect.
Another aspect is the need for periodic access to facilities that are not under the
auspices of the project or program being studied. For example, to interview welfare
clients in a welfare office about the treatment and service they are receiving there may
be to risk highly biased answers. How candid can a client be, knowing that the caseworker
who has made decisions on food, clothing, and rental allowances for the client's family is
in the next room? "Neutral turf" cannot guarantee candid answers, but it may
lessen anxiety and it can contribute to the authenticity of the evaluator's promise of
anonymity and confidentiality. The example applies equally to interviews with persons who
hold positions of power and influence.