U.S. Department of Justice Office of Community Oriented Policing Services Measuring Excellence: Planning and Managing Evaluations of Law Enforcement Initiatives Kristin Ward | Susan Chibnall | Robyn Harris ICF International, a global professional services firm, partners with government and commercial clients to deliver consulting services and technology solutions in energy, climate change, environment, transportation, social programs, health, defense, and emergency management. In October 2005, ICF acquired Caliber Associates, an organization that provides full life-cycle program consulting and support in research and evaluation, communications, training and technical assistance, and information technology. Together, ICF and Caliber have expanded capabilities to assist both civilian and national security organizations to develop more effective and innovative human resource, education, and social service programs. Prior to the acquisition, Caliber had been awarded a 2-year contract from the U.S. Department of Justice Office of Community Oriented Policing Services (the COPS Office) to develop Measuring Excellence: Planning and Managing Evaluations of Law Enforcement Initiatives. Tailored specifically for the law enforcement community, the guide emphasizes the importance of program evaluation to law enforcement. It is in a user-friendly format that assists law enforcement administrators throughout the evaluation management process—from the initial stages of thinking about and planning a program evaluation, through selecting, designing, and implementing program evaluations, and using the results to meet the needs of their communities and stakeholders. Measuring Excellence: Planning and Managing Evaluations of Law Enforcement Initiatives Kristin Ward | Susan Chibnall | Robyn HarrisJune 2007 This project was supported by cooperative agreement #2003-CK-WX-K052 awarded by the Office of Community Oriented Policing Services, U.S. Department of Justice. The opinions contained herein are those of the authors and do not necessarily represent the official position of the U.S. Department of Justice. References to specific companies, products, or services should not be considered an endorsement by the authors or the U.S. Department of Justice. Rather, the references are illustrations to supplement discussion of the issues. Table Of Contents Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Chapter I: Why Evaluate? . . . . . . . . . . . . . . . . . . . . . . 1 Purpose of the Guide . . . . . . . . . . . . . . . . . . . . . . . . 3 Why Evaluate? The Rationale for Program Evaluation . . . . . . . . . 3 Real-World Examples of Evaluation in Policing . . . . . . . . . . . 10 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Chapter II: Key Concepts in Evaluation . . . . . . . . . . . . . . 15 The Different Types of Evaluation . . . . . . . . . . . . . . . . . 17 Evaluation Designs . . . . . . . . . . . . . . . . . . . . . . . . 29 The Continuous Feedback Loop of Program Planning and Evaluation and the Key Steps in the Evaluation Process . . . . . . 40 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Chapter III: Conducting an Internal Evaluation . . . . . . . . . . 45 Steps in the Evaluation Process . . . . . . . . . . . . . . . . . . 49 Step One: Specify—Evaluation Planning . . . . . . . . . . . . . . . 50 Step Two: Collect—Data Collection . . . . . . . . . . . . . . . . . 58 Step Three: Compare—Data Analysis . . . . . . . . . . . . . . . . . 69 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Chapter IV: Managing Evaluations . . . . . . . . . . . . . . . . . 77 How Do I Manage an Evaluation? . . . . . . . . . . . . . . . . . . 79 What Do I Need to Do During the Evaluation? . . . . . . . . . . . . 88 What Do I Need to Do After the Evaluation? . . . . . . . . . . . . 89 Challenges to Managing Evaluations . . . . . . . . . . . . . . . . 92 Should I Hire an External Evaluator? . . . . . . . . . . . . . . . 93 Chapter V: Emerging Topics in Evaluation Management . . . . . . . 101 Collaborative Evaluation . . . . . . . . . . . . . . . . . . . . 103 Participatory Evaluation . . . . . . . . . . . . . . . . . . . . . 107 Performance Measurement . . . . . . . . . . . . . . . . . . . . . 110 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . 113 Exhibit I-1: Six Myths About Evaluation . . . . . . . . . . . . . . 6 Exhibit II-1: Examples of Questions Addressed in a Needs Assessment . . . . . . . . . . . . . . . . . . . . . . . . . 19 Exhibit II-2: Examples of Questions Addressed in a Process Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 21 Exhibit II-3: Examples of Questions Addressed in an Outcome Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 23 Exhibit II-4: Examples of Questions Addressed in an Impact Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 26 Exhibit II-5: Examples of Questions Addressed in a Cost-Benefit Evaluation . . . . . . . . . . . . . . . . . . . . . . 27 Exhibit II-6: Overview of Evaluation Types . . . . . . . . . . . . 29 Exhibit II-7: Strengths and Weaknesses of Different Types of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 36 Exhibit II-8: Common Threats to the Validity of Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . 39 Exhibit II-9: The Program Planning and Implementation Cycle . . . . 40 Exhibit II-10: The Evaluation Cycle . . . . . . . . . . . . . . . . 41 Exhibit II-11: Integration of the Planning and Evaluation Cycles . 42 Exhibit III-1: Difference Between Academic Research and Internal Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 48 Exhibit III-2: Tips for Developing Goals and Objectives . . . . . . 54 Exhibit III-3: Sample Logic Model . . . . . . . . . . . . . . . . . 56 Exhibit III-4: Tips for Developing Surveys and Survey Questions . . 62 Exhibit III-5: Tips for Conducting Surveys . . . . . . . . . . . . 64 Exhibit III-6: Tips for Planning and Facilitating Focus Groups . . 66 Exhibit III-7: Data Collection Methods at a Glance . . . . . . . . 67 Exhibit III-8: Options for Organizing and Reporting Qualitative Data . . . . . . . . . . . . . . . . . . . . . . . . . 72 Exhibit IV-1: Assessing Evaluator Capabilities . . . . . . . . . . 94 Exhibit IV-2: Advantages of Hiring an External Evaluator Compared to Using Internal Staff . . . . . . . . . . . . . . . . . 95 Exhibit IV-3: Sources for Finding an External Evaluator . . . . . . 98 Exhibit V-1: Difference between Participatory Evaluation and Traditional Evaluation . . . . . . . . . . . . . . . . . . . . 109 Exhibit V-2: Differences between Performance Measurement and Program Evaluation . . . . . . . . . . . . . . . . . . . . . . 110 Appendixes Appendix A: Sample Needs Assessment Action Plan Template . . . . . 117 Appendix B: Preparing Measurable Goals and Objectives . . . . . . 118 Appendix C: Planning Model Template . . . . . . . . . . . . . . . 121 Appendix D: Community Satisfaction Survey . . . . . . . . . . . . 123 Appendix E: Sample Focus Group Checklist . . . . . . . . . . . . . 126 Appendix F: Instrument Development Checklist . . . . . . . . . . . 128 Appendix G: Topics to Cover in an Informed Consent Statement for Research Participants . . . . . . . . . . . . . . . . . . . . 130 Appendix H: Sample Evaluation Plan Template . . . . . . . . . . . 132 Appendix I: Sample Interview Questions Checklist . . . . . . . . . 133 Appendix J: Glossary . . . . . . . . . . . . . . . . . . . . . . . 134 Appendix K: Sample Job Announcement . . . . . . . . . . . . . . . 139 Appendix L: Evaluation Resources . . . . . . . . . . . . . . . . . 141 References . . . . . . . . . . . . . . . . . . . . . . . . . . . .153 Foreword The primary purpose of this guide is to provide you with information for conducting and managing program evaluations in your law enforcement agency. However, during the needs assessment conducted to develop this guide, during which input was solicited from law enforcement personnel nationwide through focus groups and telephone interviews, it became clear that many police administrators and managers would benefit from an enhanced understanding of evaluation. From the needs assessment, five key factors arose that shaped the development of this guide: 1. Many of you would benefit from exploring the value, importance, and potential uses of evaluation information • Chapter I: Why Evaluate? 2. Many of you also would benefit from learning more about evaluation basics, so you could better manage and conduct a program evaluation • Chapter II: Key Concepts in Evaluation. 3. In many law enforcement agencies, the person responsible for managing evaluation activities also conducts them • Chapter III: Conducting an Internal Evaluation. 4. Law enforcement agencies around the country are conducting their own program evaluations of program processes, outcomes, and impacts. In addition, for large or complex community initiatives, law enforcement agencies are conducting external evaluations with the help of evaluation experts outside their department. Many of you could use tips on managing both types of evaluative endeavors • Chapter IV: Managing Evaluations. 5. Law enforcement agencies are increasingly engaging in new types of evaluation methodologies, including participatory evaluations, collaborative evaluations, and performance measurement activities (i.e., monitoring and reporting program accomplishments) • Chapter V: Emerging Topics in Evaluation Management. This guide presents the basic concepts and guidance for planning, implementing, and managing an evaluation, including understanding evaluation concepts and activities. Where more information might be needed or desired, an appendix is provided with references to other, more detailed, sources. This guide is designed to help law enforcement agencies that plan to begin conducting evaluations as well as those that want to refine and improve their evaluation management techniques. The authors thank Bonnie Bucqueroux for reviewing the report. Through her suggestions, she made important contributions to the work. Chapter I: Why Evaluate? Why Evaluate? Purpose of the Guide Modern policing continues to place new demands on police leadership. Policing today not only addresses crime, fear of crime, and social and physical disorder, but also requires the police to be involved in collaborative problem solving—the only government agency patrolling neighborhoods around the clock. As the scope of police work broadens, so too does its measurement, extending beyond numbers of arrests made and response time and making defining success an increasingly complex challenge. Whose views matter? The police themselves? Elected officials, the business community, schools, the faith community, the media, average citizens? Particularly in an environment of strict budget considerations, police are compelled to move beyond mere opinion and justify their work through comprehensive, quality evaluations. Persuading police to embrace evaluation prompts the question, “What’s in it for us?” Also, for evaluation to gain widespread acceptance, police administrators need the tools to conduct evaluations and do them well. Therefore, this guide begins with an explanation of why evaluation must become an integral part of police work, especially as police administrators repeatedly find themselves asked to do more with less. The guide also provides practical information and advice on conducting quality evaluations that recognize the unique challenges inherent in police work— evaluations that can withstand rigorous scrutiny and help build community trust and support. This guide is a user-friendly resource that is supplemented with appendixes that take you through several practical exercises and direct you to more detailed information from other sources. Why Evaluate? The Rationale for Program Evaluation There are many kinds of evaluation, but the most common is program evaluation, in which the goal is to discover how well a particular program is working. Program evaluation is defined as a systematic process of gathering and analyzing information for the purposes of program assessment, program improvement, and, in a broader sense, strategic management. It provides a structured way to verify, document, and quantify program activities and their effects. Done well, evaluation provides meaningful results to help you make informed judgments about the effectiveness and efficiency of your strategies and programs. You should be able to demonstrate that you are committing resources to efforts that produce tangible benefits for the entire community. Evaluation can help you in these ways: Compare alternatives. What works best? Framed correctly, evaluation can let you compare programs and strategies so you can prioritize where to put scarce resources. Determine appropriateness. Is the program the right way to address the problem? Does it enjoy not only the support but also the participation of the community? Policing is about both means and ends and evaluation can help you determine whether the program is the best and most inclusive way to deal with the problem. Clarify program objectives. What are you trying to accomplish? How will you define success? A well-planned evaluation requires you to clarify assumptions about the links between your target population, the program activities, and the immediate, intermediate, and long-term outcomes you expect. There is wisdom in the saying: “If you don’t know where you are going, any path will take you there.” Evaluation helps you decide where you are going and how to determine whether you are taking the best path to get there. Advocate for your efforts. How can you energize supporters and persuade critics to come on board? Perhaps the biggest virtue of an evaluation is that it has more validity than mere opinion. It is easy to say that you know something is working and provide anecdotes that support that view. However, as we grow more sophisticated in our decision-making, it becomes increasingly important to conduct outcome-based evaluations. Describe the program and educate the community. Do key stakeholders understand the initiative and how it works? Do average citizens know what the police are doing and why? In the rush to do good things, police often fail to explain their actions. Evaluation can require documenting not only the program but also the underlying philosophy and the results. People outside police circles often believe in the myth of the easy answer or the quick fix to crime and social disorder problems. A thoughtful evaluation can help stakeholders and citizens understand better the challenges of policing. Solidify support. How can we keep supporters engaged? Positive results from a well-planned evaluation can be used to justify a program’s existence, maintain commitments from program administrators or the community, and leverage additional resources from funders, community partners, and other stakeholders. Address program cost. Are you operating within budget? Are there opportunities for cost savings? Do the results justify the investment? Evaluation is a process that provides feedback to help you make adjustments and can give valuable information about ways to save money and whether the program delivered sufficient “bang for the buck.” Make informed program improvements. Do you need to make midcourse corrections? Are there changes that should be made before the program is allowed to continue or is replicated elsewhere? The information collected during an evaluation can provide the information you need to make intelligent changes likely to improve opportunities for success. Monitor program integrity. Are you really doing what you said you would do? Tracking the number and type of activities you offer, the number and type of participants involved, and your activity-related expenses can help determine if you actually are implementing the program as intended and promised. Provide a blueprint for peers. How will newcomers learn what to do? Will others inside the agency be able to implement the program or strategy? A quality evaluation provides a road map for others to follow, with the potential barriers and pitfalls identified. Recognize and capture unanticipated or unintended program effects. Were there “side effects” that need to be documented and understood? Evaluation often uncovers positive or negative program effects that were neither anticipated nor intended—consequences that can be lost if no evaluation process is in place. Inform internal systems. Can the findings be used to improve functioning inside the department? Evaluation also can be used more holistically and broadly as a tool for internal strategic planning and management. If, for example, you find that a new initiative’s success correlates to the communication skills of the officers implementing the program, this has implications for your human resources department and its recruiting guidelines. Assess overall effectiveness. Did the program or effort work as intended? Data collected during an evaluation will help you document and demonstrate what is working and what is not. Pilot test innovations. How can we identify new ideas for the future? Policing must never be static and managers must be willing to take reasonable risks on new ideas. But the challenge is to conduct quality evaluations to determine whether promising ideas might pan out in practice. Contribute to the field. Does this evaluation help our understanding of future directions? The lessons that you learn from your evaluation can be shared with other law enforcement agencies and communities. Sharing evaluation findings regularly will contribute to creating a much-needed body of verifiable law enforcement knowledge. Evaluation in the Law Enforcement Context Formal evaluation is rarely, if ever, required to maintain general funding of police department operations. Often, it is not even required of state and federal law enforcement grants. Consequently, many police chiefs and other law enforcement administrators often do not make evaluation a priority. In part, this reluctance to integrate and implement evaluation might stem from not wanting to invest in learning bad news. Common myths and misconceptions about evaluation, as well as a lack of understanding about the place of evaluation in the policing context, play a role in the scarcity of evaluation activity in police departments across the country. Six myths about evaluation are presented in Exhibit I-1. Exhibit I-1: Six Myths About Evaluation The Myth Police leaders don’t value evaluation; lack of internal support. The Reality Today’s police leaders must be able to prove that they are spending tax dollars wisely. Many police departments today are conducting and managing meaningful evaluations of their efforts. The Myth Police departments do not need to do evaluation (not required by law or grants). The Reality Quality evaluations are for YOU, not THEM. Evaluation provides a framework that assists you in doing the best job possible within existing resources. The Myth Police departments lack the resources to do proper or meaningful evaluation studies. The Reality Program evaluation need not be costly or time-consuming. Valuable results can be obtained by asking targeted questions or using existing data. Do what you can with what you have now and build from there. The Myth Police staff lack the skills for evaluation design and analysis. The Reality Program evaluation is not just for academics. The best evaluations are based on a common-sense approach to gathering and analyzing information. The Myth Police departments lack relevant data to evaluate their efforts. The Reality The data will be there if you weave your evaluation strategy into program planning. If you did not capture appropriate data this time, make sure to craft a better plan for the future. The Myth Police departments fear negative results. The Reality The politics of policing can make top command skittish about investing in efforts that may generate bad news. But evaluation is really about improving police effectiveness and prioritizing scarce resources. An evaluation is not a “gotcha,” but a blueprint for improvement. Evaluation informs spending decision In addition to the reasons listed in Exhibit I-1, police officials need to embrace evaluation to maintain public trust. As stewards of public funds, police departments are accountable for ensuring that taxpayer dollars are used appropriately, effectively, and efficiently. Program evaluation not only addresses those concerns, but also can provide a cost-benefit analysis, which is a tool that can help administrators decide where to invest limited funds. While there can be risks involved in evaluating programs and strategies (e.g., discovering that a popular or expensive strategy is not working), most administrators would rather know the truth as soon as possible. With that knowledge, decision-makers can make adjustments based on informed feedback, or abandon a misguided effort before more money is wasted. Evaluation supports community policing and problem-oriented policing Program evaluation is essential for, and compatible with, cutting-edge policing approaches such as community policing and problem-oriented policing. For instance, those familiar with the SARA (Scanning, Analysis, Response—and Assessment) model will recognize that evaluation is an integral component. Problem-oriented policing requires recognition (scanning) and analysis of a problem, a strategically planned response to address it, and an assessment (i.e., an evaluation) of how well the strategy solved or reduced it. Evaluations—especially those that include stakeholder participation—can further the objectives of community policing. By using strategies that capture and analyze findings from the community and participating professionals, evaluation offers insights into opportunities for enhancement and improvement. Also, evaluating community-based initiatives with the help of citizen- participants can bolster credibility, buy-in, partnership, and accountability, while contributing to long-term citizen satisfaction and neighborhood development and improvement. Moreover, law enforcement already is data-driven. In fact, it could be said that informal evaluation occurs all the time, as police administrators consult a variety of data to make decisions—crime data, calls for service, clearance rates, discussions with officers. The challenge is to move from an informal approach that might not be rigorous or comprehensive, or that has gaps, to a more informed and institutionalized process that contributes to better decision-making. As problem-oriented policing and community policing increasingly become the ways in which most police agencies deliver service, comprehensive evaluation provides a way to move beyond the narrow and incomplete measures of police performance of the past, such as crime rates and response times. This guide provides both a rationale for quality evaluation and the tools you will need to do the job. “ What you don’t know CAN hurt you. Evaluation lets you know what works and what doesn’t.” – Chief Tony Kleibecker, Muskegon (Michigan) Police Department “ We need to evaluate the short-term temporary gains associated with reactive policing so that we can effectively plan for long-term neighborhood improvements.” – Chief Harry Dolan, Grand Rapids (Minnesota) Police Department Examples of How Law Enforcement Agencies Nationwide Use Evaluation Despite the obstacles to conducting formal evaluation in the policing context, many police agencies are successfully incorporating program evaluation into their daily practice. The Community Policing Consortium offered advice regarding what to evaluate in the policing context: Internal structures and systems: Departmental organization Recruiting Training Performance evaluation Promotion Formal and informal rewards Internal communication Morale Strategic plans: Assess the process of developing a strategic plan Monitor implementation of the strategic plan Resource allocation: Assess the decision-making process for resource allocation Monitor resources in terms of what is being spent, where, why, and whether money could be spent more wisely Pilot projects: Determine the benefits and drawbacks of wider implementation Effectiveness of community policing efforts/strategies Assess outcomes from the perspectives of different stakeholders Quality of police services: Customer satisfaction Community perceptions of the police (Community Policing Consortium Resources: Strategic Planning for Community Policing, n.d.) “ We don’t have enough resources, time, or energy to waste on things that don’t work … How can we prove that the millions of dollars spent on us are actually having some effect?” – Chief Jim Bueerman, Redlands (California) Police Department Evaluations vary from large-scale, expensive, external, and experimental efforts to small-scale, low-budget, internal assessments of local problem-solving initiatives. Their purposes range from discrete, one-time evaluations of specific activities to using evaluation as a long-term strategic development and management tool. The following are examples of successful evaluations that law enforcement agencies across the country have conducted: Evaluation of the effectiveness of a blended housing and recreation initiative to lower risk factors for, and incidence of, crime in neighborhoods, using a citywide experimental design (Redlands, California: 2004). Evaluation of a 12-officer bike patrol program, using neighborhood satisfaction surveys and process/implementation evaluation to understand whether the program was being implemented as intended and whether funds were well spent (Muskegon, Michigan: 2003). Evaluation of the effectiveness of multidisciplinary collaboration among city groups and agencies to identify drug dealers and increase the number of narcotics raids (Muskegon, Michigan: 2003). Evaluation of youth violence-prevention efforts (e.g., partnerships between police departments and schools to prevent and decrease bullying), using pre- and post-program surveys with students, parents, and teachers, as well as observations (Chula Vista, California: 2003). Evaluation of collaborative efforts to reduce residential burglaries in a new housing development area, using interviews with victims, street-view environmental assessments, review of incident reports, and tracking and comparing trends in theft data in various neighborhoods (Chula Vista, California: 2003). Evaluation of efforts to address disorder in budget motels using pre- and post-program interviews with motel managers (Chula Vista, California: 2003). Real-World Examples of Evaluation in Policing This guide will discuss how you can manage and use evaluation to measure and improve programs and direct your strategic vision. Throughout the guide, we will reference real-world examples, describing how your colleagues in law enforcement have planned, implemented, and ultimately learned from evaluation in their work. We begin in this chapter with two examples that illustrate the range of evaluation activity and purpose in the policing context. We hope they will inspire and encourage you to learn more about what your department can achieve through evaluation. The first example is the Minneapolis Police Department’s Hawthorne Huddle, which clarifies how evaluation can help you gauge the effectiveness of community policing strategies and collaborations with community agencies and citizens to improve neighborhoods. The Hawthorne Huddle also illustrates how evaluation can assist in improving or complementing the strategies used to build police/ community partnerships. The second case study, the Colorado Springs Police Department’s Police Accountability and Service Standards (PASS) model, exemplifies how evaluation can be utilized for developing and assessing the strategic vision, goals, and objectives of law enforcement agencies. The Hawthorne Huddle A collaborative and action-oriented program evaluation model Conditions in the Hawthorne neighborhood of North Minneapolis declined into the mid-1990s. The area was plagued by high crime, poor rental property management, and a deteriorating housing stock. Drug dealing and other crime contributed to the disorder in the neighborhood, and residents felt fearful, disenfranchised, and abandoned by what they perceived as a lack of effort on the part of city officials to improve the situation. Police responsible for patrolling the area felt frustrated by the fact that they saw no sign of the bad times abating in the area. The Minneapolis Police Department (MPD) decided to collaborate with other city and county agencies, nonprofit organizations, a major corporate sponsor (the General Mills Foundation), and community residents to develop a plan to deal with crime and disorder in the area. MPD used crime mapping and analysis of police reports to define and target the biggest problems in the area. Next, they organized neighborhood meetings and encouraged citizen participation in block clubs to garner support and assistance with problem-solving efforts. Residents quickly accepted the block club meetings, which grew over time. To this day, agency and community members gather at monthly meetings called the Hawthorne Huddle. Additional strategies were aimed at reducing the number of criminal narcotics arrests at specific addresses, as well as the number of vandalism incidents and burglaries in the neighborhood. To assess the effectiveness of these strategies, several evaluation techniques were ultimately used. Evaluation strategies were collaborative and participatory in that MPD was not solely, or even predominantly, responsible for evaluation of the Hawthorne Huddle. Instead, the agencies involved in the project worked together to define goals and objectives and decided how success would be measured. Participating agencies and organizations volunteered to collect various kinds of relevant data. MPD gathered and analyzed crime statistics for the area while other organizations conducted surveys and focus groups to measure residents’ feelings of safety and satisfaction, and gauge ongoing concerns. This collaboration helped keep Hawthorne Huddle program activities grounded in real community issues and concerns. This collaborative and action-oriented evaluation model relies on stakeholders working together to identify the problem, develop goals and objectives, define success, decide how and what to measure, and determine how to use results to continue improving the community. Evaluation results are released to the community at monthly Huddle meetings. Stakeholders use the information to make decisions about future efforts (reflecting the values of community policing and its emphasis on building connections to local leaders), work together to identify community needs, and leverage evaluation results to improve the community. The Police Accountability and Service Standards (PASS) Model A strategic process, program, and performance evaluation model Under the leadership of Chief Luis Velez, the Colorado Springs Police Department (CSPD) developed the PASS model as a structured way to gauge the department’s progress in infusing the community policing philosophy throughout the organization. PASS is an ambitious effort that includes a commitment to evaluate how new programs are designed and implemented, recognizing that community policing cares about means and ends. It includes qualitative evaluation strategies that go beyond traditional quantitative measures of police activity such as response time and numbers of arrests. PASS serves as a model of how police agencies can develop a comprehensive evaluation strategy to ensure that all police activities reflect the philosophy, vision, mission, and values of the department. Colorado Springs is one of a handful of jurisdictions nationwide that have designed a prototype for identifying and measuring police service standards. The PASS model is a holistic strategic planning and evaluation design that requires an annual assessment of the past year’s performance, development of a strategic plan in partnership with citizens, implementation of that strategic vision, measurement of both processes and outcomes associated with police service delivery, assessment of overall organizational performance, and demonstration of police accountability and stewardship to community. It involves conducting individual, unit, and division performance evaluations to enhance the alignment of goals and objectives throughout the police organization (which are linked to the city’s strategic plan). Evaluation expert Mora L. Fiedler assisted in developing the user-friendly PASS model, which calls for collecting both process and outcome data. Analysis of process data answers questions about how efforts are conducted, with what resources, with whom, and how the police department partners with others in the community. Outcome data are collected to measure the effectiveness of the service standards that have been defined with citizen input to the PASS process. For example, CSPD collects measurement data about response times, officer deployment, traffic, drug and vice activity, neighborhood policing, and citizen satisfaction with police services. CSPD follows various evaluation methods to collect and analyze these data. Examples will be cited throughout this guide. Source: Fiedler, M. The Colorado Springs Police Department’s police accountability and service standard (PASS) model. Colorado Springs: Management Services Division, Planning Section, Colorado Springs Police Department, 2002. Chapter Summary Evaluation is important in modern policing, particularly in light of limited budgets and the adoption of problem-oriented policing and community policing strategies. Police leaders need to know what works and what does not, both inside and outside of the department. This chapter outlined the benefits of evaluation and discussed several common myths that can prevent law enforcement agencies from pursuing evaluations. However, despite both real and imagined barriers, police departments nationwide are beginning to incorporate evaluation into their practice. Suggested areas for evaluation within policing and several contemporary examples of successful evaluations in United States police departments were presented. Evaluation in law enforcement holds tremendous potential and is not a dangerous process designed to document failure or generate bad news. Instead, it is a valuable tool that agencies can use to enhance, prioritize, and justify activities. The following chapters go further in dispelling the myths that evaluation is too demanding, technical, and expensive for police departments to implement. This guide will help demystify the process of program evaluation and will provide practical advice for starting or enhancing evaluation activities. Refer to this guide often for ideas and inspiration. Chapter II: Key Concepts in Evaluation 17 Measuring Excellence Key Concepts in Evaluation The previous chapter explained the reasons why program evaluation is pertinent to policing and how police departments across the country have used evaluation to improve program and agency performance. Grasping the fundamentals of evaluation requires learning the terminology, understanding the purposes and requirements of various models, and thinking through how the advantages and disadvantages of certain approaches apply in specific situations. This chapter introduces several key concepts, with which you should be familiar, to effectively manage evaluations in your police department. The different types of evaluations are defined, followed by an overview of common evaluation designs. Information about evaluation types and designs can help you understand how to select the best and most appropriate approach for your purposes. Different types of evaluation data that can be collected, the benefits and drawbacks of each, and the importance of data validity and reliability also are discussed. The chapter concludes by introducing the continuous feedback loop of program planning and evaluation, and outlining key steps in the evaluation process. Managing or conducting evaluations in your police department requires a basic knowledge of evaluation types, designs, and methodologies and a general understanding of what certain types of evaluation can and cannot accomplish. This is not to say that you have to be an expert, know how to conduct such evaluations yourself, or that evaluations have to be complicated or academic. Rather, conducting and managing practical evaluations successfully depends on keeping the process organized, of reasonable scope, and directly related to evaluative questions. In other words, do not try to do too much and keep things straightforward and realistic. This section will educate you about the core concepts and strategies of evaluative inquiry. The Different Types of Evaluation Different types of evaluation will be more or less applicable, depending on your evaluation questions and needs. The type of evaluation you are going to manage or conduct will guide your design and the data you need to collect. The first step in deciding which type of evaluation is best suited for your needs is to ask yourself what it is you want to know as a result of the evaluation. Generally, you can ask one of these four questions: 1. What is the nature and extent of the problem of interest? 2. What is my program doing? 3. Is my program achieving its goals and objectives for program participants? 4. Are our efforts effecting change in the population in which the problem initially documented? The question you choose will direct you to data-collection methods and sources that can determine the answer. There are four main types of evaluative inquiry: needs assessment, process evaluation, outcome evaluation, and impact evaluation. The first is associated with assessing the nature and extent of the problems that your community (or geographic area or jurisdiction) is experiencing. The second focuses on improving a program and its activities, and the latter two are linked more closely with proving that a program works (or achieves the desired and anticipated results). Following are brief descriptions of the four kinds of evaluation as well as a cost analysis. Each is illustrated by an example that shows how these kinds of evaluations could be applied to a new police department initiative aimed at domestic violence. We chose a domestic violence example because this is a complex and often hidden crime, and one that demands a sophisticated and comprehensive response. Showing how evaluations could be constructed to deal with these challenges highlights the options that can be applied to a broad spectrum of situations. Needs Assessment This type of evaluation helps an agency, organization, or coalition determine the nature and extent of specific problems in the community and how a problem is perceived among diverse groups. A careful, thorough needs assessment is the road map for change that provides the basis for a strategic plan to address specific problems in the community. Needs assessment includes reviewing local archival data to establish baseline measures of problem behaviors (e.g., break- ins, vandalism, violent crime) and comparing those data to state or national data to determine the extent to which a community’s problems are above or below state or national levels. Data from surrounding communities also can be used for comparison. Needs assessments also should include data from surveys and questionnaires and key informant interviews. The goal is to prioritize those behaviors that are most problematic in your community and make them the focus of your efforts. Examples of questions addressed in a needs assessment are shown in Exhibit II-1. Domestic Violence Needs Assessment A well-designed and comprehensive needs assessment that focuses on domestic violence in your jurisdiction could include gathering the following information: Determine the extent of the problem. You can use official police department data (calls for service, arrests) to establish a baseline to monitor over time. Also work with prosecutors, courts, and corrections to gather data on how many cases go to trial, average sentences, and number of repeat offenders. Although only a fraction of domestic violence incidents are reported to police, a thorough needs assessment should explore other strategies to get a truer picture of the problem. Such strategies might include gathering data from focus groups, surveys, and interviews of beat cops who are closest to the community, professionals and volunteers who work with victims at community hot lines, personnel at domestic violence shelters and hospital emergency rooms, and those who work with batterers. Strategies also might include targeted outreach to professional women’s groups because women from middle- and upper-income families often are more reluctant to report problems to police. Other data that can aid in understanding the extent of the problem would be total deaths and injuries, hospital costs, and workdays lost that are attributable to domestic violence. Compare rates to other jurisdictions. In addition to comparing your data to state and local statistics, you can compare them to nearby jurisdictions. Exhibit II-1 Examples of Questions Addressed in a Needs Assessment What is the nature and extent of the problem(s) in our community? How does our community compare with other local communities and state and national rates of the problem? According to the data, which behaviors are most problematic in our community? What resources are already in place to deal with these problems? What strategies or activities are available that have demonstrated effectiveness in changing these problem behaviors? What is the scope of our efforts to work on this problem? end exhibit II-1 Assess trends over time. Do the data show whether the problem is getting better or worse? What factors could be influencing those changes over time? Explore the underlying dynamics. Domestic violence advocates might have demographic data about victims and offenders, such as age, education, income, cultural and language barriers, and geographic incidence. You may also want to gather data on the prevalence of substance abuse, including alcohol and illicit drugs. In addition to statistics, interviews with experts will increase understanding of prevailing theories, such as how the cycle of power and control results in eruptions of domestic violence and why victims are often so reluctant to leave. Individual focus groups with victims and offenders also could paint a better portrait of the problem of domestic violence in your community. Describe current responses. Document ongoing police and community efforts to deal with the problem, analyzing gaps, shortfalls, and obstacles to success. This step allows you to document the policies, practices, and procedures within the department that influence or hinder police response to domestic violence, and the prevailing laws that might dictate arrest policies. Identify potential partners. In looking for data and candidates to interview or involve in focus groups, you are likely to identify community members who might be willing to work with you to implement solutions. Some to approach are domestic violence shelters, women’s groups, schools, the faith community, hospitals, physicians and nurses groups, and the business community (who suffer economic consequences associated with lost employee productivity of domestic violence victims). Gather information about opportunities to make a positive difference. While a needs assessment is not designed to explore solutions, nevertheless it makes sense to provide a mechanism for capturing good ideas as you proceed. To help you organize a needs assessment in your department, see Appendix A for a Sample Needs Assessment Action Plan Template. Process Evaluation This type of evaluation answers the question, “What does my program look like?” Data about the activities and tasks that characterize your program or strategy are collected primarily through observations of program activities, document review (e.g., meeting minutes, call logs, attendance records), and interviews with program staff, program participants, and other relevant stakeholders. See Exhibit II-2 for some questions that might be included in a process evaluation. Domestic Violence Process Evaluation Imagine that one of the top findings from the needs assessment outlined above was that many domestic violence victims said they did not know where and how to apply for Personal Protection Orders (PPO) or they felt too intimidated by the system to ask for help. The focus groups uncovered victims who did not call police because past experience or community gossip made them worry about the risk of dual arrest. Domestic violence victims, particularly those with children, hesitate to reach out to police if they fear they could be removed from the home. The police department decided to launch the first phase of an initiative to improve its response to domestic violence, including a public awareness and assistance campaign to educate victims about PPOs and make it easier for them to apply. The plan included developing brochures and videos about the benefits of PPOs and partnering with emergency rooms, schools, Neighborhood Watch, and local women’s groups to spread the word. The department also would recruit and train volunteers to help domestic violence victims complete the paperwork, with volunteers working from various community locations so victims being stalked would not be easily identified as seeking a court order. Exhibit II-2 Examples of Questions Addressed in a Process Evaluation What are the critical components/activities of this project (both explicit and implicit)? What aspects of the implementation process are facilitating success or acting as stumbling blocks for the project? To what extent does the project look and act like the one originally planned? Are the differences between planned and actual implementation based on what made sense for the stakeholders and goals of the project? What project strengths can be built upon to improve the program? Where are the gaps in services/program activities, and how can the program be improved to meet those unmet needs? Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998. end exhibit II-2 To address concerns about police response, the department elected to develop and deliver specialized training to dispatchers and line-level officers on how to respond appropriately to domestic violence calls. While complaint records did not show response to be a serious problem, the chief and training staff felt the department would benefit from training in how to encourage victims to come forward and to help officers avoid overuse of dual arrest. Through observation, interviews, activity forms, and surveys, the process evaluation would address questions about the three main components of the program: 1. Public Awareness Campaign Process Evaluation. Were the brochures and videos developed, tested, revised, and distributed in sufficient numbers as promised to the partnering groups? Did the materials address language barriers and cultural concerns? Did the partnering organizations participate as promised? How many brochures were distributed within the time frame and to whom? How many times were the videos shown and to whom? What worked, what did not, and what could be improved? Were there unexpected obstacles or concerns and how were they addressed? Did the initiative identify other materials (posters, articles in church newsletters, public service announcements) that should be considered? 2. Personalized Assistance Process Evaluation. Were sufficient numbers of volunteers recruited and trained? Was the training adequate (by what criteria and according to whom)? Did schedules and locations require adjustment? How were those decisions made? How many people were served during the evaluation period? Where did participants learn about the program? Did they receive a brochure or view the video, or both? If not, how did they find out about the effort? What seemed to work well and what did not? 3. Police Training Evaluation. Was an appropriate training curriculum developed and delivered? How many dispatchers and police officers were trained? Were participants asked to evaluate the training and, if so, what were the results? How often did dispatchers and officers have the opportunity to apply what they learned in the target area? What did police officers do differently during calls for service? Additional questions to be answered for all program components are: “Were all of the activities accomplished in accordance with the proposed timetable and budget?” “If not, why not?” “Also, have issues or concerns surfaced that must be addressed before the program is expanded or replicated?” The process evaluation should gather all relevant products or written records— copies of the brochures and videos, curriculum and support materials used to train volunteers and the police, meeting minutes, calls for service records, and officer activity forms—to document how activities are implemented and help you facilitate program revisions and replication. Outcome Evaluation This type of evaluation addresses the questions, “What is my program accomplishing in the short term?” and “Am I meeting my objectives?” Data to measure where you are in relation to program goals and objectives can be collected in many ways, but common methods include testing, surveying, interviewing, and observing participants. Questions in an outcome evaluation might include those listed in Exhibit II-3. Domestic Violence Outcome Evaluation For our domestic violence example, three major outcomes are to be evaluated: (1) Public Awareness – Did the brochure and video outreach succeed in raising public awareness about PPOs and how to apply for one? (2) PPO Assistance – Did this initiative increase the number of domestic violence victims seeking a PPO? 3) Police Training – Did the training improve police response to domestic violence calls? Public Awareness and PPO Assistance Outcome Evaluation. The success of the public awareness and PPO assistance hinges on how many more people applied for PPOs as a result. We already know the number of people who received brochures and saw the video, but the marker for success is whether that prompted the desired action—domestic violence victims securing a PPO. Exhibit II-3 Examples of Questions Addressed in an Outcome Evaluation What effect is the project having on its stakeholders or participants (e.g., change in knowledge, attitudes, or behavior)? What unexpected outcomes, if any, have resulted from the program? What can be modified to make the program more effective? Is there any evidence showing that funders should continue to support this program? Source: National Center for the Advancement of Prevention. Achieving outcomes: A practitioner’s guide to effective prevention. Washington, D.C.: Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, and the Center for Substance Abuse Prevention, 2002. W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998. end exhibit II-3 Determining program outcomes in this case requires establishing a baseline for the number of PPOs that people applied for prior to the intervention and comparing that figure to the number requested during an equivalent time period after public awareness and PPO assistance activities were launched. Also important is identifying other external factors that could influence the outcome and consider these when interpreting your findings. For example, since economic stress can trigger domestic violence, if there were massive layoffs, plant closings, or if the local economy experienced a boom during your study’s time frame, you would want to note this when interpreting your findings. You would not be expected to introduce these factors into your measurement model, but simply to think about the extent to which they may have influenced the outcomes you are seeing. Also find out whether there might be new prevention activities or programs that might affect overall rates of domestic violence in the community. Perhaps the number of PPO applications increased 35 percent during the first 6 months of the new initiative, compared to the same period the previous year, and there were no other significant external events that appeared to account for the increase. An additional challenge is to determine whether the increase is a modest or great success and, as we will see when we examine cost effectiveness, whether the benefits justify the cost. - Police Training Outcome Evaluation. Evaluating the outcome of the police training is more challenging. The training was developed to address concerns raised in the victim focus groups; however, the extent of the problem was never quantified, giving no statistical baseline for easy comparison. In fact, the decision to train police was not made because problems with dual arrest were verified, but because of a notion that a new training component would make sense. To establish a baseline, the department might survey domestic violence victims with PPOs who placed a call for service before the training took place and compare the results to those from a similar group who placed calls after the training. If this approach is cost-prohibitive, the evaluation simply might survey victims after the training has taken place to determine if perceptions about police response improve over time. Ideally, the survey would be administered face-to-face by trained interviewers, including those who speak the language of the person being interviewed, and repeat visits would be scheduled to increase the response rate by developing rapport and trust with the interview subject. However, the issues of cost, time, and talent must be considered and there often is a tradeoff between conducting an ideal evaluation and a more modest one that fits your budget. You must decide whether available money might be invested more wisely in delivering a Cadillac program with a serviceable, but more modest, Chevy evaluation so that the cost of the evaluation does not detract from the program itself. Impact Evaluation This type of evaluation answers the questions, “Is my program producing long-term, global changes?” and “Am I meeting my long-term goals?” Data are collected about the long- or wide-reaching impact of the program. Many people confuse outcome and impact evaluation or are not clear about the differences between the two. While there are differences in such issues as the unit of analysis, the key difference is that an outcome evaluation documents short-term or immediate outcomes, while an impact evaluation is focused on long-term, more global changes. Using our domestic violence example again, the outcome evaluation is focused on increasing the numbers of PPOs applied for by individuals. On the other hand, the impact evaluation focuses on reducing overall domestic violence in the county, which will take much longer to achieve than increasing the number of PPOs. Domestic Violence Impact Evaluation Assuming the three-component program of our example will encourage victims to secure a PPO and improve police emergency response to domestic violence calls, an impact evaluation will assess whether the new initiative had a positive impact on reducing domestic violence in the community. Sample questions that can be addressed in an impact evaluation are presented in Exhibit II-4. The simplest indicator of intervention success is a reduction in the total number of domestic violence incidents. (Success criteria also could include extending the time between repeat incidents or a reduction in the severity of individual incidents, but those outcomes will not be addressed in this example.) While the evaluation strategy appears straightforward and relatively easy to accomplish, you cannot rely on a simple comparison of the number of domestic violence incidents reported to police before and after the program is implemented. Issuing more PPOs and improving police emergency response by themselves might not be enough to make a dent in this chronic problem. Success also depends on others, such as prosecutors and courts, having an impact, too. Also, although only a fraction of domestic violence incidents are reported to police, the public awareness component of the new initiative might bring previously hidden incidents to the attention of the police, as victims become empowered to seek help and as concerned neighbors recognize the importance of reporting suspected crimes. In fact, the number of reported domestic violence incidents actually could rise if the program succeeds in raising public awareness. The challenge is in how to pinpoint any impact that this focused but relatively limited police initiative might have on a complex, chronic, and entrenched problem like domestic violence. While the evaluation strategy still would include gathering data on the number of incidents reported to police before and after intervention, more must be done to analyze what those numbers really mean. Incident data could be supplemented with surveys or interviews of professionals who work with victims (e.g., domestic violence shelter workers, emergency room nurses) to learn whether they witness fewer suspicious incidents. Combining those findings with feedback from victims through follow-up interviews or periodic repetition of victim focus groups, will supplement statistical data with relevant qualitative case studies. A combination of skill, creativity, and experience is required to craft powerful impact evaluation strategies. If necessary, leave the more difficult strategies to professional researchers or at least include them in the evaluation team. Cost-Effectiveness Evaluation and Cost-Benefit Analysis This type of evaluation, which can be part of one of the evaluation types discussed above, answers the questions, “What does the program cost?” and/or “What do the changes caused by the program cost in relation to the changes that occur as a result of program implementation?” To answer the first question, data are collected about program activities and costs (e.g., salaries, materials, space). To answer the second question, data are collected about program outcomes or impact and costs. Cost data commonly are collected from accounting or audit records, when available. If these data are not available, you can itemize all the resources used by the program and estimate costs for each. These and other questions are listed in Exhibit II-5. While it is helpful to compare the costs of program services with costs of services otherwise received, you also can compare costs of program outcomes with costs of outcomes anticipated if there were no program. Exhibit II-4 Examples of Questions Addressed in an Impact Evaluation What effect is the project having on our long-term goals (e.g., change in the number of reported incidents or in rates associated with the problem)? What effect did the intervention activity have on components of the system in which the activity was targeted? Were there any negative outcomes? Are they the result of implementation failure or some aspect of the intervention itself? What degree of confidence is there that the outcomes can be attributed to the intervention itself? end exhibit II-4 While cost-benefit analysis has been around for a long time, especially in the environmental, technology, and physical sciences, it has been applied only very recently to social programs like the ones you are implementing. It also is a very complex analytic technique that can be difficult to generalize for small, or even large, social programs. As a result, you may find it difficult to develop a cost-benefit component for your program evaluation without soliciting input from a consultant or other outside expert who is well-schooled in cost-benefit procedures and techniques. You might be able to do some simple cost estimates and then determine whether those costs are offset by the program’s benefits (e.g., outcomes). To conduct a full cost-benefit study of your program, however, particularly a program that is large or has multiple components and target audiences, your best bet may be to identify an expert to assist in its development and implementation. Domestic Violence Cost Evaluation This evaluation examines whether the program delivered enough “bang for the buck.” The first step involves calculating costs and comparing them to outcomes and impact. Attempting to place a value on human life and human suffering is controversial; however, we must remember that relatively modest investments in effective programs can have a profound economic as well as humanitarian benefit to the community. Research shows that real costs are associated with domestic violence, including hospital and treatment costs and lost wages for the victim, lost productivity experienced by employers, and costs to taxpayers for dealing with batterers. Exhibit II-5 Examples of Questions Addressed in a Cost-Benefit Evaluation What does the program cost? What are the cost savings associated with program implementation (e.g., the return on investment)? Do the costs outweigh the benefits? Do the benefits outweigh the costs? Will the cost savings increase with ongoing implementation or will they level off? Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998. end exhibit II-5 In 1996, the National Institute of Justice attempted to quantify the costs of crime victimization. It calculated, for example, that just one murder cost society $1.03 million in tangible costs and $1.91 million in intangible costs. Cost-benefit analysis is more common in business where the focus is on profits but when the goal is achieving justice, such tools also can help decision-making about investing scarce resources. Discussion These brief examples underscore how crafting a comprehensive evaluation strategy for any police program or intervention requires careful thought and planning. Each type of evaluation can inform any others you might conduct—baselines identified in the needs assessment can provide the basis for comparison in process and outcome evaluations, and the findings in outcome evaluations can enrich the analysis of impact evaluations. Moreover, all evaluations can illuminate opportunities and obstacles and identify potential new partners. A summary of the evaluation types discussed above is in Exhibit II-6. Evaluation Designs The type of evaluation you select, based on the questions you want answered about your program, will determine what evaluation design you will use—what you will measure and how often or when you will measure it. The evaluation design is all-important because it affects what you are able to know about your program and its impact. As we introduce some of the most common evaluation designs, remember that some will be more feasible than others in the policing context. In fact, program evaluation has its roots in the fields of medicine (curing disease) and psychology (changing behavior), in which “treatments” are applied to groups of individuals, allowing researchers to identify statistically valid differences in outcomes. Evaluating police treatments (interventions) can mean adapting evaluation techniques to identify whether a program has had an impact on a geographic area (reducing drug dealing in a target neighborhood) or a problem (reducing physical disorder by organizing a cleanup) rather than on individuals within a group. Consequently, your internal evaluations are most likely to use post-test- only designs and least likely to use quasi-experimental or experimental designs. Nevertheless, in this section we have included information on all designs to offer Exhibit II-6 Overview of Evaluation Types table Table column headers: Evaluation Type, Process, Outcome, Impact, Cost Questions Answered Process - What is the program doing? Outcome - Is the program achieving its goals and objectives? Impact - Is the program effecting change? Cost - What does the program cost? What do the changes caused by the program cost? Sample Variables Process- Number of brochures distributed to ERs and schools Number of volunteers recruited and trained; number of police officers trained Number and content of training session topics Training style Outcome - Increase in the number of applications for PPOs Increase in public awareness about how to apply for a PPO Improved police response to domestic violence calls Impact - Decrease in the number of domestic violence incidents Increase in time elapsed between repeat incidents Cost - Cost per response Cost per PPO filed end exhibit II-6 you a broader understanding of the benefits and limitations of each. This information will help you appreciate what you can and cannot learn about your programs and activities, depending on which design strategy you follow. Five common evaluation designs—post-test only, pre- and post-test, pre- post- post-test, quasi-experimental, and experimental—are described below and presented in order of complexity and expertise required, from the simplest to the most rigorous and elaborate (Shadish, et al., 2002). Classic evaluation design tables are read from left to right, first noting how many groups are involved in the research and continuing by noting the order in which measurements are taken and treatments applied. Keep in mind that in program evaluation, a pre-test or a post-test refers to a measurement (either through an interview, survey, or other data-collection method) used to document change in knowledge, attitude, or behavior. For the post-test-only evaluation design, data are collected only once from the program or from people who live in the target area. Using this design within a program delivered to the target group, participants are measured after they have gone through the program. Post-test-only measures are the least helpful for evaluating the impact of a program because there often is no valid baseline to which the post-test results can be compared. To assess change, you must be able to compare post-program data with a baseline measure. Although this is the weakest form of evaluation design, it can be used when it is important (or feasible) to know if participants have reached an identified outcome, rather than to measure degree of change. Also, if you have limited resources (included limited access to program participants), this could be the best or only option. For example, consider a grant that funds a multi-agency federal, state, and local task force to collaborate in a yearlong effort to take illegal guns off the street. At the end of that year, the task force might conduct a post-test-only evaluation that shows total number of guns seized, with breakdowns about kinds of weapons and the places where they were seized. Such an evaluation also would include data on the number of cases investigated, number of arrests made, and a profile of a typical offender. Post-Test Only Design table Group - Treatment Group Pre-measure - null Treatment - X Post-measure - O X = program intervention is applied O = measurement is taken The pre- and post-test design allows you to measure an individual or groups of individuals at two different times—before and after the intervention—enabling you to assess change by comparing the two measurements. This evaluation design has the benefit of being relatively easy to implement because you simply administer the same measure twice—before and after the intervention. You must carefully determine when to take the post-program measure, making sure to allow enough time for your program to have an effect. While the pre- and post-test design is an improvement over the post-test-only design, it is not perfect. Even if your evaluation reveals a change in what was measured, you cannot be absolutely sure that your program is responsible for that change (i.e., you cannot determine causality). Many other factors that were not accounted for in your design could be at play, and it might be one of those factors, not your program, that had an impact. Without a well-defined program and sample, this type of evaluation design cannot be accomplished. To illustrate, consider the issues involved in evaluating a new police effort to reduce prostitution in a specific neighborhood where residents have complained that the problem is escalating out of control. A pre- and post-test design requires establishing baselines that will be measured after the intervention so, in this case, indicators likely would include the number of citizen complaints and number of arrests made, with the expectation that success would be reflected in a decline in complaints and a rise in arrests. As mentioned above, the challenge is making sure there are not other factors operating in the community that influence results. In other words, how do you know that it was your efforts, and your efforts alone, that caused any changes that are measured at post-test? Maybe there was a dramatic decline in complaints because the people who run prostitution in the neighborhood began intimidating residents who then feared talking to police. Or perhaps local public health officials received a grant to launch a comprehensive effort to reduce prostitution through an outreach that offers substance abuse treatment and job training. These circumstances can be addressed only through experimental and, to a lesser degree, quasi-experimental designs (described below). Pre- and Post- Test Only Design table Group - Treatment Group Pre-measure - O Treatment - X Post-measure - O X = program intervention is applied O = measurement is taken Among the benefits of a community policing approach to chronic problems like prostitution is not only that police will be more likely to find out about other community-based efforts, but also that collaborating to address the problem offers the best chance of success. By partnering with other agencies such as public health, the police also might enjoy benefits from being able to tap into their evaluation capabilities, and this may result in a stronger evaluation design and a greater understanding of whether police efforts were responsible for any changes measured at post-test. Pre- Post- Post-Test Design table Group - Treatment Group Pre-measure - O Treatment - X Post-measure - O O X = program intervention is applied O = measurement is taken The pre- post- post-test design provides measures at three distinct points in time. A pre-test measure is taken before any exposure to the program. The first post-test measure is taken immediately at the end of treatment, and the second (or follow- up) post-test measure is taken some time (e.g., 3, 6, or 12 months) after program completion. The advantage of this design is that it allows you to understand if the program treatment had any sustained impact or to know if program participants retained knowledge after a period of time. This design has the same weaknesses as the pre-post design in its ability to determine causality (i.e., to show conclusively that the program treatment is responsible for any changes measured). In addition, finding people to participate in the second post-test can be difficult and costly. Furthermore, people who are measured a third time could become bored or annoyed with the same measurement instrument, thereby affecting their responses. However, if we think of the prostitution example again, it might be valuable to examine citizen complaint and prostitution arrest data again 6 months after the intervention ends to see whether the department should renew its effort before the problem escalates again. This approach represents a practical way for police departments to achieve more longitudinal data collection and evaluation. The quasi-experimental design comes one step closer to achieving the confidence level whereby you know that it was your program that caused change. In this design, you compare your target (or treatment) group (the group, neighborhood, or community that is the target of the policing effort or program) with another similar group that did not receive the program (i.e., a comparison group). Using quasi-experimental design, you would pre-test both groups before the program begins then deliver the program to only one group. You would administer a post-test to both groups, analyze the changes, and compare the results from both groups. The challenge of the quasi-experimental design is to find two or more groups that are sufficiently similar in demographics (i.e., gender, ethnicity, socioeconomic status, education) and situation (i.e., both groups are adolescent boys at risk for juvenile delinquency) to be truly comparable. Your ability to say for certain that those who went through the program were the same as those who did not is limited, affecting your ability to say your program is responsible for differences that might exist between the groups after program intervention. The more alike the two groups are, the more confidence you can have that your program was responsible for positive outcomes. The quasi-experimental design is frequently the best option for conducting outcome evaluations in the social services context, as it is often the only practical and ethical way to study two or more groups who vary in the programming they receive, because it does not rely on random assignment (see the experimental design example that follows). Additionally, to maintain and study a control group can be costly, which is even more significant because that group does not benefit from program treatment and already limited resources are diverted from programming. The quasi-experimental design averts these problems by using groups that already exist or that are not assigned randomly. An example of a comparison group is in a school setting where one class that participates in a program is compared to another class that does not participate. Quasi-Experimental Design table Group - Treatment Group Pre-measure - O Treatment - X Post-measure - O Group - Comparison Group Pre-measure - O Treatment - null Post-measure - O X = program intervention is applied O = measurement is taken This kind of research design is used more commonly by criminal justice researchers, often in concert with police, rather than by police alone. A jurisdiction moving toward a preferred or mandatory arrest (pro-arrest) policy in domestic violence situations could conceivably apply the policy in one area or sector and not another and then compare results. This would be an example of a quasi-experiment. Obviously, there are challenges involved in training and supervision to ensure that officers are applying the policies appropriately. Also of concern is that such policies require the support of the county prosecutor or district attorney’s office, since the point of pro-arrest polices is to send the message that domestic violence will be prosecuted with or without the full participation of the victim. It also goes without saying that, if pro-arrest policies do indeed save lives and reduce injuries, there are ethical concerns about whether it is or is not appropriate to withhold the intervention to the victims who happen to live in the area where the new policies do not apply. This example qualifies as a quasi-experimental design because there could be differences between the geographic areas being compared that could affect the results of the evaluation. For example, arrest might carry more of a stigma in one area than another, so the impact of a pro-arrest strategy might show greater promise in an area where arrest induces shame than in an area where arrest is a more common occurrence. Experimental Design table Group - Randomly SelectedTreatment Group Pre-measure - O Treatment - X Post-measure - O Group - Randomly Selected Control Group Pre-measure - O Treatment - X Post-measure - O X = program intervention is applied O = measurement is taken In experimental design, people from the same target population are assigned randomly either to a treatment group or a control group. Random assignment means that each person in the target population has an equal chance of being selected for either group. This randomness is crucial, because it is the only way to ensure that two groups are similar at the beginning of the intervention period. With this design, as with quasi-experimental design, both groups are pre-tested, only one receives the program intervention, and then both groups are post-tested. Experimental design offers the best opportunity to know (and claim to others) that your program is responsible for changes in outcomes. It is used most often by researchers in the physical sciences because they can control their lab environments, repeat experiments, and determine causality. For obvious reasons, it is much more difficult to implement valid experimental designs in the policing context. Ethical concerns regarding experimental designs with humans often render experimental evaluation of human services unfeasible. Additionally, it might cost more to implement an experimental design and, despite the extra expense, several threats to the validity of the study would still exist. Threats to validity are presented in Exhibit II-8. Such threats are nearly impossible to control in real-life settings. If we think of the pro-arrest example above, the logistics involved in randomly assigning officers to use a pro-arrest strategy on one call and then not on the next pose serious logistical and ethical questions. We also know, however, that pro- arrest policies can result in problems with dual arrest, where frustrated officers find it easier to arrest both individuals involved in the altercation than to try to differentiate between victim and offender. Training half the force in strategies that are designed to reduce the problem of dual arrest might allow for the randomness required by the experimental design. Then, if the results prove positive, it would justify investing in training the rest of the force. To select the design that is right for your department, consider your needs and resources and pick the most rigorous design possible, given those restraints. Each type of design has its advantages and disadvantages, so be realistic about what kind of design you can implement. A post-test-only design might not produce the information you require, but quasi-experimental and experimental design studies could be too complex to perform without outside help. To manage evaluation effectively in your department, do not waste time and resources committing to a design that is beyond your capabilities or will not provide the information that you need in the end. In most cases, you should be able to find a happy medium. The different types of evaluations and their strengths and weaknesses are summarized in Exhibit II-7. Exhibit II-7 Strengths and Weaknesses of Different Types of Evaluation table Table Headers: Design, Characteristics, Advantages, Disadvantages, Required Expertises Post-test only Measures one group at one time only—after the treatment. Can document the reduction of a problem; requires access to only one group at only one time. There is often no valid baseline measure for comparison. Does not allow one to assess change from baseline. Low Pre- and Post-test Measures one group at two different times—before and after program completion Provides a baseline measure; requires access to only one group. Cannot know for certain that your program produced the change. Moderate Pre- Post- Post-test Measures a group at three different times—once before the program and twice after program completion. Enables you to determine if there are sustained effects from your program. Does not show conclusively that your program produced the change; may be difficult to follow up with everyone. Moderate Quasi-experimental Measures two groups at two different times— before and after the program; one group does not receive the treatment program. Offers a decent level of confidence that your program caused the observed change. May be difficult to find two groups that are similar enough to be compared; is more costly to have comparison groups. Moderate to High Experimental groups whose members are randomly assigned at two different times—before and after the program; one group does not receive the treatment program. Provides the highest level of confidence that your program caused the observed change. Difficult to implement in a real-life setting; there are ethical issues involved with the random withholding of a beneficial program from one group. High Different types of data Once an evaluation design has been chosen, focus on what types of data you will collect to answer your research questions and the data-gathering methods you will use. Quantitative data measure something that can be counted and assigned a number, and can be collected using several methods, including simple counts, surveys, and tests. Police departments often rely on quantitative data such as calls for service, arrests made, and citations issued. You might also take a simple count of how many people attended a neighborhood community meeting or calculate the percentage of survey respondents who are satisfied with police services. Quantitative measurement often assesses the extent and direction of change that has taken place as a result of the implementation of a strategy or program, and statistics often are used in quantitative data analysis. Quantitative measurement has several advantages. First, it yields data that can be standardized across a large number of people, can be replicated easily, is conducive to making comparisons, and can be less expensive and time-consuming to collect than qualitative data. However, quantitative data sometimes give a simplistic representation of reality and have been criticized for lack of attention to participant input. In addition, survey questions must be developed carefully to ensure they will capture the information you intend to measure. One common problem that police face is that easily quantifiable data may not be kept in a form that suits their needs. For example, imagine that a department is planning a new effort to reduce the number of commercial burglaries. You find, however, that your call for service data does not make a distinction between commercial and residential burglaries and that there is no easy way to disaggregate the data. We know instinctively that there are many more home burglaries than commercial burglaries, so even if the new program is very successful, the data might not show enough of a reduction in the total number of burglaries to verify success. Do you delay the intervention until you can persuade dispatch to begin capturing burglary data in the format you need? Or do you pay someone to check the address for each burglary call to see whether it is a commercial or residential address? Qualitative data are collected through activities such as observations, interviews, focus groups, and even conversations. They describe the attributes of an object without referring to quantity and are expressed in words, not numbers. Instead of measuring the extent and direction of change or program impact, qualitative data are better suited to describing the nature of the change or impact that has taken place. Qualitative data are used frequently in process evaluation to describe how a program functions. In outcome evaluations, qualitative data are useful for providing context to and humanizing the numbers. Qualitative methods capture nuances and insights that statistical analysis of survey instruments cannot measure. Qualitative data provide context and can be more interesting and informative for a lay audience than a technical report filled with numbers and statistics. However, qualitative data can be overwhelming and cumbersome to collect and analyze, and because data collection generally is more time- intensive, it is more expensive. Moreover, qualitative data lend themselves less to comparison and generalization than quantitative data. In 2003, the Michigan Bureau of Juvenile Justice launched its yearlong Building Restorative Communities (BRC) initiative in five counties, designed to help build the capacity to identify and implement “evidence-based, data-driven, and outcome-focused” interventions to reduce juvenile crime. The core group of police, prosecutors, courts, and juvenile justice officials who work with at-risk youth helped mobilize a cross section of community groups to partner with them on developing a comprehensive 5-year plan. A grant allowed each county to hire a coordinator and a part-time evaluator for the project. One of the six work groups designated for the project was tasked with hosting separate focus groups of juvenile offenders in detention and victims of juvenile crime. Focus group facilitators were given a script with questions for each group, and a list of themes they could use to sort participant comments into useful categories. One question for the juveniles was, “What could have helped you stay out of trouble?” The findings would assist the steering committee work group to prioritize its final recommendations. This project confirmed that qualitative research produces significant amounts of important information but that it is not as easy to convey (or convey quickly). In Monroe County, with only 15 participants in the juvenile offender focus group and two participants in the victim focus group, the final reports totaled more than 15 pages. Both types of data have advantages and disadvantages, therefore, it is now widely accepted that using multiple methods and mixing the methods is the best strategy for gaining the most complete understanding of a program or strategy and its effects. In fact, the use of mixed methods increasingly is considered a prerequisite for quality data. A discussion of the different methods that can be used to collect quantitative and qualitative data is presented in Chapter III: Conducting an Internal Evaluation. Whichever data instruments you choose, you must consider the ability or suitability of the instrument to capture the most valid and reliable data possible. The validity of data collected hinges on whether the data collection instrument measures what it intends or claims to measure. For example, crime statistics are not a valid measure of citizen satisfaction with police services. The reliability of data collected is a term used to describe an instrument’s ability to record consistently what is being studied. For an instrument to be reliable, it must yield the same (or similar) results even when administered in different places or at different times. Instruments must be both valid and reliable for you to claim that you have collected quality data. Exhibit II-8 summarizes several common threats to the validity of evaluation results. Exhibit II-8 Common Threats to the Validity of Evaluation Results History: Any event that happens between the time of your first measurement (pre-test) and your second (post-test) that can affect the measurement. Maturation: The normal processes of development that occur over time could affect your outcomes, independent of your intervention. For example, as children grow and mature, they are likely to develop longer attention spans, according to available research on child development. This naturally occurring phase of development may occur while the child is enrolled in your program, making it difficult to separate program effects from normal developmental growth. Testing: These are the effects that test taking have on the study results. Taking a pre-test may influence the behavior of your subjects as much or more than your intervention does. You cannot be sure that the effects you see are caused by your intervention and not by the administration of a pre-test. Instrumentation: Any flaws in your measurement device that can skew your results. For example, if you collect information from an observer, he or she may be less attentive to collecting data on one occasion than on another. Statistical Regression: A general tendency for extreme scores on any measure to move toward a less extreme score at a different time. If you measure something at the lowest possible amount, there is no way for it to change except to increase. Statistical regression occurs when you select study participants based on their extreme scores (e.g., if you decide to study only very depressed adolescents). When you remeasure, there is a tendency for these extreme scores to move back toward the overall group average regardless of the effects of any intervention provided. Selection Bias: Any determination, except random placement, by which you place some subjects in a treatment group and other subjects in a control group is a type of selection bias. Subjects assigned to groups in any way other than random placement opens the possibility that the groups differ in important ways that can affect the results of your study. For example, students who volunteer to attend a tutoring session may be more academically motivated than students who do not volunteer to attend this group. Different levels of motivation may affect these two groups as much as or more than the tutoring groups did. Random assignment to groups is the best way to protect against the effects of selection bias. Source: Mertinko, E., L.C. Novotney, T. Baker, T., and J. Lange. Evaluating your program: A beginner’s self-evaluation workbook for mentoring programs. Washington, D.C.: Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, and the United States Department of Justice, 2002. end exhibit II-8 The Continuous Feedback Loop of Program Planning and Evaluation and the Key Steps in the Evaluation process In managing or conducting an evaluation, it is critical to know where evaluation fits into the larger scheme of implementing police programs and strategies. Ideally, programs, projects, initiatives, and strategies are planned and carried out with a continuous feedback loop, of which evaluation is an integral part. Exhibit II-9 illustrates how evaluation is part of a continuous cycle of program planning, implementation, and refinement, and emphasizes the importance of connecting evaluation with program improvement for maximum results. Exhibit II-9 The Program Planning and Implementation Cycle Evaluate the program/initiative Compare and analyze results against baseline and/or objectives Specify (Plan) what is to be collected and how Collect information with appropriate tools end exhibi II-9 The specific steps in evaluation also can be detailed, as shown in Exhibit II-10. The evaluation cycle itself is composed of several steps, which can be followed without much evaluation expertise; however, there may be times when you need outside assistance in developing or implementing an evaluation step, depending on the experience of your team and the complexity of the evaluation. Whether you are conducting an internal evaluation or managing an outside one, you will need some specific information about all the evaluation cycle components: evaluation planning, data collection, data analysis, and using and disseminating evaluation findings. Chapters III and IV in this guide provide more details about these components. Exhibit II-11 highlights the integration of the planning and evaluation cycles. Exhibit II-10 The Evaluation Cycle Plan the program/initiative Refine the program based on evaluation results Implement the program/initiative Evaluate the program/initiative end exhibit II-10 Exhibit II-11: Integration of the Planning and Evaluation Cycles Plan the program/initiative Refine the program based on evaluation results Implement the program/initiative Evaluate the program/initiative Compare analyze results against baseline and/or objectives Specify what is to be collected and how Collect information with appropriate tools end exhibit II-11 Chapter Summary Chapter II presented an overview of evaluation types and designs and described evaluation’s place in the cycle of program implementation and improvement. This chapter also covered the distinction between process, outcome, and impact evaluation and the pros and cons of several evaluation designs. Understanding these key evaluation concepts and the steps in the evaluation process will prepare you to conduct and manage valuable evaluation research in your police department. Evaluation is a powerful tool and the process is logical and relatively easy to grasp, particularly when you follow the logic model approach and other useful tips outlined in the next chapter. The rest of this guide concentrates on particular steps in the evaluation cycle, including specifics on how to conduct internal evaluations and manage them successfully so that results are disseminated and used effectively for program refinement and improvement. Chapter III provides details and guidelines, including examples and worksheets, for conducting an internal evaluation, while Chapter IV offers practical tools and instructions to help you manage evaluative research in your police agency. Chapter III: Conducting an Internal Evaluation Conducting an Internal Evaluation An internal evaluation is one in which the police agency relies primarily on people within the department to conduct an evaluation of policing efforts and to disseminate results. Internal evaluations are designed to help police agencies learn more about the operation, performance, and perceptions of all or part of their departments. Evaluations can vary from a process evaluation to determine whether a new call management system has been implemented as directed, to an outcome evaluation of a narcotics task force, to a community satisfaction survey as one indicator of the long-term impact of community policing on attitudes about police. The following are some benefits of an internal evaluation: The evaluation will reflect staff knowledge about the program. The evaluation may be more successful in getting support from police administration and other staff. The evaluation may be less expensive than other evaluation options. While an internal evaluation might meet the needs of the program staff and administrators better, the following are some potential criticisms of internal evaluations: The evaluation can be perceived as less objective and less credible. The evaluation might not be adequate for answering the evaluation questions. Perhaps the biggest misunderstanding about internal evaluation is that it must follow an academic research model, which focuses on testing hypotheses. Most law enforcement agencies should focus instead on answering practical program and policy questions. While academic research requires a more controlled approach with comparison and control groups, this is seldom realistic for a law enforcement agency operating in a real community with many uncontrolled variables and where using a control group is not feasible or is unwise. For an internal evaluation designed to improve programs and practices, you might not need such control or sophisticated statistical analyses. The methodology for practical research should be based on identifying program or effort that is being evaluated, the type of data that can be collected, and the availability of resources for data collection and analysis. Some of the primary differences in purpose, method, and statistics between academic research and internal evaluation are presented in Exhibit III-1 If evaluation results are intended mainly for internal use, the purpose, method, and statistics of academic research often are too costly for most law enforcement agencies or are beyond their scope. Nevertheless, employing evaluation procedures that are explicit, formal, and justifiable is important so that police departments can have confidence in the validity of their findings. Law enforcement agencies, therefore, should do the best evaluation they can with available resources, carefully noting the limitations of findings. Ultimately, the standard by which your evaluation should be judged is whether the collected information was used to make decisions or improve the program (Loe, 2001). If you have decided, based on your evaluation needs and resources, that conducting an internal evaluation is your next step, the rest of this chapter provides an overview of how to conduct an evaluation, including developing an evaluation plan, determining data-collection strategies, and conducting data analysis. Several tips and practical worksheets are included throughout the chapter and in the appendices. The resource list in Appendix L directs you to more evaluation references. Exhibit III-1 Difference Between Academic Research and Internal Evaluation table Table Headers: Academic Research, Internal Evaluation Purpose Test hypotheses; scientifically test a program model; advance the overall literature and field of law enforcement Improve program and practice; address the needs of a single law enforcement agency or a group of agencies Method Controlled environment Context-sensitive Statistics Sophisticated Simple Source: The University of Texas-Houston Health Science Center School of Public Health, and The Texas Department of Health. Practical evaluation of public health programs. Atlanta, Georgia: The Centers for Disease Control and Prevention and The Association of Schools of Public Health Cooperative Agreement, 1998. end exhibit III-1 How We Did It: Colorado Springs Police Department The Colorado Springs Police Department’s (CSPD) Police Accountability and Service Standards (PASS) model demonstrates how internal evaluations can produce valuable information to both the law enforcement agency and the community. CSPD uses internal data collection and analysis of its operations and programs to improve policing efforts and ensure that they are meeting community expectations. Steps in the Evaluation Process The four main steps in the evaluation process were introduced in Chapter II: SPECIFY—evaluation planning, COLLECT—data collection, COMPARE— data analysis, and COMMUNICATING and using the results. The first three steps are the focus of this chapter. The final step is a management task and is discussed in greater depth in Chapter IV. Step 1: SPECIFY—Evaluation Planning. An evaluation plan provides a framework that will help you specify what to evaluate and how. A clear evaluation plan facilitates not only evaluation implementation but also its management. Evaluation planning steps that will be discussed in this chapter include the following: assessing whether your program is ready to be evaluated, assessing stakeholders’ commitment to evaluation and planning for their involvement, understanding and defining your program goals and objectives, developing a logic model, determining your evaluation questions, and selecting an evaluation design based on the careful planning you have done. Step 2: COLLECT—Data Collection. Your evaluation data will be collected through surveys, interviews, focus groups, document review, or other methods, depending on what has been specified in your evaluation plan. After determining what data already exist and will be useful, you might still need to develop your own data- collection instruments. Tips for creating and implementing surveys, interviews, and focus groups are included in this chapter. Step 3: COMPARE—Data Analysis. The data you collect will be analyzed according to your evaluation plan and the data-collection instruments that have been used. The purpose of analysis is to reduce, categorize, and summarize your data in a way that helps you and others understand what is happening with your policing programs and activities and/or what is changing as a result of those efforts. Evaluation data often are analyzed using simple counts, statistics, and content analysis. This chapter gives a brief overview of each. Step 1: SPECIFY—Evaluation Planning Preparing for any evaluation requires an investment of time and thought. Attention to planning each evaluation step will increase the likelihood of producing useful results. Evaluation planning helps you carry out activities efficiently and effectively and allows someone outside the organization to understand what and how you are evaluating. Many police agencies make the mistake of delaying their thinking about evaluation as long as possible, often until after the program is underway. Treating evaluation as an afterthought not only jeopardizes its quality, but also robs the department of the benefits that evaluation can deliver to planning and implementation. Thinking through the evaluation plan while the program or intervention is being developed helps clarify what should be done and how. Identifying the various measurements that will be used also focuses participants on what it will take to make the initiative a success. Delaying evaluation planning also might make it more difficult to gather the data you need or to put it in the format that best suits your purposes than if you had explored various options earlier. Perhaps the biggest advantage of early evaluation planning is that it can allow police to make course corrections along the way, increasing the odds of success. Police executives who worry that evaluation could trumpet bad news should understand that it is in their interest to engage in evaluation planning in advance, incorporating opportunities to monitor progress so that appropriate changes and improvements can be made quickly. Evaluability Assessment Not all programs can or should be evaluated because some are too small, too short, or are ill-conceived. For others, it might be too early in their development to conduct an outcome or impact evaluation. As part of planning, it can be helpful as a first step to identify whether your program is ready to be evaluated. To do so, you can conduct an informal evaluability assessment, which lets you verify whether the program has clearly specified activities and well-articulated and measurable goals and objectives that lend themselves to identifying data that can be collected. This process includes reviewing the program history, design, and operation, perhaps by observing the program in action; determining the program’s capacity for data collection; and assessing the likelihood the program will meet its goals and objectives based on implemented activities. While the concept of an evaluability assessment might be unfamiliar, this type of preassessment increases the likelihood that a full-blown evaluation will provide timely, relevant, and responsive findings (Juvenile Justice Evaluation Center, 2003). Poorly planned or implemented programs should not be evaluated because findings will be inconclusive at best and misleading at worst. Therefore, it is often in a program’s best interest to invest the resources necessary for either a formal or informal evaluability assessment (Juvenile Justice Evaluation Center, 2003). If your evaluability assessment results show a program is not ready for an evaluation, an outside evaluator might be able to work with the program manager and staff to bring the program to the appropriate stage. In addition to helping determine whether to conduct an evaluation, data collected from an evaluability assessment can describe the sequence of critical elements that influence program design, implementation, and effectiveness, and they can direct you toward developing suitable measures and research designs. This often is graphically represented in a logic model, which is described in greater detail later in this chapter. Planning Stakeholder Involvement and Assessing Stakeholder Commitment Another step in evaluation planning is to identify key stakeholders and assess how committed they are to an evaluation. In addition to your police department, an evaluation stakeholder could be the agency or organization that funds the grant under which your program operates. For example, if you are receiving funds from the Department of Justice to implement a gang-prevention program, you may want to talk with your federal program manager to see if he or she would supportive an evaluation. Other important stakeholders might include local government, members of the business community, schools, and nonprofit agencies, as well as community residents who have been involved in identifying problems and finding solutions. The following questions will guide you in thinking about the level of stakeholders’ commitment and the ways in which they might facilitate your evaluation: What might be the concerns stakeholders have about conducting evaluation at this time? How might obstacles be overcome? What are the reasons that stakeholders might support an evaluation at this time? What questions do they have about the implementation or effectiveness of policing efforts? What information do the stakeholders need about your program? How will key stakeholders use information from this evaluation? What information or expertise can stakeholders contribute to your evaluation? What are stakeholders willing to contribute to an evaluation at this time? How can evaluation help make a persuasive argument for community support? Understanding stakeholders’ concerns—and what they might be most enthusiastic about— can help in planning a more effective evaluation strategy. At the same time, addressing their concerns and interests generates buy-in. Stakeholder commitment often can be increased by establishing an advisory group or evaluation team. A well-constituted advisory group, made up of the evaluator, representatives from the police agency, program participants, and/or interested citizens, can be an asset for formulating evaluation questions, providing additional evaluation resources, and being advocates and spokespeople for the dissemination of results (Center for Substance Abuse Prevention, 2003). The Evaluation Plan An evaluation plan documents a system for collecting evidence of program effectiveness and will help ensure that you are collecting the information you need in the most meaningful way. Developing your plan means defining carefully what to evaluate and how to measure success, formulating evaluation questions, selecting the appropriate evaluation design, and developing data-collection methods. During evaluation planning, you also should think through a project timeline and budget and a strategy for using and disseminating your findings. These management issues are discussed in the chapter “Managing Evaluations.” Key Steps in Evaluation Planning Determine whether the program is ready for evaluation. Assess stakeholder commitment to evaluation. Define program goals and objectives. Develop a program logic model. Determine evaluation questions. Choose an evaluation design. Plan for data collection and analysis. Determine What Is Going To Be Evaluated Although it might seem obvious, determining what is going to be evaluated could be difficult if your program was not well defined at the outset or if activities were not designed and implemented so they could achieve the stated goals or produce meaningful benefits. Before conducting an evaluation, be clear about the program’s purpose, its goals and objectives, and the activities that have been put in place to meet them. Understanding the relationship among them also might help determine whether you want to examine the whole program or just a particular component. For instance, one program component might be selected if you think it has the strongest effect on participants, or if it requires the greatest proportion of your resources and, therefore, you are particularly concerned about its effectiveness. Define Program Goals and Objectives Understanding your program’s mission, goals, and objectives will give you ideas about how success can be measured in your evaluation. Before evaluation begins, therefore, become familiar with your program’s goals and objectives and know whether they are measurable. If goals and objectives have not been articulated or written, you must work with your evaluation team and stakeholders to do so to ensure you do not undermine your results. See Exhibit III-2 for tips for developing goals and objectives. A goal is a measurable statement of the desired longer term, or global, impact of the program. Goals typically address change. For example, recalling our domestic violence example from Chapter II, a goal might have been, “By 2006, our program will reduce domestic violence in our county by 10 percent.” An objective is a specific, measurable statement of the desired immediate or direct outcomes of the program that support the accomplishment of a goal. Again, using our domestic violence example from Chapter II, an objective could be, “Our program will increase the number of personal protection orders (PPOs requested in our county by 50 percent.” Note that objectives state the desired immediate or direct outcomes of programmatic efforts; therefore, it is useful to distinguish between outcomes and outputs. Outcomes refer to the changes that result from a program or its activities (e.g., increase the number of PPOs requested). Outputs are units of service that are provided by the program (e.g., the number of police officers who went through domestic violence/PPO training). Program goals and objectives should be tangible and measurable. Use the exercises in Appendix B to guide you in developing program goals and objectives or in adopting those you feel accurately reflect your policing mission. Develop a Program Logic Model Determining exactly what to evaluate can be facilitated by using a logic model, which is a graphic representation of the logical relationships among program conditions, activities, outcomes (objectives), and impacts (goals). A program logic model for an evaluation plan provides explicit insight into how the program is supposed to work to achieve the desired benefits. A logic model incorporates a series of “if-then” changes that the program aims to achieve through its activities, outputs, and outcomes, and which can help you evaluate whether the program has met its goals and objectives. A logic model does the following: Requires program staff to think through goals, objectives, and strategies and to develop a realistic idea of what the program can accomplish Clarifies assumptions and relationships between program efforts and expected results Provides a useful framework for examining program outcomes and impacts Helps specify what to measure through evaluation. Logic models usually are diagrammed as a series of boxes representing the conditions (needs or problems), activities, outputs, outcomes, and impacts. Each box in the logic model will give you a reference point for comparing your program’s progress toward achieving the desired goals and objectives (note: outcomes and impacts are the measure of your objectives and goals, respectively). Obtaining stakeholder input into the development of the program logic model can be a strong foundation for planning and prioritizing evaluation activities. Exhibit III-2 Tips for Developing Goals and Objectives Be careful about defining goals too narrowly so that they appear to be outcomes. Beware of stating activities as goals or objectives. Be sure not to write compound goals and objectives. Be realistic. Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998. end exhibit III-2 A logic model can guide all types of evaluations and help you specify what needs to be collected. For process evaluations, the logic model identifies expectations for how the program should work so you can see whether your program has derailed or is on track. For outcome and impact evaluations, the logic model displays how and for whom certain services or activities are expected to create change. A logic model with law enforcement examples is provided in Exhibit III-3. The corresponding worksheet and template in Appendix C will walk you through the steps of developing a logic model pertinent to your evaluation needs. Exhibit III-3 Sample Logic Model Conditions (problems or needs) What problem or condition is your program designed to change? (e.g., need to reduce domestic violence; need to increase citizen participation) Activities What are the strategies, techniques, or types of services that comprise a program’s service? (e.g., police officer training; block watch programs) Outputs What are the direct products of program activities? (e.g., number of police officers trained; number of people who attended block watch programs) Outcomes (short-term changes tied to program objectives) What are the immediate benefits of the program activities? (e.g., increase in number of PPOs requested; increased citizen participation) Impacts (long-term changes tied to program goals) What changes have been achieved in the original conditions? (e.g., reduced domestic violence; improved relations with community) Develop Your Evaluation Questions The next step in evaluation planning is determining what you and other key stakeholders want to know about the program through your evaluation inquiry. As noted in Chapter II, there are four major types of evaluation: needs assessment; process evaluation that documents whether a program is being implemented as planned; outcome evaluation that documents short-term outcomes and often is used for ongoing program improvement; and impact evaluation, which tells you whether the program achieves longer term goals. The evaluation questions you choose will be tied to your program goals and objectives as well as the type of evaluation you plan to implement. Evaluation questions should be developed according to what you want to learn about your program. The following are some typical evaluation questions: What is the nature of the problem (e.g., delinquency, graffiti)? Where is the problem? How widespread is it? Whom does it affect? What is the program or intervention? Is the program being implemented as intended? Is the program reaching its target population? What are the barriers to implementing the program? What approaches to overcoming implementation obstacles have been effective? How much does the program intervention cost? Is the program achieving its objectives? What are the actual outcomes? Is the program cost-effective? Choose an Evaluation Design Understanding your program goals and objectives, creating your logic model, and delineating your evaluation research questions will help point you to the appropriate evaluation design. Five of the most common evaluation designs, post-test only, pre- and post-test, pre- post- post-test, quasi-experimental, and experimental, were discussed at length with specific examples from the law enforcement context in the chapter “Key Concepts in Evaluation.” Please reference the chapter to refresh your memory and to help select the design that is most appropriate for your needs and resources. The evaluation design that you choose can help direct you to the appropriate data-collection sources, which in turn will influence the analytic procedures you use to make sense of your data and answer your evaluation questions. Data collection and analyses are the focus of the remaining section in this chapter. Step 2: COLLECT—Data Collection Once you have determined your evaluation purpose and design, you can begin preparing for data collection. The data-collection methods you choose will reflect the type of evaluation you decide to conduct, the evaluation design that you have chosen, the evaluation research questions on which you plan to focus your activities, and the specific data needed to answer those questions. Since data collection can be resource-intensive (both expensive and time-consuming), plan your data collection carefully and limit it to only the data that answer the evaluation questions you have specified. Your data-collection strategies can be relatively simple and straightforward if you plan wisely. In fact, data collection can be incorporated easily into routine program activities and often is something police officers already do, perhaps without realizing it. Key Steps in Data Collection Review existing data and data sources. Develop any necessary new data- collection protocols. Obtain the necessary clearance and permission. Collect and track data. Review Existing Data and Data Sources Prior to selecting new data-collection strategies, you should first consider and review existing data and data sources. For example, traditional measures of police performance and crime, such as Uniform Crime Report (UCR) data, provide a nationwide view of crime based on statistics from law enforcement agencies throughout the country. The UCR program routinely collects data on known offenses and persons arrested by police departments for murder and non-negligent manslaughter, forcible rape, robbery, aggravated assault, burglary, larceny-theft, motor vehicle theft, and arson. These data may be appropriate to use if you are conducting a needs assessment (to determine the nature and extent of criminal activity in your jurisdiction) or an impact evaluation (to determine the extent to which your programs, strategies, and initiatives are effecting change in longer term criminal activity in your jurisdiction). However, they might not help you understand much about any single program or specific outcomes related to your targeted activities. For that, you would need to collect outcome data, which are tied to your shorter term objectives, and which typically are gathered directly from program participants or stakeholders (e.g., citizens, businesses, or police officers who go through special training) before and after implementation of your program or strategies. These outcome data will help you assess the extent to which your efforts are meeting defined objectives; impact data will show you whether you have met your longer term goals. Data collection can be one of the most resource-intensive steps in the evaluation cycle. To find and use existing data for your evaluation, we encourage you to explore other less traditional data sources that might inform an outcome evaluation, including calls for service, medical admissions (e.g., gunshots, overdoses), vandalism and graffiti costs from parks and recreation, and insurance information on stolen cars and burglaries. Develop Necessary Data-Collection Protocols Reviewing your existing data and data sources will allow you to decide what data you need to collect that you are not capturing now. Once you have done this, make a list of topics about which you need to know more. Prioritize among the items and finalize the list based on the relative importance and usefulness of the information as well as the cost and ease of collecting those specific data. Now you are ready to develop data-collection protocols you might need to help you obtain the information on that list. Quantitative and qualitative approaches to data collection are reviewed briefly in this section (also see Chapter II), along with an overview of specific data-collection methods and tips for creating surveys, planning and conducting focus groups, and developing interview guides. Quantitative Data Quantitative methods capture data that can be counted, measured, compared, or expressed in numerical terms. Surveys, questionnaires, and archival data are commonly used quantitative data sources. Archival data already have been collected by your department or by another organization, such as a government or nonprofit agency. These documents might include program records or materials such as proposals, annual or monthly progress reports, budgets, organization charts, memoranda, policies and procedures, operations handbooks, and training materials. Archival data are analyzed through document review. These data can be the simplest and cheapest kind to gather, because someone else has collected them. Additional benefits to using archival data are that the review of program documents can provide an idea of how the program functions without interrupting program staff or activities and document review requires relatively little expertise to conduct. Surveys and questionnaires involve collecting data directly from individuals. Through self-administered or face-to-face surveys, telephone or mail surveys, questionnaires, and checklists, you can ask program stakeholders directly and in a standardized way whether and how your efforts are making a difference (Salant and Dillman, 1994). Surveys and questionnaires can be especially useful in evaluation if the same set of questions is asked at the beginning of a program (i.e., a pre-test used for baseline information) and again when the program is over (i.e., a post-test to document any change over time). In law enforcement, surveys can help you collect information about criminal victimization, community views about crime, and residents’ willingness to report crime to the police. Since such data provide a measure of police performance, they can be used to analyze the way police deliver services and possibly change resource allocation, if needed. Some community surveys provide detailed information about specific problems affecting parts of a city, which help to focus police activities (Weisel, 1999). Law enforcement agencies that implement community policing strategies often conduct citizen surveys to gauge community satisfaction and identify citizen concerns. Such surveys can help identify target areas and document progress and change from year to year. An example of a community satisfaction survey is found in Appendix D. You might find it beneficial and cost-effective to adopt or modify existing surveys that have been used in the policing context or in other relevant research. Data-collection instruments often are free, although some are copyrighted and must be purchased. For example, the Bureau of Justice Statistics and the COPS Office developed a software program for localities to conduct their own telephone surveys of residents to collect data on crime victimization, attitudes toward policing, and other community related issues. This survey can produce information similar to that published in Criminal Victimization and Perception of Community Safety, Twelve Cities. This software has recently been developed into an on-line application called Justice Survey Software and can be found at www.bjsjss.org Surveys and questionnaires have limitations, in general terms and when used in the policing context. In general, written surveys can preclude participation by people who speak other languages, who are illiterate, or who have learning disabilities, while telephone surveys can omit people who do not have telephone service. When using surveys to evaluate policing efforts, there are special concerns and barriers inherent in surveying victims and offenders. Many crimes occur in high-crime areas where people often are transient, so a disproportionately high number of potential survey subjects are likely to be harder to reach by telephone or written survey. Residents in high-crime neighborhoods also are more likely to be undereducated and under- or unemployed, which makes them harder to reach and potentially less likely to have the literacy skills required to respond to a written survey. Traumatized victims could be reluctant to share information over the telephone to strangers they cannot see, for fear of retaliation and because of the personal nature of information they might be asked to share. Honesty and candor are obvious concerns in surveying offenders and there are often insurmountable hurdles in identifying potential offenders who have not yet committed a crime. Ensuring that surveys are the appropriate means of data collection for your evaluation requires thinking through these potential barriers. Should you decide to use surveys in your research, some tips for survey development are presented in Exhibit III-4. Exhibit III-4: Tips for Developing Surveys and Survey Questions Keep your survey short, ideally no more than three or four pages. Make it easy to use—participants will be more likely to complete it. State the purpose of the survey on the first page, and also address the matter of confidentiality. Include necessary directions, explanations, or definitions. Indicate the approximate time it should take to complete the survey, where the respondent should return it, the due date, and contact information for answers to survey related questions. Keep your survey instrument simple, with short questions and clear answer categories. Look for other survey instruments with similar content that have been used in the field. Often, data-collection instruments are free, although others are copyrighted and have to be purchased. When relevant and appropriate, use existing questions from these surveys. Avoid “double-barreled” questions (i.e., questions that ask two things at one time). For example, “Did you find the training useful and timely?” The respondent may have found the training timely but not useful, so by combining what are two separate ideas into one question, you will not get an accurate answer. Use appropriate language for the audience. Questions should be devoid of vague qualifiers, abstract and biased terms, and jargon. Group your questions thoughtfully and logically. In general, begin with easier questions and progress into more sensitive or complicated topics. Request demographic information. Always include a statement at the bottom of your instrument thanking respondents for their time. Conduct a pilot test of your instrument. For written or e-mail surveys, send reminders to respondents to increase your response rate. end exhibit III-4 Qualitative Data Qualitative methods allow you to capture information that is difficult to measure, count, or express in numerical terms. Four qualitative data-collection methods and several tips for conducting interviews and focus groups are described on the following pages. Interviews Qualitative interviews are used to gather detailed information on a topic. The researcher asks open-ended questions to allow respondents to express in their own words their attitudes about, and experiences with, the program. Interviews typically ask people to describe or explain particular program practices or issues in relation to their personal experiences with them. Interviews can be a primary data source, a first step in developing other data-collection strategies (e.g., survey or focus group questions), or a complement to quantitative data collection. Interviews can be conducted by telephone or in person. Among the strengths of interviews is that they let you gather information you cannot learn through document review or observation. While interviews can provide more detail and depth than surveys can about stakeholders’ experiences and opinions, one of their disadvantages is that they are time-consuming. Interviewing tips are offered in Exhibit III-5. Exhibit III-5: Tips for Conducting Interviews Prepare your questions. Use interview questions that are open-ended, clearly worded, and neutral. Organize broad questions around the general themes in which you are interested, then you can probe to gain more information based on participant responses. Ask how and why questions and make sure you prompt respondents to elaborate on yes or no answers. Use appropriate language to which respondents will relate. Organize questions so they flow naturally and easily. Begin by asking nonthreatening questions that put respondents at ease and end the interview by asking if there is anything important you forgot to ask. Choose an appropriate setting. Interviewees should feel comfortable and not be distracted by other activities, lights, or noises. Explain the purpose and format of the interview to respondents. This explanation also should cover terms of confidentiality, approximate length of the interview, and securing permission to record or take notes during the interview. Practice “active listening skills” so participants are fully aware that you are paying close attention to their responses. Use silence to allow for elaboration. Probe for clarification. Answers to questions are not always direct and might require clarification or follow-up. Use probes to steer respondents toward the issue at hand. Tape-record the interviews, with respondents’ permission. Always thank interview participants for their time. end exhibit III-5 Focus Groups A focus group is a small-group discussion guided by a trained leader, which explores a topic in depth to steer future action. Focus groups are a quick way to gain impressions and opinions from various stakeholders and are appropriate when action should be guided by public opinion. Some law enforcement agencies use focus groups when securing community input on a critical issue before moving forward. As with all data-collection strategies, successful implementation depends on thorough planning. To prepare for a focus group, ask yourself, “What do I want to learn?” and “Why am I conducting a focus group?” Advice for preparing for a focus group is offered in Exhibit III-6 and a focus group checklist with more detailed information is in Appendix E. Exhibit III-6 Tips for Planning and Facilitating Focus Groups Find a good focus group leader. A good leader should be experienced, understand the focus group topic, and relate well to participants. While you might be tempted to facilitate the focus group yourself, it would be more appropriate to ask someone else within your agency to facilitate or hire an outside facilitator who will be familiar with focus group procedures. These include asking appropriate questions, reviewing the goals of the meeting with participants, setting participation ground rules, ensuring that all opinions are heard, concluding the meeting on time, and providing participants with information about the next steps in the process. Prepare your questions. Before the focus group starts, you should prepare a list of topics and questions that you want to ask. Word and organize them strategically. For example: Choose easy and nonthreatening opening questions designed to break the ice and make participants feel comfortable. Use transition questions to get the data you need. Prioritize your key questions and make sure they are answered before you finish the focus group. Include a final question that summarizes the discussion and gathers missing information. Find a recorder. A focus group requires someone to take notes similar to taking minutes at a meeting. You also can tape-record focus groups with the group’s permission. It can be useful to have an audio record in case you need to enhance your written notes. Decide about incentives. Sometimes focus group members get paid, even a small amount. If you can’t afford it, other possible incentives include refreshments, public recognition, or something to take home. Recruit your members. Ideally, those invited should be a representative sample of those whose opinions you value. Keep the focus group small, usually six to eight people, to encourage active participation by all group members. end exhibit III-6 An overview of the data-collection methods described in this section, including a summary of the advantages and disadvantages of each, is provided in Exhibit III- 7. Because each data-collection method has its pros and cons, you should consider using a mix of both quantitative and qualitative methods in your evaluation. In addition to describing data-collection methods, this section offered practical advice to develop data-collection protocols to meet your evaluation needs. Refer to the Instrument Development Checklist in Appendix F for more guidance as you create your own evaluation tools. Exhibit III-7 Data-Collection Methods at a Glance table Table Headers: Methods, Pros, Cons, Costs, Time To Complete, Expertise Needed Document Review Objective; least obtrusive Access to data may be tricky; data can be difficult to interpret; data may be incomplete Inexpensive for an internal evaluation Depending on the volume of data, may require a lot of time Some expertise required to access and interpret the data Interviews Gather in-depth, detailed info; data can be used to generate survey or focus group questions Takes time and expertise to conduct and analyze CInexpensive for an internal evaluation About 45 minutes per interview; analysis can be lengthy Good interview/ conversation skills; qualitative analysis skills Surveys Can be conducted anonymously; standardized data collection Selecting an appropriate sample may be difficult Moderate Moderate, depends on method of distribution (i.e., mail, telephone, or in-person) Some training needed to systematically administer and collect survey data Focus Groups Can get information quickly about needs, attitudes, and opinions Can be difficult to recruit participants, facilitate the group, and analyze data Inexpensive for an internal evaluation About 90 minutes Good group facilitation/ conversation skills; qualitative analysis skills end exhibit III-7 Obtain the Necessary Clearance Depending on the nature of your program and the evaluation you have designed, you might need approval from your agency leadership before moving ahead with data collection. Invasion of privacy and confidentiality issues are a serious concern, which is why academic researchers almost always send their survey protocols for pre-implementation review to an independent review board to certify that the research will not have an adverse impact on any human subject. Evaluation data often are very sensitive, and in some cases you might need to protect the confidentiality of program participants and obtain their informed consent. Informed consent allows participants to decide voluntarily whether to participate in the evaluation. Securing informed consent can involve describing the benefits that participants reasonably might expect to encounter, explaining alternatives to participating in the research, and telling participants the extent to which their personal information will be confidential. An example of an informed consent statement that you can modify for your evaluation needs is in Appendix G. To ensure that data are handled appropriately as they are collected, limit the use of data that personally identifies an individual (such as name and address) and keep evaluation data (e.g., notes, papers, or cassette tapes) under lock and key. Plan for Collecting and Tracking Your Data Once you have finalized your data-collection protocols and procedures and are ready to begin, you will need a way to manage data collection and organize the data you gather. A data-collection plan should be written and included as a subset of your overall evaluation plan. The plan should list a schedule of required data-collection activities matched to the appropriate data-collection instrument and person responsible for each task. An estimate of the level of effort necessary to complete each task also might be included. Strategies for data analysis can be linked to each data-collection protocol or strategy, as well (see Appendix H). In addition, you should think through processes for entering, tracking, storing, and securing your data. As you collect data, there are several procedures that will allow you to manipulate easily and understand your data once you begin analysis. Several tips for ongoing data collection and management areas follows: Data tracking. It is important to track carefully the data you collect. For instance, you can create a chart that includes the title of the instrument, the source of the data (e.g., program participant, community resident, program document), how the instrument was administered (e.g., focus group, mailed survey), and the period of data collection. Data entry and management. You will need to create a system for entering and managing your data. A database system will allow you to know at a glance how many people have responded, when, and if there are any missing participant data. Your database also will be used to carry out more complicated analysis. Before entering data into a computer from paper documents, be sure to make a copy of the form and you can make edits or comments on the copies. If the data are in electric form to begin with, make a backup copy on your hard drive or keep separate copies of your database on floppy disks or compact discs (CD). Data storage and security. Think about where and how you will store and secure the data you collect. If you collect data in hard copy form, you will need to decide where to keep the forms. Store written surveys and program documents in a place safe from damage or loss. For electronic data such as databases or interview transcripts, back up hard drives or keep separate copies of your documents on a floppy disk or CD. By following these tips while you collect information, you will be well organized to analyze your data and learn about your program. Data analysis is the focus of the next section in this chapter. Step 3: COMPARE—Data Analysis Once your data have been collected, analyze them following procedures that reduce and categorize the information to make it easier to manipulate, understand, and report. Always begin your analysis by reviewing your evaluation purpose and specific research questions, then proceeding to analyze data to yield results that are easily understandable. To be useful to stakeholders and decision-makers, data analysis should lead to straightforward, readily interpretable information about program processes and effectiveness. Techniques for analyzing data range from basic counts, to content analysis, to complicated statistical analyses. Databases, spreadsheets, and statistical or qualitative analysis software often can facilitate steps in data analysis, including creating a computer database file to enter data, coding and entering them, cleaning data, using a computer package to analyze them, and interpreting results. Depending on the type of data you collected, you might conduct quantitative data analysis, qualitative data analysis, or both. Simple procedures for analyzing qualitative and quantitative data are discussed in this section. Quantitative Analysis Using statistical analysis to examine quantitative data (i.e., data that can be counted or assigned a number) can be difficult. Law enforcement agencies that have been successful with complex statistical analysis report having sophisticated data-tracking systems and using standard analysis techniques and evaluation tools. For law enforcement agencies that do not have the expertise or resources for data analysis, or when sophisticated statistical analyses are needed, consider hiring a consultant for guidance. Simple frequencies, percentages, and cross tabulations often are all that is needed. The following a few tips for analyzing quantitative data that do not require complicated statistical procedures: Clean your data (i.e., review data for missing or incorrect responses). Tabulate the information by counting the frequency of its appearance. For example, if you want to know how many people have ever attended a neighborhood watch meeting, simply count the number of “yes” responses to the survey question, “Have you ever attended a neighborhood watch meeting in your area?” If it is ratings and rankings you are interested in, you can compute the mean (average), median, or mode. You can find the mean, or average, by dividing the sum of all responses to a particular question by the number of responses to that question. You can generate the median score by arranging the responses in a list. The middle number is the median. If there are two middle numbers, then add the two numbers and average them for a median. Last, you can tabulate the mode by looking for the number or value that occurs most often. During analysis, keep in mind the following questions: What do the raw data tell me? Are the results low, average, or high? Are there any red flags or extreme values to be aware of? What do they mean? Qualitative Analysis As with quantitative data analysis, qualitative data analysis is a process for reducing the amount of data that you have as well as aggregating and organizing individual responses into themes that allow you to identify data trends. Qualitative data include information from interviews, focus groups, written documents, observations, and open-ended survey questions. While qualitative data analysis can seem less straightforward than quantitative analysis, there are ways to analyze qualitative information systematically. Much of your analysis will be focused on providing a mechanism for organizing and describing the themes, patterns, activities, and content of the study rather than on illustrating causal linkages between processes and outcomes (Patton, 1990). While it is not within the scope of this guide to delve deeply into the strategies and nuances of qualitative data analysis, many handbooks and manuals exist for those who are interested in learning more. Basic steps in qualitative analysis are as follows: Review all the data (e.g., interview transcripts or focus group notes; program documents). Organize and label (code) responses into similar categories or themes. Try to identify patterns or associations among the data; look for similarities and differences across respondents (e.g., all law enforcement officers who attended training have a positive attitude toward the program, but officers who did not attend training think the program is a waste of time). Identify the meaning that has emerged from the data (i.e., ask yourself, “What are respondents telling me through these data?”). Interpret the data in relation to the issue of interest (i.e., ask yourself, “What does this mean in light of my evaluation questions?”). Description is the foundation of all qualitative data reporting. Several options for organizing and reporting descriptive findings (Patton, 2002) are summarized in Exhibit III-8. Footnote: For more information on qualitative analysis, we encourage you to consult Miles, M.B. and A. M. Huberman. Qualitative data analysis. Sage: Thousand Oaks, California, 1994. Another excellent reference is Patton, M.Q. Qualitative research and evaluation methods. Sage: Thousand Oaks, California, 2002. Analysis of Program Costs and Benefits The need to link criminal justice program costs to outcome measures has become critical as policy makers increasingly require decision-making tools to allocate scarce funding. Systematic economic valuation of program costs and dollar benefits improves the basis for making funding decisions. Exhibit III-8 Options for Organizing and Reporting Qualitative Data Analytic Framework Approaches Processes: Qualitative data can be organized to describe important processes. For example, if the department stations an officer in a public housing project to reduce youth violence, describing the process of how the officer begins interacting with the community, especially young people, might become an important theme that could be used to frame evaluation questions or analyze the data gathered. Issues: An analysis can be organized to illuminate key issues, often the equivalent of the primary evaluation questions; for example, variations in how participants changed as a result of the program. With the public housing example above, building trust is an important issue. The evaluation could explore various tactics that the officer used to promote trust and how perceptions and behaviors changed on both sides. Questions: Responses to interviews can be organized question by question, especially where a standardized interviewing format was used. For example, the evaluator might want to ask the same questions of the officer and community residents, and then organize the responses by question, allowing readers to compare police and community responses. Sensitizing Concepts: When sensitizing concepts such as “trust” versus “mistrust” might have played an important role in directing data collection, the data can then be organized and described through these concepts. Source: Patton, M.Q. Qualitative research and evaluation methods. Sage: Thousand Oaks, California, 2002. end exhibit III-8 Cost Analysis Cost analysis consists of a thorough examination of the type and amount of all resources used to support a law enforcement agency’s activities. Determining the cost figures for the agency as a whole, or in its component parts, is a basic form of cost analysis. Tangible program items such as salaries and equipment are given a dollar value. Costs typically are provided at several levels, from the total cost of the agency to the costs associated with each area of responsibility. Cost-Effectiveness and Cost-Benefit Analyses There are two levels of cost analysis evaluation: cost-effectiveness and cost-benefit. The former assesses whether the money for a program was well spent to reach the program outcomes, while the latter assesses costs of achieving the intended impact. Cost-effectiveness is the relationship between agency costs and its effectiveness in providing community safety. Cost-effectiveness is a technique in which costs are measured as dollars spent, whereas effectiveness is measured in changes in outcomes (alternatively, outcome counts can be used). The overall cost- effectiveness of an agency is improved by first determining the activities of the agency that contribute most to effectiveness and then determining which of those activities have the lowest cost. For example, the cost-effectiveness of a law enforcement agency might be measured as the number of offenders apprehended in one month versus the costs of various programs that an agency uses to accomplish this (e.g., community outreach, officer training, crime mapping). Cost-benefit analysis compares costs and benefits using the same monetary units. Determining the dollar value of a benefit is the primary challenge in cost-benefit analysis. The benefits of law enforcement activities are many and include decreased social disorder, decreased fear of crime, and better or more public services to respond to crime. Another prime benefit is decreased crime, resulting in decreased justice costs and lower costs to potential victims and society. The criminal justice literature provides estimates for such of benefits. During cost-benefit analysis, the present value of future benefits must be calculated and compared to present investment costs. Economists use discounting when benefits will be received in the future to make future monetary benefits comparable to current dollar values. To accomplish this, future benefits are discounted by some factor—the higher the rate, the lower value of future benefits. The discount rate is important because offender benefits might not be realized for years. Generally a rate between 3 and 5 percent is used. To determine a return on investment, the cost-benefit analysis will present results for benefit-cost ratios and net present values. An example can be found in Cohen, M.A. “Measuring the Costs and Benefits of Crime and Justice” (pgs. 263–315). In Measurement and Analysis of Crime and Justice. Washington, D.C.: U.S. Department of Justice, 2000. If we think of the domestic violence example used in Chapter II, both approaches require totaling program costs, often a combination of grant funding and regular operating costs. A cost-effectiveness analysis would explore whether the money invested was justified by the major outcomes—the number of PPOs issued and the number of police personnel trained. A cost-benefit analysis would attempt to answer whether money expended on the program was justified by a reduction in domestic violence cases, including deaths and injuries. As this example suggests, it is easier to conduct a cost-effectiveness analysis than a cost-benefit analysis, in part because it is difficult to quantify deaths and injuries prevented. Ascribing a dollar amount to those costs also is problematic. Chapter Summary While the part-time or full-time participation of an evaluation consultant could enhance the strength and objectivity of evaluations of policing programs, many departments can and should perform more evaluation activities by relying on their internal resources. This chapter detailed the major steps in the evaluation cycle and provided practical information and advice to help you conduct an internal evaluation of your policing strategies. In the first section of this chapter, the myriad components of evaluation planning were discussed, including evaluability assessment, stakeholder involvement in the evaluation process, defining program goals and objectives, and determining the logical relationship among program inputs, outputs, and outcomes. Tips were offered for developing evaluation questions and selecting an appropriate design based on evaluation planning activities. This chapter also provided guidance about ways to identify and implement data-collection strategies beyond traditional police measures. Material presented in the section on data collection included a brief summary of the pros and cons of different data-collection methods as well as tips for developing data-collection procedures and protocols such as surveys, interview guides, and focus group guides. The importance of obtaining necessary clearances and permissions to collect data from program participants was underscored. Last, simple procedures for making sense of your data were discussed in the section on data analysis. Now that you are familiar with all the steps involved in planning and implementing an evaluation, you should be able to create your own evaluation plan. A sample evaluation planning template can be reviewed in Appendix H and used to assist you in developing and documenting your general evaluation plans. The following chapter provides useful strategies for managing internal and external evaluations of policing activities. Chapter IV: Managing Evaluations Managing Evaluations There are two types of evaluations in which law enforcement agencies are likely to be involved—internal evaluations and external evaluations. As noted in Chapter III, internal evaluations are those in which police agencies rely primarily on employees within the department to conduct evaluations and disseminate results. In some instances, agencies have research or evaluation divisions (or staff) specifically to support the development and implementation of evaluation within the agency. On the other hand, external evaluations are those for which the agency hires an outside expert to conduct the research. Generally, external evaluators are hired when the agency does not possess the expertise or resources to conduct evaluations internally or when a particular policing program, intervention, or strategy requires an evaluation design that is more complex than the internal evaluation staff are prepared to handle. While internal and external evaluations are conducted for different reasons, many of the steps required in planning and managing them are very similar; although each has its own unique set of challenges as well. This chapter first outlines the steps required to plan and manage an internal or external evaluation; the section that follows outlines some of the challenges unique to managing each type of evaluation. Finally, this chapter provides important information about how to hire an external evaluator, including assessing evaluator capacities, what to look for in an external evaluator, and sources for finding external evaluators. How Do I Manage an Evaluation? The planning and ongoing management of evaluation activities, whether internal or external, can be as important as the methodological aspects. While there is no single recipe for managing an evaluation, law enforcement administrators who do must possess a basic understanding of evaluation for appreciating the work to be conducted, assisting in its planning, and identifying weaknesses in the design or data-collection plan. In addition, there are management issues to consider at critical junctures prior to, during, and following an evaluation. These are outlined in more detail below. What do I need to consider prior to the evaluation? If you are assigned to manage the evaluation, you might consider obtaining input from several colleagues, particularly those with an evaluation background or experience with the program or strategy the evaluation will target. If possible, also involve groups that are in some way connected with the program or strategy, but are not affiliated with the law enforcement agency. These might include community members (in the case of community outreach activities), stakeholders (those supporting or funding the policing activity or strategies), and program staff. Finally, involving the leadership within your agency will help ensure support at the highest level. Before conducting an evaluation, determine the evaluation requirements, develop the evaluation plan, and set up administrative and monitoring tools. These activities are described in the section that follows. Determining the Evaluation Requirements Before anything else, think about the reasons for your evaluation. Are you considering it because you want to use the information to improve a particular program or strategy? Do you want to know if something is producing its intended outcomes? Is the evaluation a requirement of a grant or other contractual relationship? Understanding the reasons will help you determine the best approach. For example, if the goal is to modify or improve a program or strategy, you might consider an evaluation that is focused on process rather than outcome. Or, if the evaluation is a grant requirement, you must develop an evaluation plan that meets the information needs of your contract with the funding source. Most important, determine if an evaluation even is feasible by considering several issues: Are the data you need to answer the research questions available? If available, can you access them? For example, school records may be available but school rules or consent procedures may prohibit you from accessing them. Do you have the financial and other resources to conduct an evaluation? If internal, this includes having staff with the time and expertise necessary to conduct the evaluation. If external, this includes having access to an experienced evaluator who can work within time and budget constraints. In either case, financial resources must be available to support evaluation costs. Do you have buy-in and support from your agency administrators? If the leaders in your agency (e.g., your chief or executive) do not support and approve of the evaluation, it may preclude you from fully implementing or funding it. Once you have determined that an evaluation is both necessary and feasible, you can proceed to the next step—developing the evaluation plan. Developing the Evaluation Plan Developing an evaluation plan, whether you are managing an internal or external evaluation, requires several important steps. Some of these are outlined in detail in Chapter III and will not be covered here, including determining what will be evaluated, establishing goals and objectives, developing logic models, developing evaluation questions, and determining the appropriate design. Other management tasks that are an important part of an evaluation plan will be explored here: determining the evaluation budget and timeline, identifying resources available to conduct the evaluation, and devising a plan for using and disseminating evaluation results. Budgets and Timelines Developing budgets and timelines are very different tasks. Because your budget will have an impact on the duration of your evaluation, and vice versa, developing one without consideration of the other is not recommended. On the other hand, each one also has unique circumstances that require consideration during planning. With this in mind, we present each issue separately. The final paragraph ties the two together. Budgets When thinking about your evaluation budget, take into account several categories of expenses. Labor typically is the most costly evaluation expense, whether your evaluation is internal or external. Internal labor expenses include staff time and, for full-time staff, associated costs such as overhead (e.g., electricity and rent) and fringe benefits (e.g., insurance, retirement, and vacation time). If you use an external evaluator, you will not need to estimate overhead or fringe benefits, but you will likely require more financial resources than if you were using internal staff because external evaluators, especially academic or other highly qualified professional evaluators, generally are more expensive than internal staff. Depending on the extent of the evaluation, an external evaluator may cost you approximately 20 percent of your overall budget. For example, you might consider setting aside $20,000 if your full budget is $100,000. A more efficient method of estimating external costs is to solicit cost proposals from several evaluators. Cost proposals require that each bidder describe in detail the tasks to be completed, the number of hours each task will take, and the costs associated with each, including labor. You should receive several bids that are approximately the same. If you receive bids that are very different from the rest, you can assume they either under- or overbid the work. In all cases, your resources are finite so be sure you have considered all possible costs and that they are fully addressed in the cost proposal of the bidder you choose. Unanticipated costs can derail an evaluation quickly, particularly if the costs are associated with activities that are critical to the evaluation (e.g., data collection or analysis). In addition to labor, evaluation budgets also can include incentives or rewards for participants, if appropriate, and other indirect costs (e.g., postage, telephone, duplication). Travel expenses (e.g., airline tickets, rental cars, per diem) also must be estimated; be careful, because extended travel expenses can add up quickly. In addition, you can never fully anticipate costs associated with cancelled and rescheduled flights, and stays in expensive hotels when the preferred (and less expensive) ones are full or cannot accommodate a government rate. If data can be collected locally rather than off site, your travel expenses can be reduced drastically. Finally, consider whether the evaluation will require any special equipment, varying from standardized instruments (e.g., existing surveys and questionnaires available for purchase) and tape recorders for data collection, to data analysis and management tools such as software and computer upgrades. As you estimate evaluation expenses, be certain you have the resources to fund your evaluation before you take any steps toward implementation. There typically are several options for funding evaluation activities, including the following: Agency budgets. Many law enforcement agencies have a line item in the budget to cover evaluation activities. If this is the case in your agency, be clear about how much money is in that budget, the types of activities it is designed to fund (some set-asides are earmarked for specific categories of expenses such as labor or data collection), and if other officers intend to use the money for their own evaluations. Grant funds. Many grants require the grantee to set aside a certain percentage of the grant funds (most often 10 percent) to cover evaluation expenses. Unfortunately, these funds often are insufficient to cover all related evaluation expenses. As a result, if you are using grant funds for your evaluation, you may need to consider additional sources of funding. Outside funding sources. Government, private, and some nonprofit agencies and foundations offer grant funds specifically for evaluation activities. These funds are offered most often in the context of advancing knowledge in a particular area or to support the implementation and evaluation of best or promising practices. Therefore, you may have to search to find an evaluation grant program focused on funding evaluations in your program area. Timelines The requirements of your evaluation most often determine its duration. For evaluations that are designed to meet funding requirements, the funder and the duration of the grant generally determine the evaluation timeline. For example, if you are awarded a 3-year grant to implement a community policing strategy in a local neighborhood, you might need to measure and report outcomes of interest to the funder at the end of the 3 years. In the interim, the funder could require that you document certain components of the strategy, and/or measure short-term or intermediate outcomes and submit the findings in quarterly or semiannual reports. In this situation, the evaluation as funded is expected to terminate at the end of the grant period and will continue only if you choose to and have resources to support ongoing evaluative efforts. For evaluations that are not bound by funding requirements (e.g., those that are funded using agency dollars or are designed to monitor program progress), it could be up to you to determine the timeline. If this is the case for your program or strategy, there are several issues to consider. First, think about what outcomes the program or strategy is designed to achieve and when those outcomes realistically might be realized. For example, if you are implementing a substance abuse awareness program in the local high school, you might expect knowledge about the dangers of drug use to be evident in the participants immediately after they complete the program. Lower substance use rates in the high school population, however, could take longer to achieve; in fact, they might not be evident for years. Determine which outcomes your evaluation is intended to achieve and when they will occur, and develop your timeline accordingly. If you conduct an impact evaluation (e.g., demonstrating long-term outcomes), extend the evaluation timeline outward to a point at which these outcomes can be realized. On the other hand, if you conduct an outcome evaluation (e.g., demonstrating immediate, short-term outcomes), your evaluation timeline will be more closely related to the duration of your program or strategy. For example, if you implement a program or strategy that is 10 weeks long, you should gather valuation data right before (pre-test) and right after (post-test) program implementation. Despite their differences, budgets and timelines should be developed simultaneously. The money and other resources that are available for the evaluation will have implications for how long it can continue. Whether internal staff conduct your evaluation or you hire external evaluators, you must know the extent and duration of available resources. You could have ample resources for evaluation one year, but as agency priorities shift, funds might be reallocated in future years. In addition, you might not start new evaluation projects if you know the chief is retiring or the governor is running for reelection. Changes in political structure sometimes mean law enforcement priorities change and budgets can be realigned to reflect those changes. You do not want to discover midway through your evaluation that funds have been exhausted. Be certain from the outset what funds are available and for how long so you can implement your evaluation completely and successfully. Evaluation Resources Even if your organization maintains a research and evaluation division through the annual budget, you might need to hire an external evaluator or outside consultants to assist internal staff with complicated evaluations or those that exceed your in-house capacity. Therefore, you could require evaluation funds in addition to those in your existing budget. Or, your organization could be one of the many law enforcement agencies that lack not only an evaluation unit but also an evaluation budget. Fortunately, regardless of your situation, several evaluation resources are available, including the following: Writing a grant proposal to request evaluation funds. Increasingly, government and foundations are willing to support thoughtful evaluations that have the potential to identify and foster promising practices. Building your agency’s internal evaluation capacity. Training and technical assistance (TA) often is available for staff development on evaluation-related topics. Providers include trained evaluators, academics, and individuals employed by federal agencies or professional evaluation associations. Because TA providers often are supported by state or federal grants, many of these resources are available free of charge or for a nominal fee. Check with agencies from which you currently receive program or other funding to learn if they offer evaluation TA opportunities to their grantees. See Appendix L for a list of technical assistance resources. Reviewing existing law enforcement evaluation reports for best practices and lessons learned. An inexpensive way to learn about evaluation is by reading evaluation reports and related materials. Evaluation reports often are available to the public and can be a good source of information regarding evaluation strategies that have been implemented in your field and those that have been particularly successful. In addition, hundreds of evaluation resources are available on the Internet, including manuals, step-by-step guides, and technical reports. Most of these materials are either free of charge or can be obtained for a nominal fee. See Appendix L for a list of evaluation reports related to policing. Other law enforcement agencies. Some law enforcement agencies have research or evaluation divisions staffed with competent evaluators who also are law enforcement agents. These individuals might be willing to assist you and your staff to plan and conduct evaluations, share information about what has worked for them, or provide technical assistance to build the internal capacity of your agency. Graduate students. Graduate students often are required to conduct a research study as part of their graduation requirements. Master’s degree students must complete a thesis, while doctoral students are required to prepare a dissertation, and they often will take on an evaluation or other research study without being paid. In addition, because the work is being conducted to fulfill graduation requirements, a team of academics generally is assigned to oversee and monitor the project from planning through completion, ensuring a high-quality and rigorous evaluation. Utilization and Dissemination Plan The likelihood that evaluation findings will be used increases with comprehensive planning, preparation, and follow-up. An evaluation might meet standards for rigorous design, careful data collection and analysis, and a well-written report, but if it does not have practical implications and is not used, it has failed. When an evaluation proves to be useful, it usually succeeds in answering a specific, clearly defined question that someone wanted answered. As evaluation manager, you must make sure the appropriate evaluation questions are asked. Soliciting stakeholder input about what they need from the evaluation and what would be most important for them to know about the program or strategy ensures that evaluation results will not simply sit on a shelf somewhere. As evaluation manager, you also should have an information dissemination strategy to fit stakeholder needs, using a variety of communication methods, which might include formal reports and briefings as well as informal strategies, if appropriate. The dissemination strategy could identify the purposes and stakeholder audiences, which might include program staff and supports, community members, funders, policymakers, and the media. Developing Monitoring Plans During planning, establish procedures to ensure that you can monitor both the evaluation work and budget effectively, regardless of whether the evaluation is conducted internally or externally. For example, how and when will your external evaluator submit invoices for payment? What means will you use to monitor the work of your internal evaluation staff? These issues are discussed in detail in the section that follows. Monitoring Procedures Monitoring your evaluation effectively requires you to supervise your staff, their work, and the budget. The procedures you use, however, depend on whether you are managing an internal or external evaluation. Questions to Guide Your Internal Monitoring Plan How will you maintain frequent contact with your evaluation staff? How will you require your evaluation team to document its progress? How will you learn of problems promptly? Do you need a project manager or primary staff person to oversee the evaluation? Internal Evaluations Having frequent and regularly scheduled meetings with internal staff is a good way to monitor the extent to which they are implementing the work plan as intended and are staying on track with the evaluation timeline. Staff meetings also provide opportunities for identifying potential problems and solving them before they grow. Frequent contact with your evaluation staff also allows you to monitor projected labor hours. If your staff are spending more hours than you have allocated for them (or fewer), you may need to modify your work plan and/or find additional resources (e.g., money or staff) to assist with implementation. You also can monitor internal staff through e-mails and telephone conversations. These methods can be used in between staff meetings to keep abreast of progress and to solve problems routinely. Also keep in mind how your internal team will document progress, including challenges to implementation that might disrupt your timeline. You might set aside time during regularly scheduled staff meetings for oral progress reports that are structured in content, refer to a specific period (the 2 weeks prior to the meeting, for example), and might cover major activities, problems or challenges, action steps taken, and future activities. If you plan to have staff report orally, be sure to have someone document what is said, perhaps the staff person responsible for taking meeting minutes. A written record of what is happening in your evaluation is important as a reference if your funder or chief has questions about the evaluation or if problems arise. In developing a plan for monitoring internal staff, decide if you want to designate someone other than yourself as the evaluation manager or primary staff person (e.g., this might be the most senior person on the evaluation team or the one with the most evaluation experience). Designating someone as the project manager not only provides a point of contact your staff can turn to when you are not available, but also identifies one person whom you (or your chief or funder) can call when questions or problems arise. You do not want to waste time figuring out which staff member to call when your chief or a federal project officer is on the line with a question about the evaluation. In addition, a project manager is someone to whom you can delegate certain responsibilities to assist you to do your job more effectively. For example, you can ask your project manager to facilitate staff meetings and report to you about progress and problems. External Evaluations Compared to an internal evaluation, managing an external evaluation can demand more effort in advance to select and hire a qualified evaluator, to work collaboratively with the evaluator throughout the evaluation, and afterwards to review the report and disseminate findings. Many of the techniques for managing internal evaluations, however, also apply to external evaluations. In-person meetings and frequent telephone and e-mail contact also are good methods for monitoring external evaluators. Monitoring an external evaluator, however, also poses a unique set of challenges. First, external evaluators are just that—external. You do not have the same opportunities to meet with and monitor their progress as you would with internal staff. Nonetheless, you can offset this problem by making regularly scheduled meetings a part of the contract with the evaluator. You might stipulate that monthly (or weekly or biweekly) evaluation meetings will take place either in- person or by telephone, and that these meetings will be used to discuss progress against the work plan, problems with or challenges to implementation, and budgetary matters. In the contract, also outline the consequences if certain requirements are not met. For example, you can reserve the right either to cancel the contract or withhold payment until requirements meet with your satisfaction. Should the evaluator fail to appear for meetings or be otherwise unavailable to you throughout the evaluation, you can initiate the agreed-on consequences. We also recommend that you require some type of progress report from your external team. Written progress reports are a standard monitoring method and, as with invoicing, you will want to determine how, and how frequently, you will require that your evaluator submit these. In the contract with your external team, you also should state clearly the purpose, content, and timeline for these reports. Determine how you will monitor the evaluation budget of an external evaluator. Will you require monthly or quarterly invoicing? More frequent invoicing might allow you to monitor your budget more closely but, if your budget is small, less frequent invoicing may be acceptable. Also consider what details you want each invoice to contain. Some contracts require little detail about how the money was spent or expenses were incurred, while others specify all items must be clearly identified and documented. In addition, you should know if your agency has standard invoicing procedures for an outside contract so that you can adhere to them. In managing an external evaluation, your contract is your leverage; therefore, it must state clearly each requirement and how it will be monitored. Be sure your evaluator or team understands the contract requirements before they sign it. If, like many large law enforcement agencies, your agency has a contracts department, consider involving the department in contract development and negotiations. Contracts are legally binding documents and can be challenged in a court of law; therefore, make sure your contract meets the standards for contracts of its kind. These steps will contribute to the leverage and documentation you need to monitor your external evaluator effectively. Questions to Guide Your External Monitoring Plan Have you contacted the evaluator’s references? Do you have a context for his or her previous work? Does the contract stipulate: How often and by what means meetings will take place? What consequences will be implemented if the contract is violated? Do you have a plan for monitoring the budget? Does it allow you to identify problems immediately? How will you learn of problems promptly so that they can be resolved efficiently and immediately? Have you identified a primary contact person with whom you can work in the principal investigator’s absence? You can use administrative procedures such as those suggested above to keep your external evaluator in check. However, nothing beats having a positive working relationship, which facilitates communication between you and your evaluator and, in turn, ensures that critical information is available to you in a timely manner for smoother process monitoring. You can increase the likelihood of a positive relationship with your evaluator by hiring an individual or agency you know, that you (or someone you know) have worked with in the past, or is referred to you by someone you know and trust. If this is not possible, do not hesitate to ask potential hires for past evaluation reports and references. Past reports let you see the kind of work done previously and identify agencies or individuals who have contracted with the person for similar work. If these agencies or individuals are not given as references, do not hesitate to contact them yourself. Also, be sure to contact the references that are provided, not only asking questions about the candidate’s work, but also about work style. You also might ask for additional references to others for whom the candidate has worked. Contacting these individuals or agencies will give you additional information about the credibility and integrity of the potential evaluator. What Do I Need to Do During the Evaluation? For the most part, what you do during the evaluation is to implement the monitoring techniques you developed during the planning phase. For internal evaluations, this includes conducting regular staff meetings, conducting e-mail and telephone communication with your evaluation staff, and reviewing project expenditures. If you have an internal project manager, that person can handle many of these activities and report to you. For external evaluations, hold regular meetings with your evaluator either in person or by telephone; review progress reports and invoices, and follow up on any issues that emerge. During the evaluation, be sure you are meeting other demands as well. For example, if you are required to give periodic progress updates to your chief or funding agency, you must plan for these activities, especially if they must be in writing. During the evaluation, find ways to provide feedback to the team (and those individuals and agencies involved with the program or strategy under study) about how things are going and what initial findings are emerging. As evaluation manager, you will have the most comprehensive information about the study and the outcomes being achieved. Keeping your team informed is important because, in completing day-to-day evaluation activities, staff can lose sight of the “big picture.” Providing staff, whether internal or external, with frequent updates about the evaluation not only provides you with an opportunity to acknowledge their hard work, but also helps them remain focused on overall project goals. During the Evaluation… Conduct frequent meetings with your evaluation team. Review project expenditures. Follow up immediately on problems or other concerns. Provide feedback about the evaluation to the team and your other stakeholders. Be prepared to make program modifications if early findings are not promising. Despite your best intentions, a variety of things can go wrong during an evaluation, whether it is internal or external. Staff turnover at the program site (if your data are being collected by program staff or those responsible for implementing your strategy directly) and on the evaluation team, low program participation rates and poor program implementation can derail even the best evaluation plans. If you can anticipate these events, you can respond to them appropriately and quickly. However, some issues, such as staff turnover, are difficult to anticipate. Generally, staff do not make public their intentions to leave a position, at least not with enough time to plan for their departure. Therefore, in addition to building a relationship with your evaluator, you should maintain contact with those close to the program or strategy being evaluated (e.g., a project director or program staff person). These relationships help you learn important information about events you might otherwise not obtain (e.g., a staff person is leaving the agency). What Do I Need to Do After the Evalaution? Your evaluation responsibilities do not end with submission of the final report. You will need to implement several important activities after the evaluation ends to ensure that your evaluation has resulted in the type of information you need to understand. This means not only how effective your program or project was but also what improvements need to be made to increase the likelihood of achieving positive results in the future. These activities are outlined in detail in the sections that follow. Review the Evaluation Report The evaluation manager has primary responsibility for responding to and using evaluation results. This responsibility encompasses reviewing the report to determine that it is a clear accounting of the evaluation process and results and that it meets the reporting requirements set out in the contract. Specifically, the evaluation manager should review the evaluation’s key findings, conclusions, and recommendations. Most well-written evaluation reports include six sections: 1. Executive summary. An executive summary should contain a very brief description of the evaluation’s purpose, questions, procedures, most important findings, implications, and recommendations. It should not be more than a few pages in length. 2. Introduction. The introduction should contain the purpose of the evaluation and the overview of the report contents. The introduction sets the stage for the rest of the report by outlining basic evaluation purposes and introducing the report contents that follow. It often states the intended audiences for the report and cautions the reader about limitations that might have affected the collection, analysis, or interpretation of information. 3. Description of the evaluation focus. The third section, the focus of the evaluation, should contain a description of the program being evaluated and the evaluation questions. This section typically discusses the rationale for the evaluation, who it is intended to benefit or serve, activities and strategies followed in implementing the evaluation, settings, and resources devoted to the program. Also in this section are the specific questions that shape the evaluation. You might want to include a copy of your program logic model here. 4. Evaluation plan and procedures. The evaluation plan and procedures section contains a description of the evaluation design, data-collection strategies and instruments, and an overview of data analysis. This is the section in which the evaluator presents the methodological and technical information to establish evaluation adequacy and credibility. It is helpful to outline the information being collected to answer each of the evaluation questions and describe the procedures for data collection and analyses and include any instruments as appendices. 5. Presentation of evaluation results. The presentation of evaluation results section should include a summary of evaluation findings and an interpretation of them. More extensive details related to the findings can be included in supporting appendices. The specific criteria being used to judge the results should be stated explicitly so they can be viewed in light of these standards. For example, if the standard you are using to judge effectiveness is citizen satisfaction for your area, data should indicate whether citizen satisfaction has increased, decreased, or stayed the same. Interpretation of the findings should reflect a careful process of synthesis and analysis that supports judgments and subsequent recommendations. It is helpful to relate the findings clearly to the specific evaluation questions. 6. Conclusions and recommendations. The conclusions and recommendations section should outline the implications of the evaluation findings for your program and specific recommendations for program modifications or improvements. Organize conclusions by strengths and weaknesses that focus attention on both positive and negative judgments. Descriptions of strengths and weaknesses must contain sufficient detail to give stakeholders an understanding of the rationale and to serve as the basis for recommendations. Disseminate Evaluation Findings After evaluation data have been collected and analyzed, communicating the results is critical if they are to be used effectively. When overseeing an evaluation, the evaluation manager should seek opportunities to share results with stakeholders and program managers throughout the process, but the final results are often what stakeholders want. Specific strategies for developing appropriate evaluation reports for many stakeholder groups are presented below. Program staff and supporters Usually, the people who are involved in implementing your program believe that it works well and brings about positive results. They will be most interested in what the evaluation can tell them about ways to improve the program’s function and about how participants feel about the program. Reports that include detailed information on these topics will be most appropriate for this audience. Funders. Funders want to be sure a program is making good use of the resources they provide and the program is accomplishing its goals. Therefore, when reporting evaluation findings to funders, present information regarding the implementation and specific outcomes of your program. Policymakers. These people make strategic decisions about the program itself and tend to be interested in the “big picture” rather than being concerned with details. A brief document highlighting the findings that address program effectiveness is appropriate for them. Media. One of your main objectives in delivering findings to the media might be to gain visibility for your program. Media outlets tend to run very brief, straightforward stories and probably will not be interested in detailed, extensive reports. Your best bet in working with the media is to provide a short, concise document that includes an overview of your program, the particular findings you want to highlight, and the implications of those findings for your community and future policing efforts. How you use your findings depends greatly on your audience. While written reports and briefings are common strategies for communicating evaluation results, you should also consider using other techniques. The most useful evaluation products are often presentations to decision makers or community groups, but a brief summary report can be distributed to a wider audience, press releases can provide more exposure to evaluation findings, and web sites can include visual supplements such as photos and color charts. Web sites, e-mail messages, question-and-answer statements, press releases, op-ed items in newspapers, speeches, written testimony, newsletters, articles in professional journals, brown- bag lunches, videotapes, or computerized evaluation presentations are increasingly being used as communication tools and should be considered when deciding how to disseminate evaluation findings (USAID Center for Development Information and Evaluation, 1997). Apply the Evaluation Results Evaluation findings can be used in several ways, depending on your organization and what evaluation stakeholders deem appropriate. Generally, findings can be used in the following ways: For internal improvement. Evaluation results can be used in internal agency decision-making to improve ongoing implementation of a program or refine strategies based on lessons learned about challenges and successes. For accountability. Findings can demonstrate to stakeholders that the program is having a positive impact or is particularly cost-effective. Evaluations often are used to secure grants or convince supporters that the program is functioning as planned. For example, to maintain or secure funding, a granting agency might ask for proof that your program is achieving its stated goals. Findings can show that the program is successful in meeting its outcomes. If you conduct a process evaluation, the report also can document how funds or resources are being used. For outreach. Evaluation results can be shared with the media and directly with the community to increase visibility for your initiatives. Strategies include releasing findings to the media, organizing town hall meetings to discuss results, providing information to school and faith community newsletters, and building an information web site. Challenges to Managing Evaluations While most evaluations run smoothly if well planned and monitored, there are challenges associated with managing an evaluation, whether internal or external. These include the following: Time. Be clear about how much time you need for management activities and how your time can be spread across the various tasks for which you are responsible, including those not associated with the evaluation. As suggested earlier, if your time is limited, consider appointing a project manager to whom you can delegate a portion of the management activities. In addition to your time, also consider staff time. Do you have staff with time to implement the evaluation effectively? Generally, law enforcement officers are very busy individuals with little time to spare. If you do not have the internal capacity to conduct the evaluation, consider hiring someone from the outside. Resources. When budgets are prepared each year, every available dollar is assigned to some activity or function. If evaluation is not part of your agency’s annual budget already, funding your research will be the most challenging aspect of your management duties. In addition, you might find that funding the evaluation fully is not a one-time event, but requires attention throughout the study. Competence. Whether managing internal or external evaluation staff, be confident they have the necessary skills to implement the evaluation effectively. Assessing the team’s competence is an issue not only before the evaluation begins, but also as it progresses. Evaluation staff require methodological and analytic skills and also must possess the capability to work cooperatively with other people, groups, and agencies. You might have a clear understanding of their evaluation skills in advance, but their ability to collaborate with others most likely will emerge over time; therefore, have mechanisms in place to monitor evaluation staff competence. In addition, you need good management skills to handle the evaluation appropriately. Have you ever managed a budget before? Supervised staff? If not, consider appointing someone to manage the evaluation for you. Support. In law enforcement agencies, any activity outside the realm of law enforcement calls for support from the administration. If your chief is not convinced of the value of evaluation, you will be hard-pressed to find the resources you need to implement your research. If evaluation is not an agency priority, you can promote it by taking advantage of opportunities to educate the administration and other officers about the benefits of evaluation. Be sure you have done your homework and be prepared with examples showing how evaluation findings increased an agency or program’s visibility or generated additional funds for the agency. While these challenges exist to some extent in most law enforcement agencies, they can be overcome or at least managed effectively. Think through these issues and ways to resolve them when planning the evaluation. If you do not plan for them, these challenges are more likely to interfere with your ability to garner the support you need to carry out your evaluation. Should I Hire an External Evaluator? If you have decided to conduct an evaluation but your agency does not have an internal evaluation unit (or it does have such a unit, but the staff do not possess the necessary expertise), you might need to hire an external evaluator. This section of the guide offers tips for decision-making and selection when hiring an external evaluator and addresses four fundamental questions: 1. How do I know if I should hire an external evaluator? 2. Why should I hire an external evaluator? 3. What do I look for in an external evaluator? 4. How do I find an external evaluator? How Do I Know if I Should Hire an External Evaluator? During the planning stage, assess the evaluation capabilities within your agency to determine if you can use internal staff or if you need to hire an external evaluator. Exhibit IV-1 presents questions to help you decide if it is in your agency’s best interest to hire an external evaluator for a specific project. If the majority of the answers to these questions are “no,” then you should consider hiring an external evaluator to assist with planning and implementation. Even if you answered “yes,” review the sections in this guide that detail the competencies, roles, and responsibilities of the evaluator to identify the expertise within your internal team. Why Should I Hire an External Evaluator? To assist you further in making the decision, this section, including Exhibit IV-2, compares the advantages of hiring an external evaluator to the potential drawbacks of relying on internal staff. Exhibit IV-1 Assessing Evaluator Capabilities Circle Yes or No Capability Questions Does your agency have a person on staff who knows how to conduct needs assessments and/or evaluations? Yes No Does a staff person from your agency know enough about evaluation to conduct one with the support of resources (e.g., manuals, toolkits, textbooks, guides, informal partners) that are available in the field? Yes No Do staff have enough time to devote to conducting a meaningful evaluation? Yes No Does your administration (e.g., chief or executive) support using internal staff to conduct an evaluation staff? Yes No end exhibit IV-1 Despite the advantages, there are potential challenges associated with hiring an external evaluator. Among these, expense and fit are discussed here along with suggestions for dealing with these possible pitfalls. Expense. Conducting a thorough evaluation can be costly and time-consuming, especially for an evaluator who is not familiar with your community or program. To alleviate the start-up factor for an external evaluator, consider hiring someone from your community or field who has a basic understanding of the issues associated with law enforcement and your program or strategy. The 10 percent set-aside for evaluation that often is associated with grant awards can Exhibit IV-2 Advantages of Hiring an External Evaluator Compared to Using Internal Staff table Table column headers: Advantage, External Evaluator, Internal Staff Specialized Knowledge Comes with specialized knowledge and ability; understands how to assess the needs of a community, document program outcomes, and collect and analyze. Often does not have expertise in the scientific approach to evaluation. Reduces Burden Reduces the burden on the internal staff who have limited time to devote to a meaningful evaluation. May be focused on law enforcement activities and not have time to conduct a meaningful evaluation also. Objective Viewpoint Remains unbiased about the program being studied; a good evaluator will point out both positive and negative aspects of a program, offering suggestions on how best to achieve the program’s goals. Can be committed to the work and believe unequivocally in the effectiveness of the program, thus running the risk of looking at evaluation findings in a biased way. Higher Level of Credibility More knowledgeable and objective about the evaluation process; conclusions and recommendations tend to carry more weight with funding institutions, the field, and the public. Not as knowledgeable about evaluation; findings often viewed as less credible to the field. end exhibit IV-2 be used to help cover the expense. In the long run, using the 10 percent set-aside to hire an external evaluator with specific expertise could be less expensive and produce better results than trying to do the work with existing staff. Poor Match. Hiring an unqualified evaluator, or one who does not work well with your organization, can produce potentially damaging results such as alienating staff, intruding on the community and staff-community relationships, and drawing incorrect conclusions about the community and/or program. To find a proper match, take the time necessary to select the best evaluator for the job to ensure that the evaluation is not burdensome and that evaluation findings address research questions about how the program and services are working. Carefully weigh all these factors when deciding to hire an external evaluator or rely on evaluation capabilities within your agency or partnering organizations to develop and carry out the evaluation plan. What Do I Look for in an External Evaluator? After careful deliberation, if you decide to hire an external evaluator, it often is helpful to organize a hiring committee. The committee should be large enough to address the issue under study and the dynamics of your agency adequately, but small enough so decision-making is efficient and not overly troublesome. The hiring committee should report to the appropriate parties (via periodic e-mails, newsletters, meetings, etc.) to keep everyone informed of the selection process. The following sections provide guidance for considering the evaluator’s competencies, roles, and responsibilities. These also can be applied to internal staff when determining their competencies to carry out internal evaluation work. Evaluator Competencies Hiring committee members should become familiar with the qualifications necessary for an external evaluator. Evaluators should possess the following skills: Identify data that can be used for planning, managing, and evaluating program policies and practices Use multiple research methods and collect various types of data from a variety of sources Establish data-collection and analysis priorities that are informed by the experiences, capabilities, and resources of program staff and participants Produce reliable interpretation of data Link data to program development and management. In addition to these basic skills, evaluators must be willing to work within the realities of everyday program operations. Strong skills in communication, team building, group processes, and negotiation are vital to a successful relationship between the evaluator and your agency. Given the focus of your evaluation, it also is important to consider the following competencies when hiring an external evaluator: Proficiency in facilitating an evaluation process that includes the experiences of the participants Ability to build capacity in program staff to use evaluation findings Willingness to develop and maintain a collaborative relationship based on cooperation and trust with program staff, managers, and stakeholders Willingness to involve all partners in the evaluation Ability to design and implement methods to provide evaluation information to the program for use in planning, monitoring, and improvement Experience in evaluating the type of programs, activities, or strategies that your agency is implementing Experience in conducting evaluations with law enforcement agencies Experience in working in culturally diverse settings with individuals of varying backgrounds and languages. Find an evaluator with aptitude in as many of these competencies as possible. While evaluation success is not contingent on finding an evaluator with a high level of expertise in each of these competencies, it will benefit the project overall if you take time to secure an evaluator with the skills that meet your needs. Doing so will increase the probability of a meaningful evaluation. How Do I Find an External Evaluator? Now that you have a better understanding of the type of external evaluator who would be right for your project, you are ready to search for the person to fill that position. This section of the guide provides information on where to look and what to include in a job announcement. Where to Look for an External Evaluator When hiring an external evaluator, you might consider an individual or organization that either is referred to you or that you have worked with in the past. Alternatively, there are several sources you can consult to find a qualified evaluator. Potential sources, and the advantages and disadvantages of using each, are summarized in Exhibit IV-3. Exhibit IV-3 Sources for Finding an External Evaluator table Table column headers: Evaluator Source, Pros, Cons College or university Educational institutions offer the expertise of faculty members or graduate students at low to moderate overhead costs. Many small, not-for-profit organizations have found low-cost data-collection assistance through colleges or universities. With proper training, graduate students (the more inexpensive option) can create instruments and collect and analyze data. Contact college or university behavioral/social science, criminal justice, social work, or public health departments to learn if they have faculty and/or graduate students to assist you. Independent researchers not working within a larger organization might be challenged to meet the variety of expertise and time commitments to complete the job within a grant cycle. Independent researcher Researchers who work alone or with minimal staff can provide high-level expertise at low overhead costs. They commonly have the flexibility to be located on site, enabling frequent interaction with initiative participants. May be focused on law enforcement activities and not have time to conduct a meaningful evaluation. Research and evaluation organization Organizations with research and evaluation departments offer a range of expertise. They are able to meet the multiple needs of an initiative by offering experts across disciplines and having state-of-the-art technology in house. These organizations typically are associated with higher overhead costs. end exhibit IV-3 What to Include in a Job Announcement The evaluator job announcement should include an overview of your project, the roles and responsibilities of the evaluator, suggested timeline for completion of work, estimated budget information, delineation of preferred skills, and a request for applicant background and education information. Preparing the job announcement should be the responsibility of the hiring committee or at least it should be reviewed and approved by those involved in developing and managing the evaluation. A sample job announcement is provided in Appendix K. Following the application submission deadline, the hiring committee should meet to review applications and select candidates to interview. Appendix I offers examples of interview questions. After committee members select a candidate they believe is right for the job, they should solicit feedback on their recommendation from all partners. Scheduling follow-up interviews to assess the personal style and characteristics of the potential evaluator helps ensure that the individual hired is acceptable to program staff, administrators, and key stakeholders. Interview more than one applicant in case your first choice is unavailable or does not accept your offer. Once an offer is accepted, engage the evaluator as soon as possible in evaluation planning. If you already have developed an evaluation plan, review the document with the external evaluator and make any changes necessary. Once your plan is finalized, you and your evaluator are ready to proceed with implementation. Additional Resources Two additional resources for finding an external evaluator are the American Evaluation Association (AEA) and the Evaluation Center at Western Michigan University. See also Appendix L. American Evaluation Association (www.eval.org) The AEA is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. AEA provides links to help you find an evaluator and information on training opportunities. The Evaluation Center at Western Michigan University (www.wmich.edu/evalctr) The Evaluation Center’s mission is to advance the theory, practice, and utilization of evaluation. The center’s principal activities include research, development, dissemination, instruction, and national and international leadership in evaluation. The center provides service to a selected group of clients—including school districts, nonprofit agencies, government agencies, foundations, and colleges—to evaluate their programs. The center also provides evaluation training to the field. Chapter V: Emerging Topics in Evaluation Mangement Emerging Topics in Evaluation Management Three emerging topics in evaluation and program accountability are presented in this section: 1. Collaborative evaluation. An evaluation conducted jointly by more than one office, agency, or partner. 2. Participatory evaluation. An evaluation that features active involvement by all stakeholders. 3. Performance measurement. An ongoing process of documenting indicators of progress toward goal achievement. Collaborative and participatory evaluations have similarities, particularly because both involve stakeholders in the process. Performance evaluation provides an opportunity for police departments to apply program evaluation results to internal decision-making on critical issues such as staffing. Collaborative Evaluation “If the process of evaluation is carried out regularly and openly, with all group members participating, the answers they arrive at are in themselves not so important as what is learned from the discussion and from the process of reaching consensus on what questions should be used to evaluate outcomes and capacity.” (Uphoff, 1991) Many of the barriers to evaluation, including a lack of skills, tight budgets, and limited access to data, can be addressed through a collaborative approach. Such an approach entails forming a team that shares decisions about the purpose of evaluation, how it will be conducted, and how results will be interpreted and used. Generally, the team includes individuals, agencies, or organizations that have some vested interest in the program, strategy, or activity that will be evaluated. For example, if your law enforcement agency is planning to implement a drug use prevention program in schools, your collaborative evaluation team might include representatives from your agency and the funding source in addition to representatives from the local school district (administrators and teachers), parents, substance abuse service providers, and other groups that might be involved in planning or implementing the evaluation. In addition, if you do not possess the expertise to facilitate evaluation planning and implementation, be sure to include an evaluation person. The key to the collaborative approach to evaluation is that no single person, agency, or organization owns the evaluation; it belongs to the team of individuals who have come together to support it. In developing this guide, we asked law enforcement representatives if they had used collaborative approaches to evaluation before and, if so, what their experiences had been. Most reported using this approach with positive results. In fact, they claimed several benefits, including access to more financial and staff resources (across agencies), the capacity to conduct more sophisticated evaluations (because of combined talents), and the opportunity to develop relationships with various organizations, agencies, and community and neighborhood groups. In addition to those reported by local law enforcement agencies, a collaborative approach to evaluation could have other benefits. According to Practical Evaluation of Public Health Programs (The University of Texas-Houston Health Science Center, 2001), a collaborative approach to evaluation accomplishes the following: Reduces suspicion and fear. Including stakeholders in decision-making about what to evaluate and how the evaluation will be conducted is likely to prompt support, whereas those who do not participate can become suspicious about the project and fearful of the outcome. They might not trust the results or those involved in the evaluation. People often fear that evaluation will result in termination of the program or their jobs, but involving them helps them understand that the focus of the evaluation is to improve the program. Increases awareness and commitment. Through a collaborative approach, participants become more aware of, and committed to, the evaluation process. If people are involved, they have an interest in the results. A candid presentation and discussion of the program’s data can spur development of a consensus about interpretation and follow-up action. Increases the possibility of achieving objectives. If people understand what is being evaluated and why, they are more likely to work toward improving those elements of the program. The collaborative approach promotes ownership of the process and responsibility for the outcomes. Broadens knowledge. A collaborative approach draws on broader knowledge, skills, and experience in evaluation. In choosing people for the evaluation team, the role each person will play and the knowledge, skills, and experience they bring to the evaluation should be considered. This is a good opportunity to look both inside and outside the organization to tap resources needed for the type of evaluation planned. The team should count people from a variety of backgrounds, including those with front-line service experience, statistical and epidemiological expertise, management and policy perspectives, and planning skills. All these skills might not be available within the program or even the organization, so it is often helpful to rely on a community college or university in your area for assistance either from faculty or student interns. Teaches evaluation skills. When people work together, they share ideas, knowledge, skills, and abilities. Evaluation team members can learn about program objectives, data-collection methods, making evaluation decisions, and even how to work on a team. Team members learn by doing and come away from the process with an enhanced set of skills. Teaches stakeholders. A collaborative approach can teach community stakeholders about policing efforts. Involving people from outside the program increases their knowledge about your program and its role in law enforcement. This is an excellent way to involve the community in a discussion of law enforcement problems and possible interventions. For example, you might solicit participation from local elected officials and representatives of community-based organizations, all of whom would finish the evaluation with a better understanding of your program’s strengths, the constraints under which law enforcement programs operate, what is needed for improvement, and how the program relates to their interests. Increases the possibility that findings will be used. Involving stakeholders increases the possibility that the findings will be used and implemented. When a variety of staff are involved in conducting evaluations, identifying problems, and determining solutions to problems, they are more likely to accept implementation of the solutions decided on by the group. Allows for different perspectives. Recalling the old adages, “two heads are better than one” and “the whole is greater than the sum of its parts,” the collaborative approach includes people from outside the program who have unique points of view. The box below provides an example of a collaborative evaluation. As explained in this case study of the Hawthorne Huddle in Minneapolis, collaborative evaluation helped this community initiative demonstrate results, facilitate decision-making, and sustain its efforts. The evaluation managers stated that the participation of the Minneapolis Police Department in providing data throughout the implementation of the Huddle evaluation helped keep it focused on the needs of the community and the primary reason the Huddle was formed—to reduce community crime and violence. In addition, participation of the Minneapolis Police Department in Huddle activities and the evaluation helped strengthen working relationships among all participating agencies. These are only some of the benefits a law enforcement agency can gain from a collaborative evaluation. A more detailed description of the Hawthorne Huddle is on page 11. How They Did It: Minneapolis Police Department The Hawthorne Huddle, a 1999 winner of the Herman Goldstein Award for Excellence in Problem-Oriented Policing, is a community-based initiative in which citizens; the faith community; the Minneapolis Police Department; city, county, and federal agencies; nonprofit organizations; and the General Mills Foundation worked together to develop a strategy to combat crime and improve neighborhood conditions. The Hawthorne Huddle used evaluation strategies throughout the initiative to identify (e.g., reviewing crime statistics and community concerns) and analyze (e.g., ongoing review of crime statistics and other community indicators) the problem, and develop a response. Many of the evaluation activities were conducted by Huddle staff and volunteers in a collaborative manner. In the fifth year, an external evaluation was conducted that was instrumental in documenting the accomplishments, lessons learned, and future goals of the initiative. The evaluation of this community initiative was managed by the General Mills Foundation, rather than by the Minneapolis Police Department. However, the results were used by all participating organizations, including the Minneapolis Police Department, to improve Huddle meetings, decide future strategic direction, and transition to new leadership. Participatory Evaluation “When conducting [evaluation] in a collaborative mode, professionals and nonprofessionals become co-evaluators…Genuinely participatory approaches to evaluation require power sharing. One of the negative connotations often associated with evaluation is that it is something done to people. Participatory evaluation, in contrast, involves working with people. Instead of being research subjects, the people in the evaluation become co-investigators. The process is facilitated by the evaluator, but is controlled by the people in the program or community.” (Patton, 2002, pg. 183) Participatory evaluation necessitates active involvement by all stakeholders and typically has several characteristics that set it apart from traditional evaluation approaches: Participant focus and ownership. Participatory evaluations are oriented primarily to the information needs of program stakeholders rather than the law enforcement agency. The agency simply helps participants conduct their own evaluations, thus building their ownership and commitment to the results and facilitating follow-up action. Flexible design. While some preliminary planning for the evaluation might be necessary, design issues are decided in the participatory process. Generally, evaluation questions and data-collection and analysis methods are determined by participants, not outside evaluators. Participant negotiations. Participating groups meet to communicate and negotiate to reach a consensus on evaluation findings, solve problems, and make plans to improve performance. Use of facilitators. Participants actually conduct the evaluation; however, external evaluators can provide a supporting role as mentors, group processors, or methodologists. Learning process. Participatory evaluation emphasizes identifying lessons learned to help participants improve program operations and assess whether targets were achieved (USAID Center for Development Information and Evaluation, 1996). Each type of evaluation has its own strengths and limitations. Exhibit V-1 presents some of the major differences between traditional and participatory approaches to evaluation. Some may be more appropriate than others, given circumstances and information needs. For example, if objectivity and credibility are key requirements, a more traditional external evaluation might be the right choice, with a hired evaluator who develops the plan and implements it with little input from you or your staff. If stakeholder ownership and program improvement are priorities, more collaborative or participatory approaches usually are better (USAID Center for Development Information and Evaluation, 1997). Some people criticize participatory evaluations as being less objective than more traditional approaches, while others suggest that a participatory approach can interfere with rigor when the evaluation requires complex analysis. Finally, some evaluators believe these approaches require more resources than more traditional approaches, because of the time and effort associated with bringing all stakeholders into the process. Regardless of the criticisms, your agency should determine the best approach to meet its information needs. How They Did It: Boston Police Department The emergence of crack cocaine in the mid-1980s fueled an escalating crisis of youth violence in urban centers such as Boston. The Boston Gun Project was one of almost 20 community-based prevention initiatives designed to reduce youth violence that were evaluated by a participant team from Harvard University. While other initiatives focused on issues ranging from drug treatment to job placement, the Boston Gun Project involved a law enforcement problem-solving approach that specifically targeted guns. The initiative used a crackdown approach (“pulling every [legal] lever”) to focus police attention on illicit gun sellers and the children who bought those weapons. The strategy included quickly arresting high-profile gangsters in the targeted high-crime district, even for trivial gun violations, making this crackdown a hot topic of conversation on the street. The idea was to send a loud and clear message that the only way to get the police off your back was to disarm. Operation Ceasefire officially began in May 1996 and within weeks there was a dramatic decline in youth violence—overall a 63 percent drop in the monthly number of youth homicides, a 25 percent decrease in the monthly number of citywide all- age gun assault incidents, and a 44 percent reduction in the monthly number of youth gun assault incidents in the target district. At roughly the same time, the city adopted a strategic plan that included a neighborhood police initiative called “Same Cop, Same Neighborhood.” According to the Harvard researchers, in a 29-month period ending in January 1998, there had not been a single teenage homicide, something virtually unheard of in a major U.S. city at the time. Exhibit V-1 Difference between Participatory Evaluation and Traditional Evaluation table Participatory Evaluation Participant focus and ownership of the evaluation Broad range of stakeholders participate Focus is on learning Flexible design Evaluators are facilitators Tradition Evaluation Agency focus and ownership of the evaluation Stakeholders have limited participation Focus is on accountability Predetermined design Evaluators conduct the evaluation end exhibit V-1 An example of the participatory approach to evaluation comes from Harvard University’s David M. Kennedy, Anthony A. Braga, and Anne M. Piehl, who developed and implemented Operation Ceasefire in Boston. Elin J. Waring conducted the evaluation. Their report, Reducing Gun Violence: The Boston Gun Project’s Operation Ceasefire (NCJ 188741, 2001), demonstrates the benefits of having the research team immersed in the initiative from the beginning. Instead of reading like a dry research report, the publication tells a magnificent story of how the police department became an integral and essential part of a citywide effort to save children’s lives. The researchers helped design the intervention so they could determine whether any drop in gun violence was the result of Operation Ceasefire or another factor. The project launch was immediate and intense to ensure that the target community got the message. The official start date of Operation Ceasefire was May 15, 1996, the post-implementation phase began only 2 weeks later, on June 1, when the data already showed a dramatic drop in youth gun violence (see sidebar). The researchers identified and controlled for rival causal factors such as changes in the unemployment rate or the number of people classified as youth, as well as citywide trends in violent crime, homicides among older (age 25+) residents, and youth involvement in street-level drug dealing (as measured by Boston Police Department arrests). Research participants credit this new kind of police-researcher partnership with enriching the entire process, from planning through evaluation. Performance Measurement “Performance measurement is the ongoing monitoring and reporting of program accomplishments, particularly progress toward preestablished goals, and is typically conducted by program or agency management. Performance measures may address the type of level of program activities conducted (process), the direct products and services delivered by a program (outputs), and the results of those products or services (outcomes). A ‘program’ may be any activity, project, function, or policy that has an identifiable purpose or set of objectives” (Artley and Stroh, 2001). Performance measurement is an ongoing process of assessing performance indicators to determine whether a program, activity, or strategy is achieving its objectives. It differs from program evaluation in several important ways, which are presented in Exhibit V-2. The main difference between performance measurement and evaluation is that the former is management-focused, while the latter is research-focused. Each is designed to answer different questions and is implemented for different reasons. Exhibit V-2 Difference between Performance Measurement and Program Evaluation table Performance Measurement Management-focused activity Continuous activity, tracking many projects across multiple fiscal years Complexity at a minimal cost General impact of grant program efforts Evaluators are facilitators Program Evaluation Research-based activity Discrete activity, focusing on fewer projects for a limited period Complexity and cost are not major issues Specific impact of grant program efforts Evaluators conduct the evaluation end exhibit V-2 For example, a performance measurement system for a demand reduction education program in which law enforcement officers participate might include indicators such as number of juveniles in programs, number of adults in programs, and number of officers providing demand reduction training. Similarly, for multijurisdictional task force programs that integrate federal, state, and/or local drug law enforcement agencies and prosecutors to enhance interagency coordination and intelligence and facilitate multijurisdictional investigations, performance indicators might include number of offenders arrested, number of offenders prosecuted, number of drug seizures, quantity by weight (e.g., ounces, grams, dose units), drug type, and total value of funds and assets forfeited. Finally, programs designed to target the domestic sources of controlled and illegal substances, such as precursor chemicals, diverted pharmaceuticals, clandestine laboratories, and cannabis cultivation might have performance indicators such as number of offenders arrested, number of offenders prosecuted, number of labs eliminated, number of drug seizures, quantity of seizure by weight (e.g., ounces, grams, dose units), and drug type. The Colorado Springs Police Department’s (CSPD) performance monitoring system is based on the PASS (Police Accountability and Service Standards) model. Police service standards parallel the high-level organizational outcome goals for police service that the Colorado Springs community expects. CSPD service standards reflect the city’s strategic plan and the police department’s vision, mission, and values. Police service standards are described in terms of quality, quantity, cost, or time, and the seven categories are as follows: 1. Response times (e.g., response to emergency calls, answering telephone calls for police service). 2. Officer deployment (e.g., scheduling and deployment efficiency). 3. Traffic (e.g., crashes at top 25 targeted intersections). 4. Clearance rates (e.g., violent and property crimes compared with national average). 5. Drug and vice activity (e.g., response to methamphetamine labs, response to narcotics tips). 6. Neighborhood policing (e.g., time consumed in proactive policing, partnerships). 7. Citizen satisfaction with police services (e.g., customer satisfaction with officer contact, community perceptions of fear, safety, and disorder). As noted by the chief of police and the city’s planning manager: “To achieve true accountability to the community, the police and the community must define police services and the level at which the services are provided. They must also identify measures of success for those services as viewed by their local communities. And finally, the police must be open to being held accountable by the community for the measures of success agreed upon. Such an open dialogue between the police and the community will help gain the trust and confidence of all of our stakeholders, police and citizens alike” (Velez and Paine, 2002). Based on data gathered through the performance system, informed decisions are made about service delivery, staffing levels, support personnel, manpower allocation, and deployment. The performance standards should reflect and guide realistic community expectations for police services. In commenting on the PASS model, the CSPD says the following: “The other thing that this model (PASS) does is it keeps the pulse of the community. . . I think departments that don’t are missing out on a huge benefit, not only on monitoring their performance internally but also using that information to quantify why they need more officers, or that they need more police cars.” Performance measurement cannot replace data on program costs, political judgments about priorities, creative solutions, or common sense. A major purpose of performance measurement is to raise fundamental questions; the measures, by themselves, provide definitive answers. Characteristics of a good performance measurement system include the following: Quality over quantity. Performance goals should be relevant to the core mission of the program and the result the program is intended to achieve. Importance to budget decisions. Achievement of performance goals should inform funding decisions. Public clarity. Performance goals should be understandable to the users of what is being measured. Feasibility. Performance goals should be feasible and based on relevancy of the outcomes. Collaboration. Often agencies must work together to achieve a single outcome. Chapter Summary Particularly as police departments embrace community policing, the benefits of collaborative and participatory evaluations become obvious. These approaches involve strategies that engage the community in the evaluation process, building opportunities to improve communication and promote trust, in addition to improving the scope and quality of evaluations themselves. Performance measurement is proving to be a valuable tool to tie program evaluation to municipal and police department decision-making about critical issues such as budgeting and staffing. Police departments that learn about and experiment with these techniques now will be fully prepared if evaluations are mandated in the future. Police departments face mounting pressure to conduct quality evaluations and many federal agencies and private foundations want a strong commitment to evaluation demonstrated in competitive grant applications. While few municipalities so far tie funding to evaluation, this could change in the future. Police agencies that embrace evaluation now and experiment with a variety of techniques ensure that they will be ready for the future, while learning valuable lessons today. Appendixes Appendix A Sample Needs Assessment Action Plan Template 1. Who needs to be included? Chief of police Upper and middle management law enforcement personnel Front-line police officers Prosecutors Court staff Corrections officers Women’s advocacy groups. 2. What do we want to learn about our community? Rate of domestic violence within a jurisdiction. 3. What do we already know about our community? Official police data including number of calls for service and arrests. 4. Do we need to collect new data? Sources: Who will have these data? Front-line police officers who work closely with the community, professionals and volunteers who work with victims at community hot lines, personnel at domestic violence shelters and hospital emergency rooms, and batterer intervention providers. Methods: How will we gather the information? Focus groups, surveys, and interviews with beat cops. Strategies also might include targeted outreach to professional women’s groups. Instruments: What tools shall we use? Focus groups, surveys, and interview protocols. 5. How will the data be analyzed? Assess trends over time: Use data from calls for service and arrest as well as information collected from focus groups, surveys, and interviews with beat cops to determine whether the problem is getting better or worse. Use these data to examine what factors could be influencing changes over time. 6. How will the findings be reported? Create a brief write-up summarizing major findings. Communicate findings to community leaders, police personnel, and domestic violence advocacy groups. 7. How will the data be used? Gain insight into factors that could be influencing the rate of domestic violence within a community. Needs Assessment Action Plan table Table Column headers: Task/Activity, Persons Responsible, Due Date, Anticipated Result/Deliverable Ex: Identify needs assessment questions Ex: Law enforcement middle manager Ex: October 15, 2006 Ex: List of needs assessment questions Anticipated Needs Assessment Budget $__________ Appendix B Preparing Measurable Goals and Objectives The ABCDEs of writing measurable goals and objectives cover the who, what, to what degree, and by when information of your program. Audience—Who? The population/target audience for whom the desired outcome is intended. Behavior—What? What is to happen? A clear statement of the behavior change/results expected. Condition—By when? The conditions under which measurements will be made. This may refer to the time frame and/or implementation of a specific intervention. Degree—By how much? The quantification of, or the level of, results expected. This often involves measuring change compared to an identified baseline. Evidence—As measured by? The definition of the method of measuring expected change. The degree of change (set forth above) will be measured using a specific instrument or criterion. Using the ABCDE method, these are the steps for developing goals and objectives: To develop measurable goals— Step 1: Identify the longer term, global outcomes you wish to achieve. Step 2: Identify each element (A, B, C, D, E). Step 3: Formulate the goal statement using each necessary element. To develop measurable objectives— Step 1: Identify the shorter term, more immediate outcomes you wish to achieve. Step 2: Identify each element (A, B, C, D, E). Step 3: Formulate the objective statement using each necessary element. Sample Worksheet for Writing Measurable Program Goals and Objectives Purpose: To develop measurable goals and objectives using the ABCDE method. Directions: 1. Identify the outcomes/impacts you wish to achieve. 2. Identify each element of your goal and objective (A, B, C, D, E) 3. Formulate the goal and objective statements using each necessary element. GOAL: The longer term, global effects you wish to achieve. Identify the elements: A. Audience is… Citizens within a community. B. Behavior to be changed is… Community response to domestic violence. C. Conditions under which change will occur is (are)… Greater communication and collaboration among community organizations that work with domestic violence victims and offenders. D. Degree of change expected is… Enhanced safety of domestic violence victims and their families. E. Evidence of change is… Decreased domestic violence mortality rate within a community. Goal Statement: To implement a Domestic Violence Enhanced Response Team (DVERT) to improve the safety of citizens within a community. OBJECTIVES: The more immediate, direct results you wish to achieve. Identify the elements: A. Audience is… Domestic violence victims and offenders and the agencies that work with these populations. B. Behavior to be changed is … Community response to high-risk-for- lethality domestic violence victims. C. Conditions under which change will occur is (are)… Enhanced coordinated community response to reports of domestic violence by creating a collaborative made up of agencies that work with domestic violence victims and offenders. D. Degree of change expected is… Improve response time to reports of domestic violence and increase communication and collaboration among partners. E. Evidence of change is… Appropriate containment of high-risk domestic violence offenders, improved response time to reports of domestic violence, faster intervention by domestic violence service providers. Objective Statements: To increase communication and coordination between law enforcement, domestic violence service providers, batterer intervention service providers, the courts, probation and parole, and the district attorney’s office. To produce greater batterer accountability by increasing batterer enrollment and attendance in offender treatment. To connect domestic violence victims with services and resources immediately following a domestic violence incident. Appendix C Planning Model Template Draw seven columns on a sheet of paper and name them Column A, B, C, D, E, F, G. Step 1: In Column A, list all background factors and conditions you think might influence the relationship between your program activities and goals. For example, lack of knowledge about the rights and needs of domestic violence victims would be a background factor. Step 2: In Column B, list program inputs. For example, the amount of funds that will go toward increasing the knowledge of the rights and needs of domestic violence victims would be an input. Step 3: In Column C, list program activities. For example, training law enforcement officers would be an activity. Step 4: In Column D, list program outputs. For example, the number of law enforcement officers trained on the rights and needs of domestic violence victims would be an output. Steps 5 and 6: In Columns E and F, list all events occurring during or after your program activities that could affect how or whether you accomplish your goals. For example, an immediate outcome of the law enforcement officer training would be a better understanding of domestic violence. An intermediate outcome among law enforcement would be improved ability to assist domestic violence victims in obtaining a personal protection order. Step 7: In Column G, list program goals. Listing program goals helps you see the overall effects of your inputs as well as the changes resulting from them. For example, one goal might be to decrease the incidence of domestic violence in your county. It is best to start with short-term goals because it often is difficult to measure and document long-term ones. Sample Planning Model Template Conditions Problem or need: Improve community response to incidents of domestic violence Inputs What we invest: Time, money, and staff. Activities What strategies or services comprise our program? Collaborative made up of police officers and victim/offender service providers. Training for police officers on how to respond to incidents of domestic violence. Outputs What are the direct products of our program activities? DVERT collaborative meetings held two times/month. 150 police officers trained on ways to respond to reports of domestic violence. Outcomes – Impacts Immediate, Intermediate, Long-term Short-term changes we expect: Increased knowledge among police officers on ways to respond to domestic violence. Medium-term changes we expect: Increased communication and collaboration between partners involved with the project (e.g., police officers and victim and offender service providers). Long-term changes we expect: Increasing batterer enrollment and attendance in offender treatment. Connecting domestic violence victims with services and resources immediately following a domestic violence incident. Contextual Factors 1. Agency budgets. 2. Staffing. 3. Political factors. 4.Community interest or media attention to this issue. 5. Legislation. Appendix D Community Satisfaction Survey The Fictitious Police Department wants to know about your attitudes, opinions, and experiences regarding our performance. All responses will remain confidential. Thank you for your time and support. Please circle the answer closest to your views: 1. Do you feel the Fictitious Police Department has improved its police service during the past 5 years? Yes No NA 2. How would you rate the competence of the Fictitious Police Department? Excellent Good Fair Poor NA 3. Do you feel that Fictitious police officers act in a professional and courteous manner? Yes No NA 4. Are Fictitious police officers actively involved in working with citizens to make neighborhoods safer? Yes No NA 5. How satisfied are you overall with Fictitious police service? Very satisfied Somewhat satisfied Neither satisfied nor unsatisfied Somewhat unsatisfied Very unsatisfied NA 6. How safe do you feel in your home? Very safe Somewhat safe Neither safe nor unsafe Somewhat unsafe Very unsafe 7. How safe do you feel on the street? Very safe Somewhat safe Neither safe nor unsafe Somewhat unsafe Very unsafe 8. Have you or a member of your family ever been a victim of a violent crime? Yes No NA 9. Have you or a member of your family ever been a victim of a property crime? Yes No NA 10. Have you interacted with a Fictitious police officer in the past year? Yes No NA If yes, please check all that apply: _ Victim of crime _ Traffic stop _ I called police because of an emergency/crime _ Others called police to my home _ Neighborhood/school meeting _ Other (please explain) _________________________________ 11. Are you? Male Female 12. Please circle your age category: Under 18 18-24 25-34 35-44 45-54 55-64 65 and up 13. How would you characterize your neighborhood? High crime Low crime Average level of crime NA 14. Would you be willing to volunteer your time and talents to work with the Fictitious Police Department on community-based efforts to make our neighborhoods safer? Yes No NA If yes, please call xxx-xxx-xxxx or visit our web site at www.fictitiouspd.gov to find out about community meetings in your area and to volunteer. Appendix E Sample Focus Group Checklist Focus Group Checklist Consider the following items when you plan focus groups and develop focus group guides. Advance Notice _ Contact participants 1 to 2 weeks before the session. _ Send each participant a letter of invitation. _ Give the participants a reminder prior to the session. _ Over-recruit by three to five participants the number of participants necessary for each focus group. Logistics Make sure the room is satisfactory (size, tables, comfort, lighting, temperataure, etc.) _ Arrive early to make necessary changes. _ Eliminate background noise as much as possible. _ Bring nametags for participants. _ Bring extra supplies like tapes, batteries, note pads, and extension cords. _ Seat loud/disruptive participants next to the moderator. _ Seat shy and quiet participants directly across from the moderator. _ Serve refreshments before discussions begin. _ Bring enough copies of handouts and/or visual aids. _ Set up a tape recorder and make sure that it works. Questions Introductory questions should be answered quickly and not identify personal characteristics of participants. _ Questions should flow in a logical sequence. _ Key questions should focus on the critical issues of concern. _ Consider probe or follow-up questions. _ Use open-ended questions. _ Provide a summary of the discussion and invite comments. Moderator Skills _ Be well rested and alert for the focus group session. _ Practice the introduction without referring to notes. _ Make sure participants are comfortable and relaxed. _ Ask questions with minimal reference to notes. _ Be careful to avoid head nodding (i.e., showing support or favoritism for answers). _ Avoid comments that signal approval (i.e., “Excellent,” “Great,” “Wonderful”) _ Avoid giving personal opinions. Immediately After the Session _ Prepare a brief written summary of key points as soon as possible. _ Check to see if the tape recorder captured the comments. Appendix F Instrument Development Checklist Consider the following items when you develop your evaluation instrument. Instrument Title _ Use clear and concise words. _ State the program name. _ Indicate the type of service provided (e.g., counseling, shelter). _ Reflect the instrument method and content (e.g., survey, interview). Introductory Statement _ Include information about the instrument’s purpose. _ Include information about how the data will be used. _ Include information about the level of confidentiality that will be arranged (e.g., who will see their responses, how responses will be reported). Always provide the “confidentiality” you promise. Demographics _ Include questions that ask respondents for relevant information about themselves and their backgrounds (e.g., victim’s name, country of origin, and language). _ If necessary, include questions that ask about the person administering the instrument, if not you or program staff (e.g., observer’s name). _ If appropriate, identify the length of respondent participation in the program. Directions _ Include general directions on how to complete the instrument itself (e.g., when, where, and how to return the instrument). _ Include specific directions on how to complete each section of the instrument. _ Make sure specific directions appear before each section. Questions _ Use language that respondents understand (e.g., “way to teach” vs. “pedagogical”). _ Avoid “double-barreled” questions (e.g., “Has there been an increase in program resources and the number of program participants?”). _ Allow enough space for respondents to write when using open-ended questions. _ Avoid biased and value-laden words or phrases. _ Include only questions asking for needed information. _ Keep question and answer options on the same page. _ Allow space for comments, concerns, or suggestions. Format _ Use icons or graphics as clarifiers (e.g., “Please place a check in the appropriate box.”). _ Use a clearly legible font (e.g., Arial, Times New Roman). _ Lay out text and graphics using an entire page. _ Allow enough space between questions. _ Develop an instrument that it is pleasing to the eye (e.g., not “busy”). _ Indicate the date of instrument administration. _ Identify whether it is a pre-, post-, or ongoing survey. _ Note the name of the program/organization that developed the instrument in the footer (at the bottom of the page). _ Include a computer file location path of where to find a copy of the instrument in the footer (e.g., X:\Trafficking Victim Survey), if appropriate. _ Include the date of each new version in the header/footer. Pilot Testing _ Clearly label a draft instrument “DRAFT.” _ Be mindful that advance permission to conduct the pilot test might be necessary. _ Arrange for pilot test participants and conditions to be as close to actual administration conditions as possible (e.g., time of day, location, methods, respondents). Source: The University of Texas at Arlington, School of Social Work. (n.d.). Instrument development checklist. Retrieved June 10, 2004 from www2.uta.edu/sswmindel/S6324/Class%20Materials/measurement/instrument development_checklist.htm Appendix G Topics to Cover in an Informed Consent Statement for Research Participants Invitation to Participate and Description of Project _ Introduce yourself and describe the scope and purpose of your research. _ Explain what agencies or groups are involved in funding and/or conducting the evaluation. _ Tell the reader who is being targeted for participation in the research project. Risks and Inconveniences _ Describe the risks associated with participating in the research, if any. _ Tell participants how much of their time will be required if they choose to participate. Benefits _ Explain why the research is important. _ Note the benefits of participating in this research. Description of Procedures _ Tell respondents what to expect if they agree to participate (e.g., how many interviews will be conducted with each person, where and when the focus group will be held; how much time it will take, whether any follow- up will be necessary). Voluntary Participation and Withdrawal _ Make sure respondents know that participation is voluntary and they can withdraw at any time without penalty. “Your participation is completely voluntary. You are free to choose not to participate in the study or to stop your participation at any time. You don’t have to answer any questions that you don’t want to, and you can withdraw your participation at any time without consequences or penalties.” Confidentiality _ Explain the steps you will take to protect the confidentiality of respondents. “The information you provide will be kept confidential and will be used only for research purposes. We will not connect your name with your responses to the research questions. Research team members will not release any of the information you provide on this survey outside the confines of this research project. Your responses to the interview questions will be kept in a locked cabinet and will be seen only by the research team members. Your answers will be grouped with the other responses and summarized later in a report.” “Also, we would like your permission for the interview to be audiotaped. Please understand that the transcripts will be used only for research purposes and that no names will be associated with your transcript or audiotape.” Questions Please feel free to ask any questions and to think about whether you want to participate before you make a decision. Do you agree to participate in this research study? 0. No 1. Yes Can we audiotape this interview? 0. No 1. Yes If you have further questions about this project or if you have a research-related problem, you may contact… Appendix H Sample Evaluation Plan Template 1. Focus What Will We Evaluate (Which Program Or Aspect Of A Program)? Examine whether a police department’s Domestic Violence Enhanced Response Team (DVERT) is enhancing the safety of high- risk-for-lethality victims. 2. Questions What Do You Want To Know? 1. Is DVERT improving safety of children and families? 2. Is DVERT helping to ensure appropriate containment of high-risk offenders? 3. Indicators- Evidence How will we know it? 1. a. Client satisfaction surveys b. Interviews with DVERT partner agency staff (e.g., child welfare and domestic violence agencies) and focus groups with clients. 2. a. Case records from District Attorney’s Office b. Interviews with DVERT partner agency staff (e.g., child welfare and domestic violence agencies) and focus groups with clients. 4. Timing When should we collect data? Work with DVERT program management to determine a timeline for data collection that will not interfere with program activities. 5. Data Collection Sources Who will have this information? Meet with District Attorney’s Office to determine accessibility of case records. Methods How will we gather the information? Client satisfaction surveys Interviews with DVERT partner agencies District Attorney’s Office Case Record Review Focus groups with DVERT clients. Sample Who will we question? Work with DVERT program management to identify staff to participate in interviews and staff who will be responsible for administering surveys to clients. Instruments What tools shall we use? Develop client satisfaction survey protocol, interview protocol, and focus group protocol. 6. How will the data be analyzed? Data from case records and satisfaction surveys will be analyzed using quantitative methods (e.g., descriptive analysis such as frequencies distributions and cross-tabulations). Data from interviews and focus groups will be analyzed using qualitative methods (e.g., descriptive analysis such as thematic coding). 7. How will the data interpreted? Compare analyses from surveys, interviews, focus groups, and case reviews. 8. How will the results be communicated? To Whom? DVERT program manager High-level law enforcement administrators Beat cops. When? Where? How? Create a brief summary of findings to be discussed during role calls and during DVERT management meetings. Appendix I Sample Interview Questions Checklist The following checklist contains a series of questions to help you interview and select an external evaluator. The questions are designed to help you judge the qualifications and characteristics of prospective evaluators, including basic training in evaluation, previous evaluation experiences, and some personal characteristics that fit your initiative and the community in which you will be implementing your program. In addition to the checklist, it will be important to look at work samples and contact references for any prospective evaluator. Interview Questions Evaluator appears to be: (Check one for each item) Well-Qualified Not Well- Qualified Cannot Determine What formal training have you had in conducting needs assessment studies? What practical experience (ask for examples of needs assessment studies they have led or directed)? What formal training have you had in conducting evaluation studies? What practical experience (ask for examples of studies they have led or directed)? What experience do you have in evaluating policing programs? What experience do you have in conducting evaluations of community initiatives? What experience do you have in integrating evaluation activities with community planning processes? What experience do you have in conducting evaluations with law enforcement (you may also ask about their experience working with any other agencies with whom you are partnering and with whom they would have to work)? How familiar are you with the issue of _____ (insert the program/strategy area you will be evaluating, e.g., domestic violence)? What indicators would you use for this type of evaluation? Source: Adapted from the W. K. Kellogg Foundation Evaluation Handbook and Evaluation Handbook for Projects Funded by S.T.O.P. Formula Grants. Appendix J Glossary Archival data – Information that has already been collected by a department or organization, such as a government or nonprofit agency. These documents may include program records or materials such as proposals, annual or monthly progress reports, budgets, organization charts, memoranda, policies and procedures, operations handbooks, and training materials. Baseline – Initial data on program participants or other program aspects collected prior to receipt of services or program intervention. Baseline data are often gathered through intake interviews and observations and are used later for comparing measures that determine changes in participants or a program. Causality –A cause and effect relationship. The causality of two events describes the extent to which one event is caused by the other. When there is causality, there is a measure of predictability between the two events. Cleaning data – The process by which quantitative data are reviewed to identify missing or incorrect responses. Comparison group – Individuals whose characteristics are similar to those of program participants or the treatment group. The individuals in the comparison group may not receive any services, or they may receive a different set of services, activities, or products. The experimental (or treatment) group and the comparison group are assessed to determine which type of services, activities, or products provided by a program produced the expected changes. Control group – A group of individuals whose characteristics are similar to those of program participants or an experimental group, but do not receive the program being evaluated. Participants are randomly assigned to either the treatment (or program) group or the control group. A control group is used to assess the effect of a program on participants compared to similar individuals not receiving the activities being evaluated. Cost benefit – An assessment of whether program costs are achieving the intended program impact. This analysis compares costs and benefits using the same monetary units. Determining the dollar value of a benefit is arguably the primary challenge in cost-benefit analysis. Cost effectiveness – An assessment of whether money expended on a program was well-spent in reaching the program outcomes. This is the relationship between agency costs and its effectiveness in its mission. Data – Information or facts that are collected to answer specific research questions. Document review – A process of data collection that involves examining existing records or documents. Evaluability assessment – A process used to identify whether a program is ready to be evaluated. This is an assessment of whether or not a program has clearly defined activities and measurable goals and objectives that lend themselves to identifying data that can be collected. This process includes reviewing the program history, design, and operation, perhaps by watching the program in action; determining the program’s capacity for data collection; and assessing the likelihood that the program will meet its goals and objectives based on the activities that are actually being implemented. Executive summary – A condensed version of a longer report’s content. Executive summaries are written for someone who most likely does not have time to read a lengthy report. An executive summary may be called an abstract when it accompanies a scholarly document. Experimental design – Individuals are randomly assigned to a test and control group and both are measured before and after the intervention, then the results are compared and analyzed. External evaluation – An external is conducted by an agency or individuals not directly involved in or responsible for the program or activities being evaluated. Evaluation design – The overall plan and approach for an evaluation. A description of how you plan to measure program components and how you plan to use the resulting measurements. The evaluation design is determined by the type of evaluation that is used and on the basic questions you want answered about your program. Evaluation plan – A description of the overall approach or design used to guide an evaluation. The evaluation plan should include information on what evaluation activities will take place, a methodology for executing these activities, a timeline for completing planned evaluation tasks, a list of the individuals who will be responsible for completing each evaluation task, and a description of the purpose of the evaluation. Formal evaluation – Rigorous and comprehensive assessments of data that are used to make decisions within a department, agency, or organization. Frequency – A measure used to determine the number of times a given response occurs. This measure can be expressed using whole numbers or percentages. Goal – A measurable statement of the desired long-term impact of the program. History – Any event that happens between the time of your first measurement (pre- test) and your second (post-test) that can affect the measurement. This can pose a threat to the validity of evaluation results. Impact evaluation – This evaluation is designed to assess whether a program or intervention had a long-term impact on a specified goal. Informal evaluation – Less rigorous or comprehensive assessments of data that are used to make decisions within a department, agency, or organization. Instrumentation – Changes over time can take place in the instruments used to measure participants’ performance. Changes caused by instrumentation can threaten internal validity if they cannot be separated from the effect of the treatment. Any flaws in a measurement device can skew evaluation results. Internal evaluation – An internal is conducted by the agency or individuals who are directly involved with or responsible for the program or activities being evaluated. Logic model – A graphic representation that clearly identifies and lays out the logical relationships among program conditions, activities, outcomes (objectives), and impacts (goals). Maturation – This describes the normal processes of development that occur over time that could affect evaluation outcomes independent of the evaluator’s intervention, and can pose a threat to the validity of evaluation results. Mean – An average score that is calculated by dividing the sum of all responses to a particular question by the number of responses to that question. Median – The midpoint or value below which half the values in a distribution fall. This score can be generated by arranging all responses in a list. The middle number is the median. If there are two middle numbers, then add the two numbers and average them for a median. Mixing methods – An approach that consists of intentionally combining different evaluation tools and techniques, not only to observe and to gather quantitative and qualitative information, but also to structure and analyze this information Mode – the number or value that occurs most often in a distribution of values. Multiple methods – See definition of mixing methods. Needs assessment – This evaluation is designed to assess problems in the community and perceptions about why they occur and persist. Objective – A specific, measurable statement of the desired immediate or direct outcomes of the program that support the accomplishment of a goal. Outcome – Refers to a change that results from a program or its activities. Outcome evaluation – This evaluation is designed to assess whether the intervention achieved its stated short-term goals and objectives. Outcome data – Information that is collected to measure the effectiveness of a program. Output – Units of service that are provided by the program. Post-test only design – A type of evaluation design in which the target group is measured only after the intervention and those results are analyzed. Pre- and post-test design – A type of evaluation design in which the target group is measured before and after the intervention and those results are compared and analyzed. Pre- post- post-test design – A type of evaluation design in which the target group is measured before the intervention, immediately afterward, and then at a later time and those results are compared and analyzed. Process data – This evaluation is designed to assess whether the intervention was implemented as intended and, if not, why. Process evaluation – This evaluation is designed to assess whether an intervention was implemented as intended and, if not, why. Program assessment – A disciplined way of assessing the merit, value, and worth of projects and programs. Qualitative data – Data that are collected through activities such as observations, interviews, focus groups, and conversations. These data describe the attributes of an object without referring to quantity and are expressed in words. Instead of measuring the extent and direction of change or program impact, qualitative data are better suited to describing the nature of the change or impact that has taken place. Qualitative data are used frequently in process evaluations to describe how a program functions. Quantitative data – Data that are a measurement of something that can be counted and assigned a number. These data can be collected by using several methods, including simple counts, surveys, and tests. Quasi-experimental design – An evaluation design in which the target group and a similar group used as a control are measured before and after the intervention, then the results are compared and analyzed. Questionnaire – See the definition for survey. Random assignment – The assignment of individuals in the pool of all potential participants to either the experimental (treatment) or control group in such a manner that their assignment to a group is determined entirely by chance. Reliable data – The extent to which a measurement produces consistent results over repeated observations or administrations of the instrument under the same conditions each time. Research questions – A data-collection tool used to gather information on the focus of an evaluation effort. Selection bias – Any determination, except random placement, by which you place some subjects in a treatment group, and other subjects in a control group. Subjects assigned to groups in any way other than random placement opens the possibility that the groups differ in important ways that can affect the results of a study. Stakeholder – People who will be affected by a project or can influence it but who are not directly involved with doing the project work. Statistical regression – A general tendency for extreme scores on any measure to move toward a less extreme score at a different time. Statistical regression occurs when study participants are selected based on their extreme scores. When an evaluator re-measures, there is a tendency for these extreme scores to move back toward the overall group average regardless of the effects of any intervention provided. Strategic management – The process of specifying an organization’s objectives, developing policies and plans to achieve these objectives, and allocating resources to implement these plans. Survey – A data-collection method that involves collecting information directly from individuals. Surveys can be self-administered, conducted face-to-face, over the telephone, or mailed. Target group – Also called a treatment group, a target group is composed of individuals receiving the activities or interventions that you are evaluating. Testing – The process of test taking can pose a threat to the validity of evaluation results. Taking a pre-test may influence the behavior of subjects as much or more than the intervention does. The evaluator cannot be certain that the effects they see are because of the intervention and not caused by the administration of a pre- test. Treatment group – Also called a target group, a treatment group is composed of individuals receiving the activities or interventions that you are evaluating. Valid data – Data that are collected using a measurement instrument or test that accurately measures what they are supposed to measure. Appendix K Sample Job Announcement Overview of Project The Fictitious Police Department is seeking the services of a qualified evaluator to assist in an assessment of its 1-year project to reduce drug dealing in the city’s public housing developments. The department has been awarded a grant from the U.S. Department of Justice’s Bureau of Justice Administration and the U.S. Department of Housing and Urban Development’s Office of Resident Initiatives to explore how police can engage community coalitions in targeted efforts to reduce the open distribution and sale of illicit drugs. The grant includes funding for an evaluation designed to assess the impact of various strategies on reducing the sale of drugs to residents of the complex and also how those strategies affect sales to outsiders who come there to purchase drugs. Evaluator Roles/Responsibilities The evaluator is responsible for developing and implementing the evaluation plan, including the methodology, types of data be gather and analyze, and the management plan for completing the evaluation according to the guidelines set forth in the grant agreement. The evaluator will work closely with top command in charge of the project to develop a task list and timeline for the department’s participation in the evaluation process. Tasks and Timeline Applicants must outline an evaluation plan designed to produce the final report by the end of the 1-year grant-reporting period, as well as fulfill the quarterly reporting requirement specified in the grant agreement. Applicants must describe how they will meet the grant requirements, including a detailed discussion of the tasks and who will carry them out within the timeline detailed in the grant agreement. Management Plan and Key Personnel Applicants also must provide a proposed management plan designed to ensure that a comprehensive and valid evaluation will be completed within the timeline. The plan also must include a description of the roles and responsibilities of key personnel, as well as the role and responsibilities of the evaluator in oversight. Preferred Skills and Experience Applicants must demonstrate appropriate skills and experience in developing and implementing an evaluation strategy to assess a police initiative in a real-world setting. Applicants must demonstrate strong program evaluation and impact evaluation skills, in addition to the management skills required to complete a complex and multifaceted assessment within a specified time. Background/Education Applicants must submit a current curriculum vita or résumé for the lead evaluator and additional key personnel, as well as a brief description of relevant experience in police evaluation. Preferred candidates will exhibit educational credentials that certify broad experience in evaluation methodology and implementation, as well as the practical skills required to work alongside police in a diverse environment. Complete applications along with references and examples of relevant work should be submitted to [Contact Person] by [Due Date] at [Address of the Agency]. Appendix L Evaluation Resources Evaluation resources with practical applications for people with little or no evaluation experience Eck, John E. Assessing Responses to Problems: An Introductory Guide for Police Problem- Solvers. Washington, D.C.: U.S. Department of Justice Office of Community Oriented Policing Services, 2002. www.cops.usdoj.gov/txt/pop/e08064504.txt Eck, John E. “Learning from experience in problem-oriented policing and situational prevention: The Positive Functions of Weak Evaluations and the Negative Functions of Strong Ones.” In Evaluation for Crime Prevention, Nick Tilley, ed. Crime Prevention Studies, 14. Monsey, New York: Criminal Justice Press, 2002. Patton, M.Q. Qualitative Evaluation and Research Methods (2nd ed.).   Beverly Hills, California: Sage Publications, 1990. Rossi, P.H. and H.E. Freeman. Evaluation: A Systematic Approach (5th ed.). Newbury Park, California: Sage Publications, 1993. Salant, P. and D. Dillman, D. How to Conduct Your Own Survey. New York: Wiley Publishers, 1994. Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement Aagencies. Washington, D.C.: U.S. Department of Justice, Bureau of Justice Statistics and the Office of Community Oriented Policing Services, 1999. www.ojp.usdoj.gov/bjs/pdf/ccspglea.pdf Web sites that offer useful information on evaluation resources, tools, and articles: American Evaluation Association: The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. www.eval.org Bureau of Justice Assistance (BJA) Evaluation Web Site: The evaluation web site, produced by the BJA Center for Program Evaluation, provides state administrative agency staff, criminal justice planners, researchers and evaluators, and local practitioners with a variety of resources for evaluating criminal justice programs. http://www.ojp.usdoj.gov/BJA/evaluation/index.html Center for Problem Oriented Policing: The Center for Problem-Oriented Policing is a nonprofit organization composed of police practitioners, researchers, and universities dedicated to advancing the concept and practice of problem-oriented policing in open and democratic societies. www.popcenter.org COPS Office: The Office of Community Oriented Policing Services (the COPS Office) was created as a result of the Violent Crime Control and Law Enforcement Act of 1994. As a component of the Justice Department, the mission of the COPS Office is to advance community policing in jurisdictions of all sizes across the country. COPS provides grants to tribal, state, and local law enforcement agencies to hire and train community policing professionals, acquire and deploy cutting-edge crime-fighting technologies, and develop and test innovative policing strategies. The COPS Office also offers publications, CD-ROMs, videos, toolkits, and training resources to law enforcement agencies nationwide. www.cops.usdoj.gov International Association of Chiefs of Police (IACP): The IACP is the world’s oldest and largest nonprofit membership organization of police executives, with more than 19,000 members in more than 89 different countries. Its goals are to advance the science and art of police services; develop and disseminate improved administrative, technical, and operational practices; foster police cooperation and exchange of information; bring about recruitment and training of qualified persons in the police profession; and encourage adherence of all police officers to high professional standards of performance and conduct. www.theiacp.org/ Juvenile Justice Evaluation Center Online (JJEC): The JJEC, a segment of the Justice Research and Statistics Association, is a tool for assisting juvenile justice practitioners, policymakers, and state agency administrators with the assessment and evaluation of programs and initiatives. JJEC Online is composed of several sections, including JJEC Information, State Information, Juvenile Justice Evaluation Program Areas, Evaluation Resources, State Information, and Technical Assistance and Training. www.jrsa.org/jjec/index.html Police Executive Research Forum (PERF): PERF, a national organization of progressive police executives, is dedicated to improving policing and advancing professionalism through research, public policy debate, provision of management services and executive development, training, and publishing. www.policeforum.org Police Foundation: Since its founding in 1970, the Police Foundation has conducted seminal research in police behavior, policy, and procedure. The foundation has established and refined the capacity to define, design, conduct, and evaluate controlled experiments testing ways to improve the delivery of police services. www.policefoundation.org/ Technical assistance resources: The American Evaluation Association’s (AEA) web site contains a list of individuals and firms that are available for evaluation consulting. www.ovcttac.org/aboutus.cfm The Office for Victims of Crime Training and Technical Assistance Center’s (OVC TTAC) training and technical assistance activities are coordinated through three core functions: Needs assessment. OVC TTAC uses a variety of measurement tools—surveys, stakeholder discussions, and literature reviews—to assess the needs of key constituencies and to identify available technical assistance and training resources to support their needs. Capacity building. By developing and delivering training and technical assistance resources, OVC TTAC helps organizations foster professional development of their staff, enhance services to their communities, and reach unserved and underserved victims of crime. Evaluation. OVC TTAC monitors customer satisfaction and measures the resulting changes of its training and TA activities over time. www.ovcttac.org/aboutus.cfm The What Works Clearinghouse provides a “Registry of Outcome Evaluators” that allows web site users the opportunity to search for evaluators (both individuals and organizations) who conduct research on the effects of educational interventions. The Registry allows users to search by geographic region, individual/organization, content area, and target populations. www.whatworks.ed.gov/technicalassistance/EvlSearch.asp Examples of evaluation reports related to policing: Braga, A., D. Kennedy, A. Piehl, and E. Waring. Reducing Gun Violence: The Boston Gun Project’s Operation Ceasefire. Washington, D.C.: U.S. Department of Justice, National Institute of Justice, 2001. Lurigio, A.J. and W.G. Skogan. “Winning the Hearts and Minds of Police Officers: An Assessment of Staff Perceptions for Community Policing in Chicago,” Crime & Delinquency, 40 (3) (1994): 315–330. Independent Commission on Policing for Northern Ireland. (1999). A New Beginning: Policing in Northern Ireland. Retrieved November 24, 2004 from www.belfast.org.uk/report.htm Evaluation journals: American Journal of Evaluation The Evaluation Exchange Evaluation and Program Planning, ISSN 0149-7189 The Evaluation Review New Directions for Evaluation Practical Assessment Research and Evaluation, a peer-reviewed electronic journal, ISSN 1531-7714 Technical reports and articles: Burt, M.R., et al. Evaluation Guidebook. Washington DC: National Institute of Justice, Office of Justice Programs, U.S, Department of Justice, Urban Institute, 1997. Clark, R.V. and J. Eck. Becoming a Problem Solving Crime Analyst in Fifty-Five Small Steps. Jill Dando Institute of Crime Science. Ad Hoc Publications, 2002. Coldren, J.R., T. Bynum, and J. Thome. Evaluating Juvenile Justice Programs: A Design Monograph for State Planners. Office of Juvenile Justice and Delinquency Prevention, 1991. www.jp.usdoj.gov/BJA/evaluation/guide/documents/documents/documentg.html The Compiler. (1995). Evaluation as an Important Tool in Criminal Justice Planning. Retrieved January 20, 2004. Eck, J. Assessing Responses to Problems: An Introductory Guide for Police Problem-Solvers. Washington, D.C.: U.S. Department of Justice Office of Community Oriented Policing Services, 2002. www.cops.usdoj.gov/txt/pop/e08064504.txt /www.cops.usdoj.gov Joppe, M. The Research Process. Ontario Hostelry Institute, AMEX Canada, and Ryerson University, n.d.. Retrieved January 19, 2004, from www.ryerson.ca/~mjoppe/ResearchProcess/RequestforProposal.htm Juvenile Justice Evaluation Center. Hiring and Working with an Evaluator: Program Evaluation Briefing Series No. 2. Washington, D.C.: Justice Research & Statistics Association, 2001. Juvenile Justice Evaluation Center, Justice Research and Statistics Association. Evaluability Assessment: Examining the Readiness of a Program for Evaluation. Program Evaluation Briefing Series No. 6. Washington, D.C.: Office of Juvenile Justice and Delinquency Prevention, 2003. www.jrsa.org/jjec/about/briefing_evaluability-assessment.html Melvin M., G.T. Henry, and G. Julnes. Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs. San Francisco: Jossey-Bass, 2002. Muraskin, L.D. Understanding Evaluation: The Way to Better Prevention Programs. Washington, D. C.: U.S. Department of Education, 1993. www.jrsa.org/jjec/resources/publications/westat-handbook.pdf Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. “Juvenile Mentoring Program: A Progress Review,” Juvenile Justice Bulletin. Washington, DC: U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention, 2000. Priest, S. “A Program Evaluation Primer,” Journal of Experiential Education, 24 (1) (2001), 34–40. Sechrest, L., and A.J. Figuerdo. “Program Evaluation.” Evaluation Group for Analysis of Data, Department of Psychology, University of Arizona. Annual Review of Psychology. 44 (1993), 645–74. Shadish, Jr., W.R., T.d. Cook, and L.C. Leviton. Foundations of Program Evaluation, Theories of Practice. London: Sage Publications, 1995. Taylor-Powell, E., S. Steele, and M. Douglah. Planning a Program Evaluation (G-3658-1). Madison: University of Wisconsin-Extension, Cooperative Extension, 1996. Todevski, D. Public Administration Evaluation. Ministry of Finance, Republic of Macedonia, 2001. United States Air Force Family Advocacy Program. A guide to family intervention and prevention program evaluation: Conducting an evaluability assessment. In G. K. Kantor, K. Kendall-Tackett, C. M. Allen, eds. Durham: University of New Hampshire, 2000. United States Department of Health and Human Services, Administration for Children & Families. The Program Manager’s Guide to Evaluation. United States Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Prevention. Achieving Outcomes: A Practitioner’s Guide to Effective Prevention. Washington, DC, 2002. www.modelprograms.samhsa.gov/pdfs/AchievingOutcomes.pdf Public Technology, Inc. Program Evaluation and Analysis: A Technical Guide for State and Local Governments. Washington, D.C.: U.S. Department of Housing and Urban Development, Office of Policy Development and Research, 1978. United States Department of Housing and Urban Development, Office of Policy Development and Research. A Guide to Evaluating Crime Control of Programs in Public Housing. Prepared for the United States Department of Housing and Urban Development by KRA Corporation. Washington, D.C., 1997. United States General Accounting Office. Designing Evaluations. Washington, D.C.: United States General Accounting Office, Program Evaluation and Methodology Division, 1991. Weidman, D. R., et al. Intensive Evaluation for Criminal Justice Planning Agencies. Washington, D.C.: U.S. Department of Justice, 1975. Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement Agencies. U.S. Department of Justice, Bureau of Justice Statistics and the Office of Community Oriented Policing Services. www.ojp.usdoj.gov/bjs/pub/pdf/ccspglea.pdf Wholey, J.S., H.P. Harty, and K. E. Newcomer. The Handbook of Practical Program Evaluation. San Francisco, California: Jossey-Bass, 1994. W.K. Kellogg Foundation. Evaluation Handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998. W.K. Kellogg Foundation. Logic Model Development Guide: Using Logic Models to Bring Together Planning, Evaluation, and Action. Battle Creek, Michigan: W.K. Kellogg Foundation, 2001. Evaluation Books Berk, R.A. and P. H. Rossi. Thinking About Evaluation (2nd ed.). Newbury Park, California: Sage Publications, 1999. Bickman, L., ed. Using Program Theory in Evaluation. San Francisco, California: Jossey-Bass, 1987. Boruch, R. F., P.M. Wortman, and D.S. Cordray, eds. Reanalyzing Program Evaluations. San Francisco: Jossey-Bass, 1981. Boyle, R. and D. Lemaire. Building Effective Evaluation Capacity: Lessons from Practice. New Brunswick, New Jersey: Transaction Publishers, 1999. Bradley, V. J. Assessing and Enhancing the Quality of Services: A Guide for the Human Services Field. Boston: Human Services Research Institute, 1984. Braybrooke, C.B., and C.B. Lindblom. A Strategy of Decision: Policy Evaluation as a Social Process. New York: Free Press, 1963. Brinkerhoff, R.O., D.M. Brethower, T. Hluchyj, and J.R. Nowakowski. Program Evaluation: A Practitioner’s Guide for Trainers and Educators. Boston: Kluwer-Nijhoff, 1983. Brinkerhoff, R.O., ed. Evaluating Training Programs in Business and Industry, No.44, Winter. San Francisco: Jossey-Bass, 1989. Bryk, A.S., ed. Stakeholder-Based Evaluation. San Francisco, California: Jossey-Bass, 1983. Campbell, D.T., and J.C. Stanley. Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally, 1963. Caro, F.C., ed., Readings in Evaluation Research. New York: Russell Sage Foundation, 1971. Chambers, D.E., K.R. Wedel, and M.K. Rodwell. Evaluating Social Programs. Boston: Allyn & Bacon, 1992. Chen, H. Theory-Driven Evaluations. Newbury Park, CA: Sage Publications, 1990. Conner, R., and M. Hendricks, eds. International Innovations in Evaluation Methodology. San Francisco: Jossey-Bass, 1989. Cook, T.D., and D.T. Campbell. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago: Rand McNally, 1979. Cronbach, L.J. Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass, 1982. Datta, L., ed. Evaluation in Change: Meeting New Government Needs. Newbury Park, California: Sage Publications, 1981. Denzin, N.K., and Y.S. Lincoln, eds. Handbook of Qualitative Research. Thousand Oaks, California: Sage Publications, 1994. Eck, J. “Learning from Experience in Problem-Oriented Policing and Situational Prevention: The Positive Functions of Weak Evaluations and the Negative Functions of Strong Ones.” In Evaluation for Crime Prevention, Nick Tilley, ed. Crime Prevention Studies, 14. Monsey, New York: Criminal Justice Press, 2002. Eisner, E.W. The Enlightened Eye: Qualitative Inquiry and the Enhancement of Educational Practice. Upper Saddle River, New Jersey: Prentice Hall, 1998. Fetterman, D.M., S. J. Kaftarian, and A. Wandersman, eds. Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, California: Sage Publications, 1996. Grinnell, R.M. Social Work Research and Evaluation (4th ed.). Itasca, Illinois: F.E. Peacock, 1993. Guba, E.C. and Y.S. Lincoln. Effective Evaluation: Improving the Usefulness of Evaluation Results through Responsive and Naturalistic Approaches. San Francisco: Jossey-Bass, 1981. Guba, E.G. and Y.S. Lincoln. Fourth Generation Evaluation. Newbury Park, California: Sage Publications, 1989. Guttentag, M. and E.L. Struening, eds. Handbook of Evaluation Research. Newbury Park, California: Sage Publications, 1975. Hausman, J. and D. Wise, eds. Social Experimentation. Chicago: University of Chicago Press, 1985. Herman, J.L., L.L. Morris, and C.T. Fitz-Gibbon. Evaluator’s Handbook. Newbury Park, California: Sage Publications, 1987. House, E.R., ed. School Evaluation: The Politics and Process. Berkeley, California: McCutchan, 1973. House, E.R., ed New Directions in Educational Evaluation. London: Farmer, 1986. King, J., L. Morris, and C.T. Fitz-Gibbon. How to Assess Program Implementation. Newbury Park, California: Sage Publications, 1987. LaVigna, G.W., T. J. Willis, J.F. Shaull, M. Abedi, and M. Sweitzer. The Periodic Service Review: A Total Quality Assurance System for Human Services and Education. Baltimore: Paul H. Brookes, 1994. Levin, H.M. Cost-Effectiveness: A Primer. Newbury Park, California: Sage Publications, 1983. Levine, R.A., M.A. Solomon, C.M. Hellstern, and H. Wollmann. Evaluating Action Programs: Readings in Social Action and Education. Boston: Allyn & Bacon, 1981. Lincoln, Y.S. and E.C. Cuba. Naturalistic Inquiry. Newbury Park, California: Sage Publications, 1985. Lofland, J. and L.H. Lofland. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont, California: Wadsworth, 1984. Love, A.J. Internal Evaluation: Building Organizations from Within. Applied Social Research Methods Series, Vol. 24. Newbury Park, California: Sage Publications, 1991 Madaus, C.F., M.J. Scriven, and D.L. Stufflebeam, eds. Evaluation Models: Viewpoints on Educational and Human Services Evaluation. Boston: Kluwer- Nijhoff, 1983. Miles, M.B.and A.M. Huberman. Qualitative Data Analysis: A Sourcebook of New Methods. Newbury Park, California: Sage Publications, 1984. Mishan, E.J. Cost-Benefit Analysis (4th ed.). London: Unwin Hyman, 1988. Mohr, L.B. Impact Analysis for Program Evaluation (2nd ed.). Thousand Oaks, California: Sage Publications, 1995. Nathan, R.P. Social Science in Government: Uses and Misuses. New York: Basic Books, 1989. Nay, J.N. and P. Kay. Government Oversight and Evaluability Assessment. Lexington, Massachusetts: D.C. Heath, 1982. Paine, S.C., G.T. Bellamy, and B.L. Wilcox, eds. Human Services that Work: From Innovation to Standard Practice. Baltimore, Maryland: Paul H. Brookes, 1984. Palumbo, D., ed. The Politics of Program Evaluation. Newbury Park, California: Sage Publications, 1991. Patton, M.Q. Practical Evaluation. Newbury Park, California: Sage Publications, 1982. Patton, M.Q. Creative Evaluation (2nd ed.). Newbury Park, California: Sage Publications, 1987. Patton, M.Q. How to Use Qualitative Methods in Evaluation. Newbury Park, California: Sage Publications, 1987. Patton, M.Q. Qualitative Evaluation and Research Methods (2nd ed.). Beverly Hills, California: Sage Publications, 1990. Patton, M.Q. Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, California: Sage Publications, 1997. Popham, I.W., ed. Evaluation. in Education: Current Applications. Berkeley, California: McCutchan, 1974. Pressman, I.L. and A. Wildavsky. Implementation (3rd ed.). Berkeley: University of California Press, 1984. Rossi, P.H., ed. Standards for Evaluation Practice. San Francisco: Jossey-Bass, 1982. Rossi, P.H. and H.E. Freeman. Evaluation: A Systematic Approach (5th ed.). Newbury Park, California: Sage Publications, 1993. Royce, D. . Program Evaluation. Chicago: Nelson Hall, 1992. Rutman, L., ed. Evaluation Research Methods: A Basic Guide. Newbury Park, California: Sage Publications, 1977. Salant, P. and D. Dillman. How to Conduct Your Own Survey. New York: Wiley Publishers, 1994. Saxe, L. and M. Fine. Social Experiments: Methods for Design and Evaluation. Newbury Park, California: Sage Publications, 1981. Schmidt, R.E., J.W. Scanlon, and J.B. Bell. Evaluability Assessment: Making Public Programs Work Better. Rockville, Maryland: U.S. Department of Health and Human Services, Project Share, 1979. Schulberg, H.C. and J.M. Jerrell, eds. The Evaluator and Management. Newbury Park, California: Sage Publications, 1979. Scriven, M. The Logic of Evaluation. Inverness, California: Edgepress, 1980. Scriven, M. Evaluation Thesaurus (4th ed). Newbury Park, California: Sage Publications, 1991. Sechrest, L., I. Bunker, I., and B. Perrin, eds. Improving Methods in Nonexperimental Research. Newbury Park, California: Sage Publications, 1991. Shadish, W., T. Cook, and L. Leviton. Foundations of Program Evaluation: Theories of Practice. Newbury Park, California: Sage Publications, 1991. Smith, N.L. ed. New Techniques for Evaluation. Newbury Park, California: Sage Publications, 1981. Smith, N.L., ed. Communication Strategies in Evaluation. Newbury Park, California: Sage Publications, 1982. Stake, R. B., ed. Evaluating the Arts in Education. A Responsive Approach. Columbus, Ohio: Merrill, 1975. Stufflebeam, D.L. and A.J. Shinkfield. Systematic Evaluation: A Self-Instructional Guide to Theory and Practice. Boston: Kluwer-Nijhoff, 1985. Suchman, B.A. Evaluative Research: Principles and Practice in Public Service and Social Action Programs. New York: Russell Sage Foundation, 1967. Thompson, M.S. Benefit-Cost Analysis for Program Evaluation. Newbury Park, California: Sage Publications, 1980. Trochim, W.M.K. Research Design for Program Evaluation: The Regression- Discontinuity Approach. Newbury Park, California: Sage Publications, 1984. Veney, J.E. and A.D. Kaluzny. Evaluation and Decision-Making for Health Services (3rd ed.). Ann Arbor, Michigan: Health Administration Press, 1998. Webb, B.J., D.T. Campbell, R.D. Schwartz, and L.B. Sechrest. Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally, 1966. Weiss C.H. Evaluation: Methods for Studying Programs and Policies (2nd ed.). Englewood Cliffs, New Jersey: Prentice Hall, 1997. Weiss, C.H. Evaluation Research: Methods for Assessing Program Effectiveness. Englewood Cliffs, New Jersey: Prentice-Hall, 1972. Weiss, H.B. and F. Jacobs. Evaluating Family Programs. New York: Aldine DeGruyter, 1988. Welch, W., ed. Case Study Methodology in Educational Evaluation. Minneapolis: Minnesota Research and Evaluation Center, 1981. Wholey, J.S. Evaluation and Effective Public Management. Boston: Little, Brown, 1983. Williams, W. The Implementation Perspective. Berkeley: University of California Press, 1980. Yin, R.K. Case Study Research: Design and Methods. Newbury Park, California: Sage Publications, 1984. Zweig, F.M., ed. Evaluation in Legislation. Newbury Park, California: Sage Publications, 1979. Zweig, F.M. and K.E. Martin, eds. Educating Policymakers for Evaluation. Newbury Park, California: Sage Publications, 1981 References References Artley, A. and S. Stroh, eds. The Performance-Based Management Handbook: A Six- Volume Compilation of Techniques and Tools for Implementing the Government Performance and Results Act of 1993 (GPRA), Volume 2: Establishing an Integrated Performance Measurement System. Oak Ridge Institute for Science and Education, Training Resources and Data Exchange, The Performance- Based Management Special Interest Group, 2001. www.orau.gov/pbm/pbmhandbook/Volume%202.pdf Braga, A., D. Kennedy, A. Piehl, and E. Waring. Reducing Gun Violence: The Boston Gun Project’s Operation Ceasefire. Washington, D.C.: U.S. Department of Justice, National Institute of Justice, 2001. Center for Substance Abuse Prevention (CSAP). Building a Successful Prevention Program: A Planning Tool for Substance APrevention Professionals. CSAP Western Center for the Application of Prevention Technologies (CAPT). Reno: University of Nevada, 2003. Community Policing Consortium Resources. Strategic Planning for Community Policing, Section 5: Evaluation. Washington, D.C.: Community Policing Consortium. www.communitypolicing.org/strategic/sec6.html Fiedler, M. The Colorado Springs Police Department’s Police Accountability and Service Standard (PASS) Mmodel. Colorado Springs: Management Services Division, Planning Section, Colorado Springs Police Department, 2002. Juvenile Justice Evaluation Center, Justice Research and Statistics Association. Evaluability Assessment: Examining the Readiness of a Program for Evaluation. Program Evaluation Briefing Series No. 6. Washington, DC: Office of Juvenile Justice and Delinquency Prevention, 2006. www.jrsa.org/jjec/about/briefing_evaluability-assessment.html Loe, H., Jr. Practical Evaluation of Public Health Programs Workbook. Houston, TX: The University of Texas-Houston Health Science Center, School of Public Health and The Texas Department of Health, 2001. www.phppo.cdc.gov/phtn/Pract-Eval/workbook.asp Mertinko, E., L.C. Novotney, T. Baker, and J. Lange. Evaluating Your Pprogram: A Beginner’s Self-Evaluation Workbook for Mentoring Programs. Washington, D.C.: Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, and the U.S. Department of Justice. Miles, M.B. and A.M. Huberman. Qualitative Data Analysis. Thousand Oaks, California: Sage Publications, 1994. Patton, M. Q. Qualitative Evaluation and Research Methods (2nd ed.). Beverly Hills, California: Sage Publications, 1990. Patton, M.Q. Qualitative Research and Evaluation Methods. Thousand Oak,s California: Sage Publications, 2002. Salant, P., D. Dillman. How to Conduct Your Own Survey. New York: Wiley Publishers, 1994. Shadish, W. R., T.D. Cook, and D.T. Campbell. Experimental and Quasi- Experimental Designs for Generalized Causal Inference. New York: Houghton Mifflin, 2002. United States Department of Health and Human Services, Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention. Achieving Outcomes: A Practitioner’s Guide to Effective Prevention. Washington, D.C.: SAMHSA, 2002. www.modelprograms.samhsa.gov/pdfs/AchievingOutcomes.pdf The University of Texas at Arlington, School of Social Work. Instrument Development Checklist. www2.uta.edu/sswmindel/S6324/ Class%20Materials/measurement/instrument_development_checklist.htm The University of Texas-Houston Health Science Center, School of Public Health and The Texas Department of Health. Practical Evaluation of Public Health Programs. Atlanta, Georgia: The Centers for Disease Control and Prevention and The Association of Schools of Public Health Cooperative Agreement, 1998. University of Wisconsin-Extension: Program Development and Evaluation. Enhancing Program Performance with Logic Models. www.uwex.edu/ces/lmcourse/ Uphoff, N. “A Field Methodology for Participatory Self-Evaluation,” Community Development Journal, 26, (1991) 271–286. Urban Institute. Evaluation Guidebook for Projects Funded by S.T.O.P. Formula Grants under the Violence Against Women Act. Washington, D.C.: National Institute of Justice, 1997. USAID Center for Development Information and Evaluation. Performance Monitoring and Evaluation Tips: Conducting a Participatory Evaluation. Washington, D.C.: United States Agency for International Development, 1996. USAID Center for Development Information and Evaluation. Performance Monitoring and Evaluation Tips: The Role of Evaluation in USAID. Washington, D.C.: United States Agency for International Development, 1997. Velez, L. and T. Paine. Service Standards in the Colorado Springs Police Department. Retrieved May 1, 2007 from www.calea.org/Online/newsletter/No83/ servicestandards.htm Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement Agencies. Washington, D.C.: U.S. Department of Justice, Bureau of Justice Statistics and the Office of Community Oriented Policing Services. www.ojp.usdoj.gov/bjs/pub/pdf/ccspglea.pdf W.K. Kellogg Foundation. Evaluation Handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998. For More Information U.S. Department of Justice Office of Community Oriented Policing Services 1100 Vermont Avenue, N.W. Washington, DC 20530 To obtain details on COPS programs, call the COPS Office Response Center at 800.421.6770 Visit COPS Online at the address listed below www.cops.usdoj.gov e06072996