CHAPTER TWO

THE EVALUATION
MENTALITY

Evaluation is like regular exercise. We all know that exercise is good for us. And we all try it from time to time. But the majority of Americans fall far short of the recommended exercise requirements. Why is that?

In most cases, we don’t exercise regularly because we have convinced ourselves that exercise requires too much effort, that it will hurt, and that it probably won’t give us a perfect body anyway.

It’s the same way for evaluation. Everyone acknowledges that it is always a good idea to evaluate any program that uses taxpayer dollars, but when it comes time to build an evaluation into a program plan, dozens of excuses are offered as to why it just can’t or shouldn’t be done, in this particular case. So why do many people shy away from conducting program evaluations regularly?

Most excuses for not doing an evaluation are variations on the following four themes.

• “Evaluation is too complicated. Program evaluations require complex research designs for sophisticated experiments. I don’t fully understand what it is involved but it sounds pretty scary.”
  
  
• “If I conduct an evaluation, I may discover that my great idea was a total flop. I want to avoid that embarrassment.”
  
  
• “I have a very limited budget. Evaluations are expensive and time-consuming. I just can’t spare the resources required.”
  
  
• “Evaluation is a lot of work, and I don’t have the time. Besides, I never understand the gobbledygook that I read in evaluation reports. So why bother?”

Do some of these sound familiar? Have you found yourself thinking these thoughts? Let’s go through them one by one to show you why they aren’t true and may be standing in the way of your success in traffic safety.

“Evaluation is too complicated.”

In reality, however, traffic safety evaluation need not involve “Ivory Tower” laboratory science. Traffic safety evaluation is an applied science that works within the constraints of state and local program implementation. Most local communities simply do not have the volume of traffic deaths and injuries to conduct that kind of countermeasure effectiveness evaluation. Instead, these communities can focus their evaluation efforts on determining if the particular program they implemented achieved its specific objectives.

If you are implementing an occupant protection program, your evaluation dollars would be better spent demonstrating an increase in safety belt use rather than proving, once again, that safety belts save lives. Documenting an increase in safety belt use over baseline levels involves a much simpler evaluation and should not deplete your program resources.

“I may not like the answer so I better not ask the question.”

When you implement a program, you put a lot of yourself into the effort. You believe in your heart that it is a good program, and you do everything you can to make it work. But you never know what might happen.

Some people shy away from evaluations because they don’t want their good idea to be proven wrong. The mistake they are making is viewing evaluation as the last step in a process, like a final exam. If evaluation is tacked on at the end of the project, you may very well come up with answers that you don’t like.

The key to successful evaluation is to build evaluation in from the start, so it can help you frame the questions you are asking, and even clarify the problem that you are trying to solve. A well-planned evaluation should not yield last minute unpleasant surprises. Instead, it will provide useful information that helps fine-tune the program at every step of the way. It also tells you what’s working and what can be made better.

“I have a limited budget; I prefer to spend my dollars on implementation.”

These days, everyone is expected to do more with less. Project directors struggle to stretch every program dollar to the limit. When asked to choose between delivering more high school presentations or conducting a program evaluation, many choose more presentations because they believe that their impact will be greater if they can reach more kids. There are two errors in this logic.

1. They believe that an evaluation will cut into their program implementation resources, when in fact there are many ways to conduct an evaluation inexpensively.
   
   
2. Reaching more kids with a perfect traffic safety message is always a good idea. But how do you know if your message is being well-received unless you do some sort of evaluation? Collecting and monitoring feedback throughout a project provides the opportunity to fine-tune your message as you go in response to audience reaction. At the conclusion of the program you will also have the data you need to improve the entire program before you offer it again.

Another point to keep in mind when thinking about conserving program dollars is satisfying your funding source. When it is time to ask your funding source to extend your program for another year, your proposal will be much more persuasive if it is backed up with solid data demonstrating that you accomplished your objectives.

“Evaluation is too much work.”

Evaluation can be labor-intensive (note that we didn’t say expensive) and very tedious. This is why evaluation is one of those areas that, as a project director, you should delegate to someone else. This could be an employee from another department in your agency, a faculty member or graduate student at a local university, or a private evaluation consultant. In addition to doing the time-consuming work involved in data collection, you should expect an evaluation specialist to be able to explain the results in language you and everyone else can understand. There is no place for gobbledygook in an evaluation report. (Section V of this Guide talks about what to look for in evaluators and how best to work with them.)

Now that we have eliminated all your old reasons for not doing an evaluation, we can concentrate on some new reasons for doing one.

Let’s go back to our exercise analogy, for a moment. We all know some people who are fully committed to getting regular exercise. They don’t have to think about if they will exercise or how they will fit it into their schedule. Exercise is an essential part of their daily existence, just like eating and sleeping. These individuals report that exercise improves every aspect of their life, not just their physical conditioning. They have more energy, they are more productive at work, and they are less prone to depression. What separates them from the rest of us is their exercise mentality.

To get the full range of benefits from evaluation, you need to have an evaluation mentality. This means that you never even consider implementing a program without first thinking like an evaluator. You approach each new problem with the same set of questions:

• What do I know about the problem?
• If I tried to fix this problem, what could I accomplish?
• How could I measure my results?
• How can I collect the data I will need?
• What are my criteria for success?

With the answers to these questions in hand, you are prepared to convince any funding source that you know what needs to be done and that your ideas have a high probability of success. Your evaluation mentality will also ensure that at the end of this project, you can report back to these same funding sources with solid information on what you accomplished.

An evaluation mentality cannot ensure that every project you implement will be a resounding success, but it can ensure that you fully understand what you tried to do and why things turned out as they did.

If you have an evaluation mentality, you design your program and your evaluation at the same time. The benefits of this approach are substantial. An evaluation mentality will enable you to:

• Accurately identify the problem you are trying to solve—All too often people jump into implementing a program without really understanding the underlying cause of their problem. Is underage DWI a problem because the liquor stores are selling to teenagers or because the police are not targeting the locations where kids are drinking? It is not enough to suspect that safety belt use is low in your community. You need to determine up front what target groups make up your non-users. An evaluation mentality frees you from having to take a “shot-gun” approach to solving every problem.
  
  
• Uncover some problems you didn’t know you had—You might assume that your pedestrian safety problem involves the very young and very old until you discover that a significant percentage of your pedestrian fatalities are working age adults who had been drinking at the time of their crash. This problem would require a totally different set of countermeasures than a child pedestrian problem.
  
  
• Establish reasonable, practical objectives for dealing with these problems—Global objectives are the hardest to accomplish. With good problem identification data, you can focus your objectives on the specific problem you are trying to solve. Instead of trying to “reduce unsafe driving behaviors” you might want to reduce red-light running which is the unsafe behavior that is causing the most concern in your community.
  
  
• Determine if you have accomplished your program objectives—A major purpose of any evaluation is to determine if your program accomplished its objectives. Well- thought out objectives are an important first step, but an evaluation mentality will also help you zero in on what you truly need to measure. Too often, project directors waste time and money collecting data that they can never use, because they can’t compare it to any baseline or because it does not relate to their program objectives.
  
  
• Provide information to funding sources, the media and the public to continue support for program—Support for a good idea can evaporate if there is no evidence that the idea really works. With an evaluation mentality, you create consensus from the beginning on the criteria for success, and you stay focused on the data you will need to measure your performance against those criteria. You also understand that a significant, but barely discernible, change may not constitute success in everyone’s eye.
  
  
• Determine if and how a program should be revised to increase its effectiveness—With an evaluation mentality, you don’t wait until the end to find out how your idea worked. Very few projects work perfectly. There are always aspects that could be tweaked to make them more effective. With an evaluation mentality, you monitor performance throughout the project so that you can institute mid-course corrections if needed, and so that you are ready to revise your concept for next year.

You do not need to be an evaluation specialist to have an evaluation mentality, just as you don’t have to be body builder to have an exercise mentality. You just have to recognize that evaluation, when built in from the beginning, provides benefits throughout the life of your program.