Scott T. Walters
University of Texas School of Public Health
Melissa Alexander
U.S. Probation
Eastern District of Missouri
Amanda M. Vader
University of Texas School of Public Health
MOTIVATIONAL INTERVIEWING (MI) is a style of interaction intended to help individuals resolve ambivalence about behavior change (Miller & Baca, 1983; Miller & Rollnick, 2002). It involves a collaborative partnership between the provider and client, a focus on drawing out internal motivation for change, and a respect for the client’s right and capacity to choose what to do about his/her problematic behavior. Although originally developed as a counseling intervention for substance abuse, MI has broadened into areas such as HIV risk behaviors, medication compliance, criminal justice, and other behavioral areas (Hettema, Steele, & Miller, 2005; Rubak, Sandboek, Lauritzen, & Christensen, 2005). Two recent meta-analyses found an overall significant effect of MI (Hettema et al., 2005; Rubak et al., 2005). In 70 published studies, MI was significantly better than other approaches in three out of four studies, and outperformed traditional advice-giving in 80 percent of studies (Rubak et al., 2005).
In the push toward evidence-based practice, MI has gained attention in the criminal justice arena, as practitioners recognize the need to actively engage offenders in order to encourage behavior change. Broadly speaking, MI is consistent with the responsivity principle because it suggests an interaction style that helps motivate clients for change (Walters, Clark, Gingerich, & Meltzer, 2007). Among the more than 180 published trials of MI (www.motivationalinterviewing.org), there are also a handful of studies suggesting that MI can improve criminal justice outcomes (McMurran, in press). However, in translating research into practice, some have questioned whether MI needs to be modified to meet the unique role of probation officers. One difficulty has to do with the dual role of the probation officer. Specifically, in contrast to counselors and healthcare workers who are mainly concerned about the well-being of a client, probation officers are tasked with two sometimes competing roles—protecting society and working collaboratively with the offender. Another difficulty in translating MI into a probation setting involves the kind of interactions that officers tend to have with probationers. Compared to counseling interactions, probation interactions tend to be brief, multi-focused, and balanced with an assessment of progress. Finally, there is a need to identify the most effective ways of teaching and assessing the specific skills that are most valuable in this setting.
In their MI training model, Miller and Moyers (2006) propose an eight-stage learning model, starting with a foundation in the spirit of MI and then moving to client-centered counseling skills, most importantly the skill of accurate empathy, followed by other stages (recognition of client speech, eliciting/strengthening change talk, rolling with resistance, negotiating change plans, consolidating client commitment, switching flexibly between MI and other styles). They stress that the ability to listen empathically is a foundational skill of MI, no matter what the setting. In the MI model, empathic listening involves an active interest in understanding what the client is saying, including drawing out the client’s own motivations, thoughts, or plans for change. Empathy is specifically differentiated from warmth, approval or genuineness; rather, it involves a curiosity and deep understanding of the client’s perspective. Although some probation officers may come to the table with a natural ability to listen empathically, there is also evidence that accurate empathy is a skill that can be learned (e.g., Miller & Baca, 1983). Along these lines, several instruments have been developed to rate reflective listening and other aspects of MI practice (see Madson & Campbell, 2006 for a review). These instruments can be used to measure gains after training or for ongoing supervision or quality control. Some instruments ask providers to respond to written (Miller, Hedrick, & Orlofsky, 1991) or videotaped (Rosengren, Baer, Hartzler, Dunn, & Wells, 2005) scenarios, while others rate audio or videotapes of actual client interactions (Miller, 2000; Moyers, Martin, Manuel, & Miller, 2003). However, none have been specifically designed to rate probation interactions.
The Helpful Responses Questionnaire (HRQ) was developed by Miller and colleagues to measure the ability to respond empathically (Miller et al., 1991). The HRQ presents six hypothetical client statements and asks the respondent to “write the next thing that you would say if you wanted to be helpful.” The responses are then rated for depth of reflective listening. Since its development, the HRQ has been used to evaluate the effectiveness of training in several studies (e.g., Baer, Rosengren, Dunn, Wells, Ogle, & Hartzler 2004; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004). Although the HRQ does not match the quality of other assessments that rate actual interactions (see Madson & Campbell, 2006 for a review), HRQ scores are modestly related to actual MI skills in practice (Miller & Mount, 2001).
Methods
The purpose of this project was to develop a brief measure of reflective listening that was specific to the work of probation officers: the Officer Responses Questionnaire (ORQ). (For a formatted copy of the questionnaire and scoring procedures, contact Scott Walters, University of Texas School of Public Health, 5323 Harry Hines Blvd., V8.112, Dallas, TX 75390 . Scott.walters@utsouthwestern.edu) We started with the HRQ and, with the permission of the authors, modified the scale to address situations that were more common to probation and parole officers.
We began by identifying situations that officers might face in their daily interactions with individuals on probation/parole. These scenarios were compiled by two of the authors (MA and SW) who had extensive experience training probation officers and with feedback from probation officers in an urban probation department. The five scenarios depict relatively common situations in probation: substance use, aggressive behavior, drug testing, and employment (see Table 1).
Table 1. Officer Responses Questionnaire Items
Item 1: A 22-year-old man tells you: “I do want to stay clean and sober, but I can’t get a job because of this court thing, and so I have to live with my brother who drinks all the time.” Item 2: A 30-year-old woman tells you: “I like to smoke weed. The way I see it, I never hurt anyone, so people should just leave me alone. I’ll stay clean while I’m on probation, but after I’m off, I’m going to do what I want.” Item 3: A 41-year-old man tells you: “The cops always take the woman’s side. Sure, we got in a fight and I yelled at her, but she was the one who grabbed the knife. What am I supposed to do? This is so typical. It’s always the man’s fault.” Item 4: A 24-year-old man tells you: “I know I was supposed to take care of the UA. I tried to get there, but my car was in the shop. I waited for the bus, but for some reason all the bus times had changed.” Item 5: A 24-year-old woman tells you: “I’ve been looking for work, but it’s impossible for someone on probation to find a good job.” |
After each paragraph, a space was provided for the respondent to write a response. Instructions for the questionnaire were:
The following statements are things that a probationer might say during a probation interaction. Think about each statement as if you were really in the situation, with that person talking to you. For each statement, write the next thing you would say if you wanted to let the person know that you were listening. Write only one or two sentences for each statement.
Each officer response was rated on depth of reflection and ability to avoid communication roadblocks (Gordon, 2003). We maintained the 5-point ordinal scoring scale developed by Miller, Hendrick, & Orlofsky (1991), but made significant modifications to accommodate some of the unique tasks of the officer. First, we separated closed from open questions, and truncated the HRQ’s three levels of reflection into two basic levels—simple vs. complex. We also did not penalize officers for asking questions alongside reflections, provided that they did not include a communication roadblock. We made these accommodations because, compared to counseling interactions, probation officer interactions tend to be briefer, multifocused, and contain more questions to assess probation progress and risk to the community. We retained the HRQ’s roadblock category as the lowest score. Thus, a response is given a score of 1 if it contains any roadblock response such as ordering, disagreeing, or giving advice without permission. A response is given a score of 2 if it contains a closed question, affirmation, offers of help, or other non-reflective response. A response is given a score of 3 if it contains an open question. A response is given a score of 4 if it contains a “simple” reflection that restates the basic content of the original statement. A response is given a score of 5 if it paraphrases or infers a deeper meaning from the original statement. If a response contains multiple elements (e.g., open question and simple reflection), it receives the score of the highest elements (4 for the simple reflection), unless it contains a roadblock, in which case it receives a score of 1.
Following initial development of the ORQ, we began utilizing the questionnaire in trainings as a pre/post measure of reflective listening. The questionnaires were administered as part of two-day MI training provided by various trainers in several federal districts throughout the United States . Trainings used a similar format, including lecture, role play and feedback, and were conducted by members of the Motivational Interviewing Network of Trainers who had extensive experience training probation officers. IRB approval was obtained from the University of Texas Health Science Center at Houston . Officers were asked to complete the questionnaires at the beginning of the training and again at the end of the two days. Several districts allowed the officers to turn in questionnaires after the end of training (ranging from several days to several weeks later).
Two of the authors (MA and AV) independently scored an initial sample of six questionnaires. Questionnaires were randomly selected from both pre- and post-training responses. Interrater reliabilities of these initial questionnaires were quite poor. The authors discussed the scoring discrepancies, many of which were due to answers that did not seem to fit in any category. Most often, the discrepancies involved responses that did not qualify as reflections, but were still intended to let the offender know the officer was listening (e.g., “I understand what you are saying.”). As a general rule, we considered responses that focused on the probationer’s statement to be evidence of reflective listening (e.g., “It makes you angry.”), while those that focused more on the officer’s reaction or affect were considered to be neutral responses (e.g., “I understand how angry that makes you.”). Based on discussions with other MI experts, the scoring rules were further modified and several examples were provided in the scoring scheme. The authors scored two additional rounds totaling 23 questionnaires, again discussing discrepancies and further modifying scoring. The final scoring provides direction and examples incorporating common responses from officers. Additionally, we included a continuum diagram to assist scorers in making decisions about responses (see Figure 1).
Figure 1.
Results
Once the scoring criteria were finalized, two of the authors independently scored125 questionnaires. Intraclass correlations (ICCs) were calculated for each of the five ORQ items. As noted by Miller et al. (2004), ICCs are a more conservative estimate of interrater reliability than Pearson’s correlations, as it adjusts for chance agreement and systematic differences. The ICCs for the five ORQ items are displayed in Table 2. Using Cicchetti’s (1994) proposed categories for ICCs in clinical instruments, all items were in the excellent range (.75 to 1.00) except item 1, which at .73 was in the good range.
Table 2. Intra-class Correlations by Item
Item |
Single Measures ICC |
F-Value |
Significance |
1 |
.726 |
6.307 |
.000 |
2 |
.757 |
7.223 |
.000 |
3 |
.790 |
8.508 |
.000 |
4 |
.890 |
17.211 |
.000 |
5 |
.856 |
12.929 |
.000 |
As the ORQ was given as a pre/post training measure to officers completing an initial 2-day training in MI, we were also able to compute changes in scores following training. Pre/post scores were obtained for 80 officers in five districts. Paired t-test analyses indicated that officers significantly increased their scores on all items (see Table 3), with post-training mean scores nearing a score of 4, indicating moderate use of reflective listening. These scores reflect a 68 percent increase, suggesting that officers did benefit from the training.
Table 3. Officer Responses Questionnaire Mean Scores Pre/Post-Training
|
N |
Pre-test Mean |
Post-test Mean |
T-Value |
Significance |
Item 1 |
80 |
2.5 |
3.9 |
-8.894 |
.000 |
Item 2 |
80 |
2.2 |
3.6 |
-8.805 |
.000 |
Item 3 |
80 |
2.7 |
4.1 |
-8.292 |
.000 |
Item 4 |
79 |
2.1 |
3.8 |
-10.220 |
.000 |
Item 5 |
80 |
2.1 |
3.9 |
-10.793 |
.000 |
Discussion
The ORQ was designed to be a brief measure of reflective listening within the framework of Motivational Interviewing. The ORQ focuses on written empathic responses to scenarios that depict common situations seen in community corrections settings. Our initial analyses suggest that the ORQ can be reliably scored and that officers improve their ability to generate reflective responses as a result of attending a two-day introductory workshop in MI. As part of a training evaluation effort, the ORQ is being used as one skill measure in the federal probation system.
It should be noted, however, that the ORQ is not a measure of overall MI skill, nor should it be used in lieu of actual observations of clinical skill. Reflective listening is an important building block of MI, but there are additional skills, such as the ability to identify and respond to self-motivational statements, that probably account for the excellent track record of MI (Armhein, Miller, Yahne, Palmer, & Fulcher, 2003). Also, prior research has suggested that paper-and-pencil measures may not be good predictors of responses within actual sessions with clients/offenders, and as mentioned previously, are not a substitute for direct observation of skills (Miller & Mount, 2001; Walters, Matson, Baer, & Ziedonis, 2005). Written measures such as the ORQ may provide evidence of an officer’s capacity to provide empathic responses, but they do not tell us whether officers can or will use such skills in actual interactions with offenders. Rather, a comprehensive assessment of MI skill should include multiple measures of officer skill (either audio/videotaped or via direct observation). As previously mentioned, several measures that involve coding of actual interactions have been developed to assess MI skills (see Madson & Campbell, 2006 for a review). Combining the ORQ with these more in-depth measures of MI skill will give agencies a more accurate picture of officer skill as it relates to use of MI during interactions with offenders.
Research on MI skill attainment also highlights the importance of focusing on the underlying spirit of MI and a “learning-to-learn” model, which involves paying attention to offender responses as a sign of whether MI is occurring (Miller et al., 2004). Officers must not only use new skills connected with MI, but also suppress prior habits that are inconsistent with MI (Miller & Mount, 2001). As noted in the eight-stage learning model, the authors hope that officers will be able to blend the skillful use of MI with other effective interventions that will allow them to fulfill their multiple duties to the offender, court and community.