Improving Quality of Documents Reference Service

My purpose this morning is share with you what we’ve learned about improving reference performance.

For a long time as a profession we’ve been guilty of giving half-right reference. Peter Hernon and Charles McClure in their April 15, 1986 Library Journal article titled, "Unobtrusive Reference Testing: the 55 Percent Rule," proposed that reference staff in academic and public libraries, regardless of department, are only able to correctly answer the factual and bibliographic questions they receive about 55 percent of the time. Further, in this article, they stated "for approximately 20 years now, the library community has been aware of the 55 percent accuracy rate, yet few tangible ongoing strategies have been developed to address this finding."

A year earlier, in a Reference Quarterly article, Terry Crowley asked "what changes in library training, policy and practice can help us improve public service?" Maryland has attempted to address this problem and thanks to the support and commitment of public library directors and staff in Maryland public libraries we now have a formula for improving reference.

Over a ten-year period, the Division of Library Development & Services (DLDS) at the Maryland State Department of Education conducted four separate massive (more than 11,000 questions) unobtrusive surveys. Each of these surveys was designed to assess the quality of reference service from the customer’s point of view. The premise of the study has been that people expect, and deserve to receive, complete and accurate answers to their questions. From the results of the first survey in 1983 to the completion of the fourth survey in 1994, we were able to identify the need, discover what was wrong with the process, and took steps which have dramatically improved the quality of service people receive from Maryland public libraries.

How did Maryland get started in looking at reference service from the customer’s point of view? In 1981, several library administrators came to DLDS and asked for help in measuring reference performance in their library. Using the unobtrusive method developed by Dr. Terry Crowley and Dr. Thomas Childers, we piloted the survey in five library systems. The results of this pilot study created overwhelming interest among other public library directors.

In 1983, DLDS conducted the first statewide reference/performance using the unobtrusive observation methodology. The same 40 questions (20 walk-ins—20 telephone) were asked in 60 branches of 22 public library systems, with a total of 2,400 questions. (Maryland has 24 public library systems. Participation in the study was voluntary and two library systems chose not to participate.) We sought to answer two questions:

1. To what degree is a user likely to receive a correct answer to a "moderately difficult" question?
2. What levels of resources and kinds of activities are most likely to lead to desired levels of performance?

We learned three major things:

• The likelihood of a person getting a correct answer to a moderately difficult reference question (55% rule)

• Those factors traditionally thought to contribute to correct answers (e.g., size of reference collection, size of staff, degree of busyness, length of time for conducting a transaction, etc.) are not associated with reference performance.

• Factors that contribute to improved reference performance are basic communication behaviors that are within the control of the individual librarian.

In addition to measuring to what degree correct answers were provided, we also investigated what librarians did as they attempted to answer questions--that is, what behaviors they exhibited. We then analyzed the data to discover which of these behaviors were associated with providing correct answers. After identifying the behaviors, we designed an intensive three-day workshop and titled it, "Better Communication Equals Better Reference." More than two hundred library staff members in fourteen public library systems were trained.

In order to determine the effects of the training a second statewide survey was conducted in the same sixty branches. The results of the second survey showed that librarians whom we trained answered more than 77 percent of the questions correctly and those librarians who were not trained by DLDS answered only 60 percent of the questions correctly. We now had a pre-test, post-test design with a treatment group and a comparison group. Analysis of the variables showed that the difference in performance was due to the training. Also, in the library that performed the best, there was strong anecdotal evidence of supervisory support for the use of the behaviors. Several of these libraries promoted the development of behavioral job descriptions, which included the Model Reference Behaviors.

We have conducted two other surveys since 1986. An objective in the Maryland Plan for Libraries, 1986-1991, specifically addressed a library’s user’s need to have a complete and correct answer to his or her reference question and laid the foundation for a third state-wide survey. In addition to investigating reference performance and the use of the Model Reference Behaviors, an additional survey objective investigated the impact of training, peer coaching and activities which support the continued use of the behaviors. The 1994 survey corroborated findings from the three previous studies. The use of the Model Reference Behaviors is very strongly correlated with good performance.

Our experience in working with staff who are responsible for answering reference questions has shown that the primary reasons for providing incorrect answers are lack of skill in applying certain behaviors and the inability to apply these behaviors consistently on the job. The training design consists of three days with pre-workshop and interim assignments. The three days are progressive, that is, each day builds upon the day before and brings participants to the point where they are ready to return to their workplace with an improved set of skills.

The workshop is highly participative and has three major components: self-awareness, simulation (practice) and transfer. These components produce far greater impact when used together and designed as a whole than training that does not include all three.

The practice sessions are meant to be as close to real job situations as is possible in a workshop environment. Actual patron questions are used in the practice session and participants are asked to play out the role of librarian, patron, or observer.

I am certain that most of you would not find it surprising to know that a lot of training does not transfer back to the job. We took this fact into consideration and decided to include four strategies for the transfer of training when we designed this workshop

• the model behaviors checklist
• action plans
• management coaching, and
• peer coaching

The peer coaching strategy is the cornerstone of the training. Peer coaching is a mutual relationship in which two or more people agree to help each other apply new skills. In learning new skills we often go through an awkward period where the skill does not yet feel natural and may not bring the desired results. A peer or co-worker as coach can provide encouragement, reinforcement, and support to assist the person in overcoming the discomfort of using the new skill. Peer coaching is not easy, but when applied diligently, it contributes greatly to the successful transfer of skills learned.

Over time reference/information performance has improved in Maryland public libraries. This is very evident when we compare the range of scores from the first survey to the fourth. The lowest system score in 1994 was higher that the highest system score in 1983. The public library systems of Maryland worked successfully over a decade to break the so-called 55% rule of reference accuracy. Our users may not have noticed the difference but the data indicates that there are real results and service is much improved and is more consistent statewide. The work done in Maryland has served as a model to libraries throughout the nation, many of whom have adopted these strategies for training and customer service.

The outcome of the reference/information performance improvement project is an excellent example of the partnership and shared vision between public libraries and a state library agency. Using available resources, librarians were able to work together to improve a very basic and essential service for their customers.