This is the accessible text file for GAO report number GAO-05-130 entitled 'Medicare: Accuracy of Responses from the 1-800-MEDICARE Help Line Should Be Improved' which was released on December 08, 2004. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Committees: United States Government Accountability Office: GAO: December 2004: Medicare: Accuracy of Responses from the 1-800-MEDICARE Help Line Should Be Improved: GAO-05-130: GAO Highlights: Highlights of GAO-05-130, a report to congressional committees Why GAO Did This Study: In March 1999, the Centers for Medicare & Medicaid Services (CMS) implemented a telephone help line—1-800-MEDICARE—to provide information about program eligibility, enrollment, and benefits. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA) directed GAO to examine several issues related to this 24-hour help line and the customer service representatives (CSRs) who staff it. In this report, GAO evaluated (1) the accuracy of the information the help line provides, (2) the training given to CSRs, and (3) CMS’s efforts to monitor the accuracy of information provided through the help line. What GAO Found: The 1-800-MEDICARE help line provided accurate answers to 61 percent of the 420 calls we made and inaccurate answers to 29 percent. We were not able to obtain any answers for the remaining 10 percent of our calls at the time we placed them. Most of these calls were not answered because they were transferred to other contractors responsible for processing Medicare claims that were not open for business at the time we called or these calls were inadvertently disconnected. To facilitate accurate responses, the 1-800-MEDICARE help line provides CSRs with written answers—called “scripts”—that CSRs use during a call. When CSRs provided inaccurate information, it was largely because they did not seem to access and effectively use a script that answered our questions. CMS and its contractor do not routinely pretest the scripts to ensure that they are understandable to CSRs or potential callers. Percentage of Calls with Accurate, Inaccurate, and No Answers: [See PDF for image] Note: Based on 420 calls placed in July 2004. [End of figure] The training for CSRs meets CMS’s requirements, but it is not sufficient to ensure that CSRs are able to answer questions accurately on the help line. Before handling calls, CSRs must complete about 2 weeks of classroom training; accurately answer two simulated calls consecutively out of six; and score at least 90 percent on a written exam. In addition, all CSRs receive ongoing training. However, the results from our calls indicate that the testing and simulated call answering did not sufficiently measure whether CSRs were prepared to answer questions accurately. CMS delegates most accuracy monitoring to one of its contractors and reviews the results. The bulk of the monitoring focuses on how accurately individual CSRs answer questions. However, this monitoring does not systematically track questions answered inaccurately by CSRs as a group, which could help target training and script improvement. Through two smaller studies that measured how accurately specific questions were answered, CMS was able to identify areas to improve scripts and training. What GAO Recommends: To improve the accuracy of the information the help line provides, GAO recommends that CMS (1) revise procedures so that calls are not transferred to other contractors that are closed, (2) assess current scripts and pretest new and revised scripts to ensure that they are understandable, (3) provide more testing of CSRs’ ability to accurately answer questions and use the results to target training efforts as needed, and (4) monitor the accuracy rate for each frequently asked question and use the results to modify scripts or provide training, if necessary. CMS agreed with the recommendations. www.gao.gov/cgi-bin/getrpt?GAO-05-130. To view the full product, including the scope and methodology, click on the link above. For more information, contact Leslie G. Aronovitz at (312) 220-7600. [End of section] Contents: Letter: Results in Brief: Background: 1-800-MEDICARE Provided Accurate Information for Less Than Two-thirds of the Calls We Placed: CSR Training Met CMS's Requirements, But Was Not Sufficient to Ensure Accurate Responses: Monitoring Generally Focused on Performance of Individual CSRs, Rather than Accuracy Rates for Specific Questions: Conclusions: Recommendations: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix IIPrescription Drug Discount Card Questions and Information Used to Develop Accuracy Criteria: Appendix III: $600 Credit Question and Information Used to Develop Accuracy Criteria: Appendix IV: Medigap Question and Information Used to Develop Accuracy Criteria: Appendix V: Power Wheelchair Question and Information Used to Develop Accuracy Criteria: Appendix VI: Medicare Part B Question and Information Used to Develop Accuracy Criteria: Appendix VII: Eye Glasses/Exam Question and Information Used to Develop Accuracy Criteria: Appendix VIII: Comments from the Department of Health & Human Services: Appendix IX: GAO Contact and Staff Acknowledgments: GAO Contact: Acknowledgments: Tables: Table 1: Questions and Criteria Used to Determine Answer Accuracy: Table 2: Summary of Unanswered Calls by Question: Figure: Figure 1: Percentage of Calls with Accurate, Inaccurate, and No Answers, by Question: Abbreviations: CDC: Centers for Disease Control and Prevention: CMS: Centers for Medicare & Medicaid Services: CSR: customer service representative: IRS: Internal Revenue Service: MMA :Medicare Prescription Drug, Improvement, and Modernization Act of 2003: United States Government Accountability Office: Washington, DC 20548: December 8, 2004: The Honorable Charles E. Grassley: Chairman: The Honorable Max Baucus: Ranking Minority Member: Committee on Finance: United States Senate: The Honorable Joe Barton: Chairman: The Honorable John D. Dingell: Ranking Minority Member: Committee on Energy and Commerce: House of Representatives: The Honorable William M. Thomas: Chairman: The Honorable Charles B. Rangel: Ranking Minority Member: Committee on Ways and Means: House of Representatives: Medicare is a federal program that helps pay for a variety of health care services and items for approximately 41 million elderly and disabled beneficiaries. One of the responsibilities of the Centers for Medicare & Medicaid Services (CMS), the federal agency that administers Medicare, is to provide beneficiaries and other members of the public with clear, accurate, and timely information about the program. To help do so, in March 1999, CMS implemented a nationwide toll-free telephone help line--1-800-MEDICARE--which Medicare beneficiaries, their families, and other members of the public can call to ask questions about program eligibility, enrollment, and benefits. By 2001, the help line had customer service representatives (CSR) answering calls 24 hours a day, 7 days a week. For information about coverage and payment for medical services and items, beneficiaries and others have also been able to call the companies under contract with CMS to process and pay Medicare claims.[Footnote 1] In 2004, the 1-800-MEDICARE help line significantly expanded its operations in order to handle an increased number of calls. This increase occurred after the enactment of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA) in December 2003, which established a prescription drug discount card program for Medicare beneficiaries and required that information on this new program be made available through the 1-800-MEDICARE help line. During the 6 months following the enactment of MMA, the 1-800-MEDICARE help line handled over 9 million calls, more than triple the number handled in the previous 6 months. Many of these callers asked about the prescription drug benefit that will be available beginning in 2006, the prescription drug discount cards available in the interim, and the $600 credit for prescription drugs purchased by low-income beneficiaries with the cards. In response to the increased call volume, in the first half of 2004, CMS added over 800 CSRs--more than doubling the number of staff who had previously been available to respond to 1-800-MEDICARE help line inquiries. In June 2004, CMS also increased the number of contractors managing the 1-800-MEDICARE help line from one to two. MMA also required that GAO examine the accuracy and consistency of answers provided through the 1-800-MEDICARE help line and the training and education given to its CSRs.[Footnote 2] To address this requirement, as discussed with the congressional committees with jurisdiction over Medicare, we evaluated (1) the accuracy of information that 1-800-MEDICARE provides, (2) the training given to 1- 800-MEDICARE CSRs, and (3) CMS's efforts to monitor the accuracy of information provided through the 1-800-MEDICARE help line. To evaluate the accuracy of information provided by 1-800-MEDICARE, we made 420 calls to the 1-800-MEDICARE help line during July 2004 and posed 1 of 6 questions about the Medicare program during each call.[Footnote 3] For each question, we randomly placed calls at different times of the day and days of the week to match the typical pattern of calls reported by the 1-800-MEDICARE help line in April 2004. To develop our 6 questions, we initially formulated 20 questions from the 100 topics most frequently addressed by the 1-800-MEDICARE help line's CSRs in April 2004. For each of our questions we developed criteria that we used to define an accurate response, using information on the Medicare Web site's frequently asked questions section.[Footnote 4] CMS confirmed the responses we provided, and provided us with the written answers that the CSRs would be expected to use to respond to each question. From the list of 20 questions, we chose 6 questions, asked each one a total of 70 times, and evaluated the accuracy of the responses using information provided by CMS. If the CSR provided some information on the topic, but did not provide sufficient and complete information to meet our criteria, we considered the answer to be inaccurate. To evaluate the training provided to CSRs, we interviewed officials at the primary contractor[Footnote 5] and CMS staff, reviewed the instructional materials used in CSR training, and observed a training session in June 2004. In addition, we reviewed previous GAO reports on the training provided by help lines, including the training provided to the CSRs answering calls on the Internal Revenue Service's (IRS) help line.[Footnote 6] To evaluate CMS's efforts to monitor the accuracy of information provided through the help line, we interviewed CMS officials and officials from the primary 1-800-MEDICARE contractor. In addition, we reviewed related documents and information about call centers in other industries. We did not verify the reliability of CMS's monitoring data. Appendix I includes a more detailed discussion of our scope and methodology. Our work was conducted from May 2004 through December 2004 in accordance with generally accepted government auditing standards. Results in Brief: In response to our 420 calls to the 1-800-MEDICARE help line, CSRs accurately answered the questions posed in 61 percent of the calls we placed. The accuracy rate varied significantly among the six questions we posed. Twenty-nine percent of the calls were answered inaccurately. We were not able to obtain any answers to our questions for the remaining 10 percent of our calls at the time we placed them. This generally occurred when our morning, evening, and weekend calls were transferred to claims administration contractors that were not open for business at the time we called or when our calls were inadvertently disconnected. To facilitate accurate and consistent responses, the 1- 800-MEDICARE help line provides CSRs with information in the form of written answers--known as "scripts"--that CSRs can access by typing in key words on their computers during a call. The scripts are designed to address the help line's frequently asked questions or provide links to additional information. However, we found that CSRs provided inaccurate information largely because they did not always understand enough about the Medicare program to access a script that answered the question or could not clearly explain the material in the scripts that they were using. For example, one question about income eligibility for the $600 prescription drug credit was answered inaccurately in 55 out of 70 calls, generally because the CSRs did not seek other needed information from a second script to correctly answer the question. In addition, CSRs sometimes did not understand the language used in the scripts or other written material available to them. For example, in answering a question on power wheelchair payment, one CSR confused "trunk strength"--which is upper body strength--with car trunk space and incorrectly explained that Medicare would only cover a power wheelchair if a beneficiary had adequate space to put it in the trunk of his car. As this example shows, when CSRs do not understand the scripts, they provide inaccurate answers. Nevertheless, CMS and its contractor do not routinely pretest the scripts to ensure that they are understandable to the CSRs or potential callers. The training for CSRs meets CMS's requirements, but it is not sufficient to ensure that CSRs are able to answer questions accurately on the 1-800-MEDICARE help line. As required by CMS, all CSRs receive training. The primary contractor, which is responsible for training all newly hired CSRs, provides them with a 2-week session of classroom instruction on accessing and using scripts, customer service etiquette, and general information about the Medicare program. As part of the training, CMS requires newly hired CSRs to score 90 percent or higher on a written exam before they are allowed to answer questions on the help line. CMS requires CSRs to pass an exercise answering six simulated calls. As required by CMS, both contractors also provide their CSRs with periodic refresher training on Medicare program changes and other issues. Although the 1-800-MEDICARE contractors meet CMS's training requirements by providing instruction and testing, the number of inaccurate responses we received on our calls indicates that not all CSRs had the necessary knowledge and skills to answer our questions accurately. Testing how accurately CSRs answer frequently asked questions using scripts provides a better measure of their ability to handle calls. However, in their testing, CMS and its contractors do not emphasize CSRs' ability to answer questions accurately using scripts. For example, to pass the simulated call handling exercise, new CSRs have to demonstrate their ability to use scripts to answer calls accurately on only two consecutive calls out of six possible attempts. Similarly, on the written exam that newly hired CSRs must pass before they work on the help line, 24 of the exam's 52 questions ask CSRs to identify scripts with information to answer specific questions while the remaining 28 questions target other skills needed by help line staff. Further, while all CSRs receive continuing training, they are not required to demonstrate that they have effectively mastered new material in answering calls. CMS delegates most accuracy monitoring to its primary contractor and reviews the results, but this monitoring is not designed to identify and systematically track responses to questions that CSRs as a group commonly answer incorrectly. The primary contractor listens to the conversations CSRs have with callers, observes the scripts they select on their computers to answer questions, and then scores the CSRs' performance in several categories, including customer service etiquette and accuracy. Calls can be evaluated as acceptable if CSRs provide some correct information, even if they do not provide enough information to accurately answer the question. As required by CMS, the primary contractor evaluates four calls per month for each CSR--which is a standard amount for the help line industry--and reports the results to CMS. Although the primary contractor's monitoring includes an assessment of CSRs' accuracy, the monitoring does not categorize and identify accuracy rates by question, which could help CMS target training and other improvement efforts. In contrast to most of the 1- 800-MEDICARE help line monitoring efforts, on two occasions CMS contracted for studies that examined accuracy rates by question. Like our methodology, these studies used specific criteria to evaluate an answer's completeness and accuracy. The findings from these studies were similar to ours--accuracy rates varied by question and were sometimes lower than the accuracy rates found through the primary contractor's monitoring. For example, one study found that CSRs accurately answered between 39 and 69 percent of the questions asked about Medicare prescription drug discount card and benefit; in the same month the subcontractor found an overall accuracy rate for all questions to be about 94 percent. CMS used information from these smaller studies that focused on the accuracy rates for specific questions to improve the scripts and training for CSRs. To improve the accuracy of responses provided to the 1-800-MEDICARE help line callers, we are making several recommendations. Specifically, we are recommending that CMS revise procedures so that calls are not transferred to claims administration contractors that are closed for business at that time; assess current scripts and pretest new and revised scripts before approving them to ensure that they are understandable to CSRs and potential callers; provide more testing of CSRs' ability to use scripts to accurately answer the most frequently asked questions, and use the results to target additional training efforts as needed; and monitor the accuracy rate of each frequently asked question and use the results to modify scripts or provide training, if necessary. In its comments on a draft of this report, CMS agreed with our recommendations and provided additional detail on the challenges it faced in administering 1-800-MEDICARE after the massive call volume increase that followed MMA. CMS also described the steps that it was considering, or had begun, to address our recommendations. CMS expressed concern that we considered answers from 1-800-MEDICARE CSRs to be inaccurate if they were incomplete. However, we believe that our accuracy criteria included the minimum detail needed to avoid misleading callers. Background: CMS administers the 1-800-MEDICARE help line to answer beneficiaries' questions about Medicare eligibility, enrollment, and benefits. The help line currently operates 24 hours a day, 7 days a week, and has eight call center locations that are run by two contractors. As of October 2004, the primary contractor managed 1,332 of the 2,137 CSRs and operated seven of the eight 1-800-MEDICARE call centers. In addition, the primary contractor is responsible for other activities, such as distributing program material requested by callers, training all new CSRs before they handle calls on the 1-800-MEDICARE help line, and researching answers to more complex questions some callers may have. Prior to 2004, one contractor managed the 1-800-MEDICARE help line. In June 2004, in response to increasing call volume,[Footnote 7] CMS hired a second contractor, which in October 2004 managed 805 CSRs and operated one of the eight 1-800-MEDICARE call centers. A call placed to 1-800-MEDICARE is answered initially by an interactive voice response system. This is an automated system that, depending on the caller's responses to the system's automated prompts, routes a call to a CSR or to other information sources. These other information sources can include the other help lines maintained by Medicare's claims administration contractors or recorded information. All CSRs must have a high school diploma or its equivalent, but they are not required to be knowledgeable about the Medicare program at the time they are hired. To help provide clear, accurate, and timely answers to callers' questions, CMS expects the CSRs to use written scripts, which contain information about the program.[Footnote 8] CSRs listen to a caller's question and then type in related keywords to generate a list of suggested scripts that could be used to answer the question. The CSRs select the script they consider best suited to answer the question and read either excerpts or the entire script. CSRs can also consult other information sources. For example, CSRs can use Web-based tools available on the Medicare Web site[Footnote 9] to help beneficiaries select a prescription drug discount plan, nursing home, or home health agency. Because the types of questions frequently posed to 1-800-MEDICARE change in response to program or other policy changes, new scripts may need to be created or existing ones updated. Either CMS or the primary contractor may decide to develop a new script or update an existing one for clarification or in response to program changes. CMS officials approve scripts that are developed by the primary contractor, and check them for accuracy and completeness. 1-800-MEDICARE Provided Accurate Information for Less Than Two-thirds of the Calls We Placed: The 1-800-MEDICARE help line provided accurate answers to 61 percent of the 420 calls we made. The accuracy rate varied significantly among the six questions we posed. Overall, 29 percent of our calls were answered inaccurately. In general, CSRs erred because they did not understand enough about the Medicare program to access the script with information to answer the question or clearly explain the material in it. In addition, for 10 percent of the calls we placed, we were unable to get a response to our question at the time we contacted the 1-800-MEDICARE help line, mainly due to problems when CSRs transferred calls. Accuracy Rate Varied Significantly by Question: In response to our calls to the 1-800-MEDICARE help line, CSRs answered our questions accurately for 256 out of 420 calls, a rate of 61 percent. The criteria we developed to determine the information that constituted an accurate answer for each question are shown in table 1. (A more detailed version of the questions and information to answer them are in apps. II through VII.) The criteria were based on answers we developed from information on the Medicare Web site and were confirmed by CMS, which provided us with the scripts that contained information to answer the questions.[Footnote 10] We considered all calls we placed to the 1-800-MEDICARE help line to be part of our test of its accuracy, even if the call was transferred to a claims administration contractor to provide the answer. Table 1: Questions and Criteria Used to Determine Answer Accuracy: Medicare prescription drug discount card questions: Question 1: Drug card; Question: Medicare prescription drug discount card questions: What drug card can my father-in-law get that will cover all of his drugs at a particular pharmacy and have his drugs cost the least amount? Criteria: An accurate response would include at least one prescription drug discount card recommendation with the lowest total prices for the drugs the beneficiary used. Question 2: $600 credit; Question: Medicare prescription drug discount card questions: Can my mother qualify for the $600 credit with three specified sources and amounts of income? Criteria: An accurate response would indicate that a beneficiary with the amount of income from the specified sources could receive the $600 credit. Question 3: Medigap[A]; Question: Medicare prescription drug discount card questions: Can my grandmother get a Medicare-approved prescription drug discount card if she has Medicare and a Medigap policy? Criteria: An accurate response would indicate that a beneficiary could have a Medigap policy and still receive a Medicare-approved prescription drug discount card. Other Medicare coverage or eligibility questions: Question 4: Power wheelchair; Question: Medicare prescription drug discount card questions: Will Medicare pay for a power wheelchair for my father? Criteria: An accurate response would indicate that (1) a physician must prescribe the power wheelchair or determine it to be medically necessary, and (2) a power wheelchair requires a copayment on the part of the Medicare beneficiary. Question 5: Medicare part B[B] enrollment; Question: Medicare prescription drug discount card questions: Should my husband sign up for part B if I am still working and we have health insurance coverage from my employer? Criteria: An accurate response would indicate that a beneficiary could wait to enroll and then sign up for Medicare part B during a special enrollment period. Question 6: Eye exam and glasses; Question: Medicare prescription drug discount card questions: Will Medicare pay for an eye exam and a new pair of eyeglasses if my mother's prescription has changed? Criteria: An accurate response would indicate that Medicare does not pay for routine eye exams and eyeglasses. Source: GAO analysis. Notes: The analysis is based on a report of the most frequently accessed 1-800-MEDICARE scripts in May 2004 provided by CMS, scripts related to these questions provided by CMS on July 1, 2004, and information from Medicare's Web site on frequently asked questions and their answers, http://medicare.custhelp.com/cgi-bin/medicare.cfg/php/ enduser/std_alp.php, accessed on May 21, 2004. [A] Medigap is a privately purchased health insurance policy that supplements Medicare by paying for some of the health care costs that are not covered by the program. [B] Eligible individuals can choose to enroll in part B and pay a monthly premium. Medicare part B services include physician and outpatient hospital services, diagnostic tests, mental health services, outpatient physical and occupational therapy, ambulance services, some home health services, durable medical equipment, prosthetics, orthotics, and medical supplies. [End of table] The percentage of calls CSRs answered accurately varied by question, as shown in figure 1. For example, CSRs accurately answered 81 percent of the calls asking whether a beneficiary could receive a prescription drug discount card if they had a Medigap policy. The answer to the Medigap question was clearly described in a script, which allowed CSRs to respond with the highest accuracy rate for all of our questions. Similarly, for question 1--choosing a prescription drug discount card- -CSRs answered accurately 76 percent of the time. By July 2004, when we placed our calls, a large number of CSRs had been recently hired and trained specifically to answer this question, using a script and a Web- based tool.[Footnote 11] In contrast, for question 2 calls about the $600 prescription drug credit, CSRs answered inaccurately 79 percent of the time. CSRs scored poorly on this question primarily because they based their answers on the beneficiary's total income without considering that some specific types of income should not be included in the calculation of eligibility for the credit. CSRs would have had to access two scripts to correctly answer the question, because the more general script on the topic did not contain all of the needed information. Question 5, which addressed Medicare part B enrollment, also had a relatively high inaccuracy rate--41 percent. We were not able to obtain an answer to some of our questions at the time that we called, most commonly when CSRs or the interactive voice response system transferred calls concerning questions 4 and 6 to other help lines. Figure 1: Percentage of Calls with Accurate, Inaccurate, and No Answers, by Question: [See PDF for image] Note: Numbers may not add to 100 percent due to rounding. [End of figure] Inaccurate Responses Were Largely Due to Ineffective Use of Scripts: CSRs responded inaccurately to our questions largely because they did not seem to know enough to effectively use the scripts. According to a CMS official and the primary contractor's staff, CSRs are expected to use scripts to guide their discussion with callers; they are not supposed to rely solely on acquired knowledge of Medicare to answer questions. We found, however, that in responding to our questions CSRs usually had one of four problems using scripts. The CSRs (1) did not seem to access a script, even when one was available; (2) did not seem to access a script with the right information to answer the question; (3) did not obtain enough information from the script; or (4) misunderstood some of the words in the scripts. We found instances when CSRs did not seem to access scripts when responding to calls. For example, when responding to our calls concerning the prescription drug discount card question, 2 CSRs indicated that they were not able to inform the caller about which card had the lowest drug prices--even though 53 other CSRs successfully used a script and a Web-based tool to answer this question. One other CSR referred our caller to AARP for an answer, rather than respond with the appropriate script and Web-based tool.[Footnote 12] These 3 CSRs did not seem to know how to correctly answer this question, which was addressed by one of the most commonly accessed script for the first half of the year. During 20 of the calls to answer our question on whether a spouse should enroll in Medicare part B if he had current employment-based health insurance, CSRs told our callers that enrolling in Medicare was a personal decision and they could not answer the question, which we classified as an inaccurate answer. They did not seem to recognize that they could access a script that contained information designed to answer that question. CSRs sometimes seemed to be accessing the wrong script to answer our question. For example, in answering the question on whether a beneficiary could receive a prescription drug discount card if she had a Medigap policy, one CSR incorrectly stated that the caller needed to complete a survey before receiving an answer. There is a script available that provides the answer to the Medigap question, but the script does not mention a survey. This CSR seemed to be using a different script about the prescription drug discount card, which has the right information to answer our question on the best prescription drug discount card to choose. In some cases, CSRs did not obtain enough information from the scripts they were using to accurately answer the question we asked. For example, to answer our question concerning whether a beneficiary could qualify for the $600 credit toward prescription drug purchases, the CSR should consider the source, as well as the amount of the beneficiary's income. Some sources of income are not counted in determining a beneficiary's eligibility for the $600 credit. According to CMS, to answer this question accurately the CSR would have to access two different scripts. The first script provides general information about the $600 credit and refers CSRs to the second script, which lists the sources of income that are not included in the eligibility calculation. However, the CSRs who answered this question incorrectly in 55 calls-- or 79 percent of the time--focused on the total amount of income and did not seem to seem to consider the sources of the income or to access and use information from the second script. In 14 of the calls--or 20 percent of the time--CSRs were able to answer this question correctly, because they did consider the sources and amounts of income that we indicated the beneficiary had. Finally, CSRs sometimes misinterpreted or did not understand the words they were reading from the scripts or other written materials. For example, to answer our Medigap question, a CSR incorrectly told the caller that the beneficiary would automatically receive a prescription drug discount card if enrolled in a Medigap plan. The CSR may have been confusing Medigap policies with Medicare managed care plans, because both are discussed in the script that answered this question. In another example, for our question related to power wheelchair coverage, a CSR misread the requirement that a beneficiary must have adequate trunk--or upper body--strength. The CSR mistakenly informed us that a Medicare beneficiary needs to have adequate "trunk space" in order to qualify for a power wheelchair. When we asked for clarification, the CSR stated that Medicare requires a qualifying beneficiary to have adequate trunk space in his or her car to hold a power wheelchair. Similarly, during one of our calls about eye exam and glasses payment, the CSR informed us that an eye exam would be covered and then stated, "the only part of the exam that is not covered is 'refraction,' but I don't know exactly what that is." Because the CSR did not understand that a typical eye exam would be considered a refraction, she gave the caller the incorrect impression that Medicare would pay for a routine eye exam. CMS and contractor staff acknowledged that scripts for the 1-800- MEDICARE help line are not routinely pretested to ensure that both the CSR and the caller can understand the script before it is used to answer callers' questions.[Footnote 13] On occasion, the 1-800-MEDICARE contractor has obtained CSRs' feedback on draft scripts before they are used on the 1-800-MEDICARE help line to ensure that scripts can be easily read and understood. But this is not done as a routine step before new and revised scripts are used in handling calls. In addition, even if the CSRs consider the script understandable, it may still be confusing to Medicare beneficiaries. We found that pretesting to ensure that written material is understandable to its intended audience is a standard practice used to develop effective communications materials. For example, prior to issuing the first Medicare & You handbook[Footnote 14] nationwide to beneficiaries, CMS conducted consumer testing of its publication to evaluate its effectiveness as a communication tool. CMS has revised subsequent editions of the handbook to make it easier to read and use, based on feedback from beneficiaries. Moreover, other HHS agencies, such as the Centers for Disease Control and Prevention[Footnote 15] and the Substance Abuse and Mental Health Services Administration,[Footnote 16] have developed guidance on steps for ensuring that written material is understandable for intended readers and pretesting the materials before use.[Footnote 17] Ten Percent of Calls Were Not Answered, Often When Calls Were Transferred to Other Contractors' Help Lines: We did not receive answers to our questions for 10 percent (42) of the 420 calls we placed at the time we originally called. Several reasons accounted for this, as table 2 shows. For half (21) of the unanswered calls, the CSRs or the interactive voice response system transferred the calls placed during morning, evening, and weekend hours to claims administration contractors that were not open for business at the time of our call. Although the 1-800-MEDICARE help line is open 24 hours a day, 7 days a week, these other help lines are not. The transferred calls pertained to our questions concerning Medicare coverage about power wheelchairs and eye exams and glasses. The 1-800-MEDICARE CSRs or the interactive voice response system transfer such questions to the claims administration contractors' help lines because these contractors generally have greater knowledge about Medicare coverage issues.[Footnote 18] Once our calls were transferred to closed help lines, we generally heard recordings that stated the contractors' regular business hours and suggested calling back at that time. However, for 7 of those 21 calls, the contractors' recorded messages did not provide a telephone number to use to call back during the stated business hours.[Footnote 19] Table 2: Summary of Unanswered Calls by Question: Medicare prescription drug discount card questions: Question 1: Drug card; Call transferred to closed help line: 0; Call disconnected: 3; Web-based tool inoperative: 4; Call routed to a wrong telephone number: 0; Total unanswered calls: 7. Question 2: $600 credit; Call transferred to closed help line: 0; Call disconnected: 1; Web-based tool inoperative: 0; Call routed to a wrong telephone number: 0; Total unanswered calls: 1. Question 3: Medigap; Call transferred to closed help line: 0; Call disconnected: 2; Web-based tool inoperative: 0; Call routed to a wrong telephone number: 0; Total unanswered calls: 2. Other Medicare coverage or eligibility questions: Question 4: Power wheelchair; Call transferred to closed help line: 10; Call disconnected: 5; Web-based tool inoperative: 0; Call routed to a wrong telephone number: 0; Total unanswered calls: 15. Question 5: Medicare part B enrollment; Call transferred to closed help line: 0; Call disconnected: 0; Web-based tool inoperative: 0; Call routed to a wrong telephone number: 0; Total unanswered calls: 0. Question 6: Eye exam/glasses; Call transferred to closed help line: 11; Call disconnected: 5; Web-based tool inoperative: 0; Call routed to a wrong telephone number: 1; Total unanswered calls: 17. Total number of responses by category; Call transferred to closed help line: 21; Call disconnected: 16; Web-based tool inoperative: 4; Call routed to a wrong telephone number: 1; Total unanswered calls: 42. Source: GAO analysis. [End of table] The second most common reason we did not receive answers to our calls was that our calls were disconnected. Sixteen of the 42 unanswered calls were disconnected. For example, calls were disconnected before we were able to obtain a response to our questions. In one instance, the call was placed on hold for 30 minutes and then was disconnected. Four calls made on the same day did not receive a response because computer maintenance prevented the CSRs from accessing the Web-based tool they needed to use to answer our question about the prescription drug discount card. Finally, one other call was unanswered because the call was routed to a wrong telephone number. CSR Training Met CMS's Requirements, But Was Not Sufficient to Ensure Accurate Responses: As required by CMS, both newly hired and experienced CSRs receive training to help them answer questions posed on the 1-800-MEDICARE help line. The training for newly hired CSRs includes instruction on accessing and using scripts, customer service etiquette, and information about the Medicare program. As part of the training, CMS requires newly hired CSRs to score 90 percent or higher on a written exam before they handle calls on the help line. All CSRs also receive continuing training, and take written quizzes on the new material. Although the 1-800-MEDICARE contractors met CMS's training requirements by providing instruction and testing, the testing does not fully measure CSRs' ability to accurately answer real questions from callers. The primary contractor develops and conducts the training new CSRs receive. Most of the training consists of 2 weeks of classroom instruction. In general, the instruction introduces CSRs to scripts and provides general information about the Medicare program. For example, in a training session we observed for newly hired CSRs in June 2004, the instructors helped the CSRs prepare for the types of inquiries that might be expected from callers on the 1-800-MEDICARE help line. The instructors posed different questions to the class, and each CSR attempted to identify and access a script with the right information to answer the instructor's question.[Footnote 20] One CSR would be selected to read the script that they chose, and participants discussed whether they thought this was the script with the best information to answer the question. After completing their initial instruction, CMS requires the new CSRs to score at least 90 percent on a written exam and successfully complete a call handling simulation exercise before they answer calls on the help line without supervision. To successfully complete the call handling simulation exercise, CSRs must accurately answer two consecutive simulated help line calls out of six possible attempts. In addition, new CSRs generally spend about 4 hours listening to calls answered by an experienced CSR. In addition to the initial training for newly hired CSRs, CMS requires all CSRs to receive continuing training.[Footnote 21] Continuing training is delivered through three methods: refresher classes, online broadcast announcements, and small group meetings. Weekly refresher classes provide a means of instructing CSRs about Medicare program changes. Following each refresher training class, CSRs complete a short quiz to show that they understand the new information. While there is no minimum score that CSRs must achieve on the short quiz, CMS staff informed us that help line supervisors review each quiz to ensure that any questions that posed problems for CSRs would be addressed with further training. To provide CSRs with information quickly, the primary contractor sends online broadcast announcements to each CSR's computer workstation. These online announcements usually contain information that may affect CSRs' responses to help line questions, such as news about a change in a specific script. Lastly, small group meetings of about 12 CSRs and their supervisor are held for 30 minutes each week so that CSRs can discuss topics that can help them improve their call handling skills. After gaining experience in answering calls, some CSRs receive 4 additional days of special training and are promoted to a senior position. These CSRs receive classroom training on using Web-based computer programs that can assist Medicare beneficiaries in selecting a managed care plan, a nursing home, or other Medicare-related services. Like other CSRs, they must score 90 percent or higher on a written exam, and successfully complete a simulated call handling exercise before they can handle calls using the Web-based computer programs. Currently, about 200 senior CSRs answer calls on the 1-800-MEDICARE help line. Although all CSRs receive training and are tested as required, the responses we received indicate that not all CSRs had the necessary knowledge and skills to answer our questions accurately. In our opinion, testing how effectively CSRs use scripts to answer frequently asked questions provides the best measure of their preparation to do so. While 24 of the exam's 52 questions ask CSRs to identify scripts that could be used to answer specific inquiries, the remaining 28 questions target other skills.[Footnote 22] In addition to the written test, new CSRs must appropriately answer questions posed in two consecutive simulated calls before they staff the help line.[Footnote 23] This simulated call handling and some of the written exam questions are the only measures of the CSRs' ability to accurately answer calls using scripts. In combination, the test and the two simulated calls do not appear to be a sufficient measure of new CSRs' ability to accurately answer the most frequently asked questions, given our findings on the accuracy of their responses. Further, while all CSRs receive continuing training, they are not required to demonstrate that they have effectively mastered the new material in handling calls. Developing a more targeted assessment of where CSRs need to augment their skills helped focus another help line's training efforts and allowed it to meet its accuracy goals. In 2001, we assessed the telephone help line maintained by the IRS to answer taxpayers' questions and found that it had not met the agency's goals for accurately answering general questions about tax law and specific questions about individuals' tax returns in 2001.[Footnote 24] In response, the IRS analyzed the specific types of inquiries within the area of tax law and individual returns that were answered inaccurately and identified the knowledge and skills its CSRs needed to answer questions more accurately. The IRS also identified the CSRs most in need of training to improve accuracy in those knowledge and skill areas and provided additional training to them before call volume increased for the 2002 tax season. By the end of the training period, these CSRs were required to be certified by their managers as capable of providing correct responses to taxpayer questions. The IRS also assigned responsibility for selected tax law topics to individual call center managers, making them accountable for ensuring that CSRs were trained and could accurately address inquiries on these topics. After these initiatives were complete, we found that the help line had improved its accuracy enough to meet its 2001 goals. In the span of 1 year, the accuracy rate on answering tax law questions increased from 79 to 85 percent and the accuracy rate for answering questions about individuals' tax returns increased from 88 to 91 percent.[Footnote 25] Monitoring Generally Focused on Performance of Individual CSRs, Rather than Accuracy Rates for Specific Questions: CMS monitors the 1-800-MEDICARE help line mostly by requiring its primary contractor to evaluate four individual conversations that each CSR has with callers each month. Based on these conversations, the primary contractor evaluates the performance of individual CSRs in several categories, including accuracy, and reports the overall results to CMS. CMS also occasionally directly monitors a small number of individual CSRs' calls. However, the contractor's and CMS's monitoring does not systematically track the accuracy rates for commonly asked questions. As a result, the monitoring does not assess how accurately CSRs as a group answer particular questions, which could help CMS target additional training efforts. Two smaller evaluation efforts did focus on specific questions answered inaccurately, and these targeted monitoring efforts provided information that CMS used to improve CSR training and the scripts used on the help line. CMS's Primary Contractor Monitors Performance of Individual CSRs: At the time of our review, CMS had delegated most of the responsibility for monitoring the accuracy of the 1-800-MEDICARE help line to the primary contractor, while maintaining oversight by reviewing the primary contractor's results. To monitor 1-800-MEDICARE, the primary contractor focuses on the performance of individual CSRs, evaluating four calls per month for each person. The primary contractor evaluates either live conversations--known as blind monitoring--or recorded conversations on the help line, while viewing displays of the CSRs' computer activity during calls. Viewing the CSRs' computer activity allows the primary contractor's staff to observe the scripts or other materials that CSRs access to answer callers' questions. After monitoring a call, the primary contractor's supervisory staff uses a checklist to evaluate the CSR's response to the caller. Help line supervisors share the results with each CSR to help improve performance. The primary contractor provides monthly reports to CMS on the results of its monitoring.[Footnote 26] The primary contractor has a subcontractor, which is responsible for conducting some independent call monitoring, as well as reviewing the results of some of the primary contractor's call monitoring. In addition to the four calls per CSR per month that CMS requires the primary contractor to monitor, the subcontractor monitors up to one additional call per month per CSR. The subcontractor reports its monitoring results monthly to CMS and the primary contractor. In addition to meeting CMS requirements, the amount of call monitoring per CSR approximates industry standards. A survey of 735 North American call centers that represent help lines in various industries, including telecommunications, financial services, and health care, found a wide variance in the number of calls monitored per month. The most commonly reported monthly monitoring frequencies were 4 to 5 calls per CSR or 10 or more calls per CSR.[Footnote 27] The evaluation checklist used by CMS's contractor for monitoring calls indicates that a CSR's performance should include certain components-- such as using an appropriate greeting, showing respect to the caller, actively listening to the caller, responding accurately to the question, providing a complete response, using appropriate tone and speed, offering to provide additional information if necessary, and ending the call politely. The primary contractor's staff uses the checklist to evaluate both the customer service skills and knowledge skills demonstrated during a call and classify these into one of four categories--"unacceptable," "needs improvement," "achieves expectations," and "exceeds expectations." CMS requires the primary contractor to reach a quality rating of "achieves expectations" or higher for at least 90 percent of the total number of CSR calls evaluated each month. The primary contractor evaluates a call as either "accurate" or "inaccurate," and because accuracy is weighted more heavily than other components, a CSR cannot provide inaccurate information on a call and still have the call scored as "achieves expectations." However, in contrast to our methodology for this report, a CSR can provide incomplete information--information that is correct but does not fully answer the question--and still have the call scored as "achieves expectations." In addition, the evaluation checklist does not indicate the specific criteria used to determine a call's accuracy. Although CMS's main role in monitoring the help line is to review the efforts of the primary contractor, the agency also conducts some monitoring of CSRs on its own. Like the primary contractor, CMS occasionally uses blind monitoring to evaluate the performance of individual CSRs--listening to real-time calls and watching the scripts and other materials the CSRs use. CMS does not conduct blind monitoring routinely and document the results, and therefore, CMS staff could not provide us any information on the extent of this monitoring.[Footnote 28] According to a CMS official, the agency conducts blind monitoring on a "limited and as-needed" basis. CMS Has Tracked Accuracy Rates by Question on Occasion: Although CMS's primary 1-800-MEDICARE contractor monitors the accuracy of individual CSRs, CMS does not use the regular monthly monitoring to identify trends in inaccurate responses by question. Specifically, the primary contractor does not routinely classify or categorize CSRs' answers by specific question to identify the questions that collectively were answered less accurately. While routine information about a question's accuracy rate could be used to target improvement efforts, CMS has only taken this approach twice in recent years. Both of these efforts were small compared to the primary contractor's monitoring. The larger effort to monitor accuracy by question lasted 29 months and involved 300 calls a month, whereas the primary contractor evaluated about 7,350 calls in July 2004.[Footnote 29] CMS contracted for a study to evaluate the 1-800-MEDICARE help line's accuracy in answering specific questions, but did not receive results quickly enough to immediately address problems. Beginning in January 2002 and until May 2004, the CMS contractor hired to assess the "Medicare & You" program[Footnote 30] placed about 300 calls per month to the 1-800-MEDICARE help line. These callers used a set of hypothetical scenarios to assess how specific questions were answered. This study also established criteria specifying the information an accurate answer should provide[Footnote 31] and made a distinction between fully responsive answers--in other words, complete and accurate answers--or partially responsive answers--not complete but providing some accurate information.[Footnote 32] For the first 19 months studied, the average percentage of calls that received fully responsive answers ranged from under 40 percent to over 90 percent, depending on the question and the period of time studied. The study helped CMS identify questions that the CSRs were answering less accurately. However, CMS staff told us that the agency received the reports 4 to 5 months after monitoring occurred, which did not allow CMS to immediately address any identified problems. Nevertheless, CMS staff indicated to us that the results of these evaluations were used to identify areas where CSR refresher training was needed. Due to funding constraints, this project was terminated in May 2004. CMS told us it planned to resume a similar project in November 2004 around the time when the next cycle for the "Medicare & You" program contract begins. In another study, CMS measured 1-800-MEDICARE's accuracy by question rather than individual CSR and found that certain questions were answered more accurately than others. In April 2004, a private consultant group that was under contract with CMS placed 49 calls to 1- 800-MEDICARE to determine whether CSRs were relaying accurate information about Medicare's new prescription drug discount card and benefit.[Footnote 33] The study established specific criteria on the information that CSRs should include for an answer to be accurate.[Footnote 34] Evaluating accuracy by question, the study found that CSRs accurately answered between 39 percent and 69 percent of the questions asked about the new Medicare prescription drug discount card and benefit. In contrast, in that same month the subcontractor--using its evaluation checklist--determined that the overall accuracy rate for the calls it measured on all topics to be about 94 percent. As a result of this study, CMS improved its scripts and related training. The private consultant's report indicated that CSRs were having difficulty distinguishing between the Medicare prescription drug coverage benefit that will be in effect in 2006 with the Medicare prescription drug discount card that is currently available. CMS responded by clarifying the scripts used to answer these questions and improving the related materials used to train CSRs. For example, CMS worked with the contractor to rename the titles of the different scripts to include the term "benefit" or "card" as a method of differentiating between them. When the subcontractor noted about 3 months later that a few CSRs were continuing to confuse the prescription drug discount card and prescription drug benefit, CMS further clarified the scripts and its primary contractor conducted additional refresher training to attempt to correct the problem. Conclusions: Each year, millions of Medicare beneficiaries, their family members, and other callers rely on the 1-800-MEDICARE help line for information. Providing them with accurate answers is critical to keeping them informed about Medicare's benefits. However, we found that 6 out of 10 calls were answered accurately, 3 out of 10 calls were answered inaccurately, and we were not able to get a response for 1 out of 10 calls. To answer inquiries accurately, CSRs have to be able to effectively access and use scripts. Given the lack of prior Medicare knowledge among CSRs, the 1-800-MEDICARE help line's script-based approach is a reasonable means to facilitate accurate and consistent responses to caller's questions. However, this approach makes CSRs--and thus the help line they support--dependent on the clarity and accuracy of the scripts available. Pretesting scripts might have identified ones that were difficult to understand by either the CSR or by potential callers, but this is not routinely done. Further, the training that CSRs receive on using scripts is also essential to their ability to answer questions accurately. However, the written exam that newly hired CSRs must pass and the continuing training quizzes do not measure the ability to use information in scripts to provide accurate and complete answers on the help line. Monitoring the help line could identify areas where CSRs' knowledge and skills are lacking. Although CMS ensures that the amount of the contractor's monitoring per CSR falls within industry standards, the bulk of the monitoring methods are not designed to systematically assess how accurately CSRs as a group answer particular questions. Evaluating how accurately particular questions are addressed is an important step to improving scripts and CSR training for those topics. Finally, 1-800-MEDICARE is advertised as providing information 24 hours a day, 7 days a week, but we could not always obtain answers to our questions when we called. When we called with questions about Medicare payment for power wheelchairs and coverage of eye exams and glasses, the help line frequently transferred our calls to claims administration contractors that were closed at the time. For a third of these transferred calls, we were not given a call-back number. This practice of transferring calls to claims administration contractors that are closed, in effect, reduces the benefit of a 24-hour help line to a business-hour help line for many beneficiaries. Recommendations: In order to improve the accuracy of responses made on the 1-800- MEDICARE help line and callers' ability to have their questions answered, we recommend that the CMS Administrator take four actions: * Assess the current scripts for the most commonly asked questions to ensure that they are understandable to CSRs and potential callers and if not, revise them as needed and pretest new and revised scripts to ensure that CSRs can effectively use them to accurately answer callers' questions. * Enhance testing of CSRs' skills in accurately answering the most commonly asked questions using scripts and, if needed, provide additional training to improve the accuracy and completeness of their responses. * Supplement current monitoring efforts to include a systematic review of the accuracy of information provided by the CSRs as a group for the most frequently asked questions and use the results to modify scripts or provide more training, as needed. * Revise routing procedures and technology to ensure that calls are not transferred or referred to claims administration contractors during nonbusiness hours. Agency Comments and Our Evaluation: In its written comments on a draft of this report, which are reprinted in appendix VIII, CMS agreed with our recommendations and stated that it had begun several efforts to address them. CMS also provided more detail on the challenges it faced in administering 1-800-MEDICARE due to the massive increase in call volume that occurred after the passage of the MMA. CMS agreed with our recommendation to assess current scripts and pretest new and revised scripts to ensure that they are understandable. In its comments, CMS stated that the written information used to develop 1-800-MEDICARE scripts often comes from Medicare publications that have been consumer tested as part of the publication preparation process. Language that has undergone some consumer testing is often incorporated into the scripts to improve clarity. While this step may be helpful, we believe that pretesting scripts verbally is also important, as consumer testing of material intended for written publication may not be adequate to determine whether the scripts are understandable to CSRs and the public. CMS also stated that it is considering implementing an editorial board to review scripts, which we believe would be another positive step to help assure the scripts' clarity. CMS agreed with our recommendation to enhance testing of CSRs' ability to accurately answer questions and provide additional training, as needed. CMS indicated that it was reassessing its testing requirements to determine better ways to ensure that CSRs are prepared to handle calls, once they are certified. The agency stated that it planned to benchmark its efforts against industry standards to determine more effective approaches. In its comments, CMS expressed concern that we had characterized customer service skills as less meaningful than knowledge skills. While we believe both are important, in keeping with our congressional mandate, this report focused on the accuracy of information provided by 1-800-MEDICARE and did not address the quality of its CSRs' customer service skills. CMS agreed with our recommendation to supplement its current monitoring efforts by including a systematic review of the accuracy of information provided by CSRs as a group for the most frequently asked questions and using the results to modify scripts and provide more training, as needed. CMS indicated that it believed it had done a good job developing a quality assurance program that focuses on the most important requirements for both accuracy and customer service skills needed to answer calls from the elderly population. We agree that CMS has focused on quality assurance for 1-800-MEDICARE. Our recommendation did not deal with changing, but with enhancing, its quality assurance efforts. To address our recommendation, CMS indicated that it would implement a plan to develop trend information on the results of its quality assurance activities and would focus on improving the accuracy of responses to frequently asked questions. CMS agreed with our recommendation to revise procedures so that calls are not transferred to claims administration contractors during their nonbusiness hours. The agency indicated that although the 1-800- MEDICARE CSRs are available 24 hours a day, 7 days a week, they do not have access to claim-specific information. Therefore, the 1-800- MEDICARE CSRs would have to direct callers asking questions about specific claims to contact the claims administration contractors during their normal business hours. Our report focused on questions for which the CSRs had scripted responses and did not need to access claim- specific information. Nevertheless, for 7 of the 21 calls that were routed to claims administration contractors that were closed at the time we called, the contractors' recorded messages did not provide a telephone number to call back during stated business hours. In its comments, CMS indicated that it had implemented additional routing plans to address this problem and is expanding access to claims data that will help reduce this problem in the future. CMS also raised concerns that we did not release the detailed audit documentation on our test calls while our work was still in progress. GAO's policy does not allow us to provide audit documentation to an agency while work is ongoing. At the time of CMS's request, we described our policy and offered to verbally provide more detail on telephone disconnections, but CMS did not follow up with us to obtain this information. After our report is published, we will address this request. Finally, CMS expressed concern that we did not describe the criteria we used to evaluate the accuracy of responses to our six questions and stated that incomplete answers should not be considered inaccurate responses. As noted in the draft report, table 1 lists the criteria that we established for each of our six questions. We developed these criteria so that we could objectively evaluate responses received from CSRs. For four of these questions (numbers 1, 2, 3, and 6), there was only one element in the correct response, so an incomplete response was not possible.[Footnote 35] For the remaining two questions, we considered the answer accurate if it included two elements. We believe that by not including both elements for each of these questions, callers would be left with a false impression, rather than with an accurate answer. For example, in evaluating the response to the question of whether Medicare would pay for a power wheelchair, we thought it was important for the caller to know that (1) the wheelchair needed to be prescribed by a physician and (2) the beneficiary would be responsible for a copayment. Because the copayment for a power wheelchair is at least $1,000, we believe that it would be misleading not to mention either a copayment or cost sharing when a caller asks whether Medicare pays for this item. Likewise, needing to have a physician prescribe the power wheelchair is a Medicare requirement, and we did not think a response could be accurate without mentioning it. For the question on Medicare part B enrollment, we thought that it was important to know that a beneficiary could wait to enroll but, once other health insurance coverage ended, had a limited time period to enroll in part B without incurring higher premiums. Without knowing both elements of this answer, beneficiaries would not have enough information to guide their decision on part B enrollment and, therefore, the answer provided would be misleading. We are sending copies of this report to the Administrator of CMS, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request. This report is also available at no charge on GAO's Web site at http://www.gao.gov. If you or your staff have any questions about this report, please call me on (312) 220-7600. An additional GAO contact and other staff who made contributions to this report are listed in appendix IX. Signed by: Leslie G. Aronovitz: Director, Health Care--Program Administration and Integrity Issues: [End of section] Appendix I: Scope and Methodology: To determine the accuracy of information provided, we placed a total of 420 calls to the 1-800-MEDICARE help line. We made 420 calls in order to have a sample that was large enough to determine if differences in accuracy were significant. We selected six questions about Medicare-- three related to the Medicare prescription drug discount card and three related to Medicare coverage or eligibility for benefits.[Footnote 36] We asked each of the selected questions a total of 70 times. We randomly placed calls at different times of the day and different days of the week between July 8 and July 30, 2004, to match the daily and hourly pattern of calls reported by 1-800-MEDICARE in April 2004. To select the 6 questions, we initially chose 20 questions that related to the 100 topics most frequently addressed by the 1-800-MEDICARE help line's CSRs in May 2004 and developed criteria for an accurate response from information on the Medicare Web site's frequently asked questions section.[Footnote 37] We then presented the 20 questions and answers to Centers for Medicare & Medicaid Services (CMS) officials, who provided us with a script number and text for each question. CMS officials did not object to using any of the 6 questions that we ultimately chose, or suggest that the answers that we had provided for these questions were incorrect. We informed both CMS and one of the 1-800-MEDICARE contractors that we would be placing these calls. However, we did not tell them which 6 of the 20 questions we selected, or the specific dates and times when we would be placing our calls. Before placing our calls, we created a scenario with fictional names and zip codes for each of the six questions to make them sound more realistic. (Appendices II to VII contain the scenarios that we used.) We made pretest calls for each question before we finalized its wording. During our actual calls, the CSRs were not aware that their responses would be included in a research study. We recorded the length of, and routing process for, each call. We evaluated the accuracy of the responses by CSRs to the 420 calls we placed by whether key information was provided. The results from our 420 calls are limited only to those calls and are not generalizable to the population of calls routinely made to call centers by beneficiaries or other callers. Although the six questions we posed were among topics most often accessed by CSRs, they do not encompass all of the questions callers might ask. In addition, we did not verify the reliability of CMS's monitoring data. To examine the training provided to 1-800-MEDICARE CSRs, we interviewed officials representing CMS and the 1-800-MEDICARE contractor responsible for training CSRs. We reviewed the primary contractor's training requirements and the instructional materials that are used to educate new CSRs. We also observed a training session for new CSRs at a 1-800-MEDICARE call center. In addition, we reviewed previous GAO reports on the operations of other help lines, including the training provided to the CSRs answering calls on the Internal Revenue Service's help line. To evaluate CMS's role in overseeing the accuracy of information provided through the 1-800-MEDICARE help line, we interviewed officials from CMS and one of the two 1-800-MEDICARE contractors about their monitoring and oversight activities. For our first objective, we focused on the accuracy of information provided by the CSR regardless of the contractor who managed their work. For our other two objectives, we relied on information provided by one of the two contractors--which we refer to as the primary contractor. We also identified CMS requirements for call center operations and reviewed contractor reports to identify the types of problems encountered through the help line. We performed our work from May 2004 through December 2004 in accordance with generally accepted government auditing standards. [End of section] Appendix II: Prescription Drug Discount Card Questions and Information Used to Develop Accuracy Criteria: For question 1, which is about choosing a prescription drug discount card, we used scenarios with different combinations of prescription drugs and one of four different locations, in order to ensure our anonymity. If a CSR named one of the two prescription drug discount cards with the lowest cost for the combination of prescription drugs in the scenario posed, we considered it to be a correct response to our question. To ensure that we obtained the correct answer for each question, we periodically checked the prescription drug prices using the prescription drug tool on the Medicare.gov site. This is the same tool CSRs used to answer our questions. The answers shown in this appendix were accurate as of July 15, 2004. Question 1a posed to CSRs: My father-in-law lives in Wayne, Pennsylvania, and wants to continue to shop at Yorke Apothecary (located at 110 S. Wayne Ave., Wayne, Pennsylvania, 19087). What drug card can he get that will cover all of his drugs at Yorke Apothecary, and cost the least amount? He takes the following drugs: Hydrochlorothiazide; 25MG; 30 TABS. Lipitor; 20MG; 30 TABS. Warfarin sodium; 5MG; 30 TABS. [End of table] Other information to provide to the CSR if asked: * He is single. * He lives in Wayne, Pennsylvania, in Delaware County. His zip code is 19087. * He currently has fee-for-service Medicare with no other drug benefits. * He does not use an American Indian Health pharmacy. * He does not live in a long-term care facility. * He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. * His sources of income are a pension and Social Security, but the amount from each is unknown. * He has some bank accounts, but their value is unknown. * The amount he currently pays for drugs is unknown. * Default answer for other questions: "I don't know." Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: myPharmaCare Prescription Drug Discount Card; 1-800-601-3002; Monthly drug costs: $116.69; Annual enrollment fee: $25.00. U Share Prescription Drug Discount Card; 1-800-707-3914; Monthly drug costs: $115.59; Annual enrollment fee: $19.95. Question 1b posed to CSRs: My father lives in Homewood, Illinois, and wants to continue to shop at the K-Mart Pharmacy in Homewood, Illinois (located at 17550 Halsted Rd., Homewood, Illinois, 60430). What drug card can he get that will cover all of his drugs at the K-Mart Pharmacy, and cost the least amount? He takes the following drugs: Aricept; 10 MG; 30 TABS; Celebrex; 100 MG; 30 TABS. [End of table] Other information to provide to the CSR if asked: * He is single. * He lives in Homewood, Illinois, in Cook County. His zip code is 60430. * He currently has fee-for-service Medicare with no other drug benefits. * He does not use an American Indian Health pharmacy. * He does not live in a long-term care facility. * He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. * His sources of income are a pension and Social Security, but the amount from each is unknown. * He has some bank accounts, but their value is unknown. * The amount he currently pays for drugs is unknown. * Default answer for other questions: "I don't know." Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: U Share Prescription Drug Discount Card; 1-800-707-3914; Monthly drug costs: $174.91; Annual enrollment fee: $19.95: Any of several prescription drug discount cards available with this combination of drugs priced at $182.80.[Footnote 38] Question 1c posed to CSRs: My father lives in Cincinnati, Ohio, and wants to continue to shop at the CVS Pharmacy (located at 3195 Linwood Ave., Cincinnati, Ohio). What drug card can he get that will cover all of his drugs at the CVS Pharmacy, and cost the least amount? He takes the following drugs: Lipitor; 10 MG; 30 TABS. Ambien; 5 MG; 30 TABS. Lipitor: Vioxx; 25 MG; 30 TABS. [End of table] Other information to provide to the CSR if asked: * He is single. * He lives in Cincinnati, Ohio, in Hamilton County. His zip code is 45226. * He currently has fee-for-service Medicare with no other drug benefits. * He does not use an American Indian Health pharmacy. * He does not live in a long-term care facility. * He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. * His sources of income are a pension and Social Security, but the amount from each is unknown. * He has some bank accounts, but their value is unknown. * The amount he currently pays for drugs is unknown. * Default answer for other questions: "I don't know." Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: myPharmaCare Prescription Drug Discount Card; 1-800-601-3002; Monthly drug costs: $202.79; Annual enrollment fee: $25.00: Anthem Prescription Drug Discount Card; 1-800-730-2804; Monthly drug costs: $209.87; Annual enrollment fee: $14.95: Question 1d posed to CSRs: My father lives in Brooklyn, New York, and wants to continue to shop at the Neergaard Pharmacy (located at 454 Fifth Avenue, in Brooklyn, New York). What drug card can he get that will cover all of his drugs at the Neergaard Pharmacy, and cost the least amount? He takes the following drugs: Cardura: 2 MG; 30 TABS. Hydrochlorothiazide; 25 MG; 30 TABS. Lisinopril; 5 MG; 30 TABS. [End of table] Other information to provide to the CSR if asked: * He is single. * He lives in Brooklyn, New York, in Kings County. His zip code is 11215. * He currently has fee-for-service Medicare with no other drug benefits. * He does not use an American Indian Health pharmacy. * He does not live in a long-term care facility. * He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. * His sources of income are a pension and Social Security, but the amount from each is unknown. * He has some bank accounts, but their value is unknown. * The amount he currently pays for drugs is unknown. * Default answer for other questions: "I don't know." Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: EnvisionRx Plus Prescription Drug Discount Card; 1-866-250-2005; Monthly drug costs: $46.33; Annual enrollment fee: $30.00: Any of several prescription drug discount cards available with this combination of drugs priced at $50.45.[Footnote 39] [End of section] Appendix III: $600 Credit Question and Information Used to Develop Accuracy Criteria: Question 2 posed to CSRs: I've heard about the $600 credit that can help pay for prescriptions and wanted to know if my mother was eligible for it. Could she qualify for the credit? I know she has three sources of income. * She has about $765 per month from Social Security. * She also gets $250 each month in rental income from the apartment in the downstairs part of her house. She has a tenant that pays rent to her. * She's also getting a payout from my father's life insurance policy of $70 each month. Other information to provide to the CSR if asked: * She is single and lives alone. * She only has fee-for-service Medicare as health insurance. * She owns her house. * She lives in Miami, Florida, 33129. * Default answer for other questions: "I don't know." Information from Medicare.gov that GAO used to develop accuracy criteria: If your annual gross income is currently no more than $12,569 ($1,048 per month) as a single person or no more than $16,862 ($1,406 per month) for a married couple, you might qualify for a $600 credit to help pay for your prescription drugs and Medicare may pay your annual enrollment fee. If you and your spouse both qualify for the credit, the credit will be put on each of your cards. You cannot qualify for the credit if you already have outpatient prescription drug coverage from Medicaid, TRICARE for Life,[Footnote 40] Federal Employees Health Benefit Plan, or other health coverage that includes outpatient prescription drugs such as an employer group health plan. Note: If your other coverage is provided through a Medicare Advantage plan or a Medigap plan, you may still qualify. The following sources of income should be included when calculating your gross income for your $600 credit enrollment form: * Employee compensation (salary, wages, tips, bonuses, awards, etc.) * Unemployment compensation: * Pensions and annuities: * Social Security benefits (including Social Security Equivalent portion of Railroad Retirement): * Railroad Retirement benefits: * Veterans Affairs benefits: * Military and government disability pensions - armed forces, Public Health Service, National Oceanic and Atmospheric Administration, Foreign Service (based on date pension began, combat-related pension, etc.) * Individual Retirement Account distributions: * Interest (savings accounts, checking accounts, etc.) * Ordinary dividends (stocks, bonds, etc.) * Refunds, credits, or offsets of state and local income taxes: * Alimony received: * Business income: * Capital gains: * Farm income: * Rental real estate, royalties, partnerships, trusts, etc. * Other gains (sale or exchange of business property): * Other income (lottery winnings, awards, prizes, raffles, etc.) The following sources of income should not be included when calculating your income for $600 credit enrollment form: * Inheritances and gifts (taxed to estate or giver if not under limits for exemption): * Interest on state and local government obligations (e.g., bonds): * Workers compensation payments: * Federal Employees Compensation Act payments: * Supplemental Security Income benefits: * Income from national senior service corps programs: * Public welfare and other public assistance benefits: * Proceeds from sale of a home: * Lump sum life insurance benefits paid upon death of insured: * Life insurance benefits paid in installments: * Accelerated life insurance death benefit payments (e.g., viatical settlements, terminal illness, chronic illness): * Medical Savings Accounts withdrawals for medical expenses: * Payments from long-term care insurance policies (subject to limitation): * Accident or health insurance policy benefits: * Accident compensatory damages: * Child support payments received: * Most foster care provider payments received: * Disaster Relief grants: * Disability payments as the result of a terrorist attack: [End of section] Appendix IV: Medigap Question and Information Used to Develop Accuracy Criteria: Question 3 posed to CSRs: I'm calling with a question about my grandmother. She is 69 and she has Medicare, and she also has a Medigap policy. Could you please tell me if she can still get a Medicare-approved drug discount card? Other information to provide to the CSR if asked: * She is single and lives alone. * She lives in Miami, Florida. I don't know the zip code off-hand. * She is not in a long-term care facility. * She is enrolled only in Medicare fee-for-service. She doesn't have a Medicare managed care plan. * She is not enrolled in Medicaid. * I don't think she's interested in the $600 credit right now; I was just wondering if she could get the prescription drug discount card. * Default answer for other questions: "I don't know." Information from Medicare.gov that GAO used to develop accuracy criteria: Having a Medigap policy does not preclude a Medicare beneficiary from being eligible for a Medicare prescription drug discount card. [End of section] Appendix V: Power Wheelchair Question and Information Used to Develop Accuracy Criteria: For question 4 about power wheelchairs, we provided the CSRs with one of four different city and state combinations, as shown below. The four city and state combinations were randomly assigned to different power wheelchair calls. We did this to ensure that if our call was transferred to one of the four claims administrator contractors that administer Medicare's durable medical equipment claims--including power wheelchair claims--we were not biasing our results toward any particular claims administrator. Question 4 posed to CSRs: My father is having trouble getting around. He has a hard time walking and doesn't have much upper body strength. Could you please tell me if Medicare will pay for a power wheelchair for him? Other information to provide to the CSR if asked: * He is enrolled in Medicare, both parts A and B. * He lives in [select one, based on random assignment]: * Philadelphia, Pennsylvania. His zip code is 19105. * Detroit, Michigan. His zip code is 48209. * Pensacola, Florida. His zip code is 32516. * Scottsdale, Arizona. His zip code is 85262. * His doctor has suggested he get a power wheelchair to improve his mobility. * He doesn't have enough strength to use a manual wheelchair. * He lives alone and is not married. * Default answer for other questions: "I don't know." Information from Medicare.gov that GAO used to develop accuracy criteria: Power wheelchairs and/or scooters are covered if they are medically necessary based on Medicare's criteria for coverage. In order for Medicare to cover a power wheelchair/scooter, the beneficiary's doctor must provide a prescription or certificate of medical necessity[Footnote 41] that states that he needs it because of his medical condition. If your father qualifies for coverage, Medicare will pay 80 percent of the Medicare-allowed amount.[Footnote 42],[Footnote 43] [End of section] Appendix VI: Medicare Part B Question and Information Used to Develop Accuracy Criteria: Question 5 posed to CSRs: Should my husband sign up for part B if I am still working and we have health insurance coverage from my employer? Other information to provide to the CSR if asked: * My husband is about to turn 65 next January. * If asked whether working for a large or small employer: I work for the federal government. * I have full medical coverage, including dental and vision. * My husband is fully covered under my insurance plan. * Neither of us is disabled. * The city/zip code information that corresponds with the location of the caller. * Default answer for other questions: "I don't know." Information from Medicare.gov that GAO used to develop accuracy criteria: Your husband might want to wait to sign up for part B, because he would have to pay the monthly part B premium and the benefits may be of limited value as long as the group health plan [spouse's insurance] is the primary payer. You could save on monthly premiums by waiting to sign up. If your husband doesn't sign up for part B when first eligible because he has group health coverage through an employer, he can sign up for Medicare part B during a special enrollment period. This can be anytime he is still covered by the employer's group health plan or during the 8 months following the month when either the coverage or the employment ends--whichever is first. Most people who sign up for Medicare part B benefits during a special enrollment period do not pay higher premiums. [End of section] Appendix VII: Eye Glasses/Exam Question and Information Used to Develop Accuracy Criteria: Question 6 posed to CSRs: My mother is 66 and is enrolled in Medicare. She has been complaining lately that she is having trouble reading the paper and thinks she may need new eyeglasses. Will Medicare pay for an eye exam and a new pair of eyeglasses if her prescription has changed? Other information to provide to the CSR if asked: * The city/zip code information that corresponds with the location of the caller. * She is not married. * She is enrolled in Medicare fee-for-service only. * I do not know the name of the county she lives in. * Default answer for other questions: "I don't know." Information from Medicare.gov that GAO used to develop accuracy criteria: Medicare does not pay for routine eye exams, eyeglasses, or contact lenses. The beneficiary must pay 100 percent of these services. [End of section] Appendix VIII: Comments from the Department of Health & Human Services: DEPARTMENT OF HEALTH & HUMAN SERVICES: Centers for Medicare & Medicaid Services: Administrator: Washington, DC 20201: NOV 30 2004: TO: Leslie G. Aronovitz: Director, Health Care -Program Administration and Integrity Issues: Government Accountability Office: FROM: Mark B. McClellan, M.D., Ph.D.: Administrator: SUBJECT: Government Accountability Office (GAO) Draft Report: MEDICARE: Accuracy of Response from the 1-800-MEDICARE Help Line Should Be Improved, (GAO-05-130): The Centers for Medicare & Medicaid Services (CMS) thanks GAO for it's analysis and recommendations to help improve 1-800-MEDICARE. We concur with the recommendations and have taken steps to implement further improvements. We appreciate the work GAO has done and hope that we will be able to benefit from more of their specific work products in developing our improvement strategies. In addition to the recommendations, we are asking for additional information from GAO (see Attachment A). We hope GAO will provide us with the information we have discussed with GAO regarding the findings on disconnected calls, so we can determine if these are, as we suspect, the result of telephone network problems. While we concur fully with the goals specified by GAO, we also want to stress some basic points related to their work and findings. The 1-800- MEDICARE has experienced astounding growth since it was first launched at the end of 1998. In 2000, the first full calendar year of nationwide operations, the call center received 3.6 million calls asking about Medicare benefits, and health plan information and coverage, handled by approximately 300 customer service representatives (CSRs). In four short years, CMS has expanded the service to be available 24 hours a day, 7 days a week, added 3,000 CSRs, provided additional training on such important topics as choosing a nursing home, long-tern care, and choosing a drug discount card. We were faced with an unprecedented volume of calls about a new part of the Medicare program that required new training efforts and many new CSRs, and we believe we responded as well as we reasonably could given the unique and demanding circumstances. During the month of May 2004 alone, 1-800-MEDICARE received 3.8 million calls as a result of the overwhelming interest in the Prescription Drug Discount Card. The Prescription Drug Discount Card and the subsequent drug benefit, which takes effect January 2006, are historic improvements to the Medicare program. We are continually making improvements to the services provided through 1-800-MEDICARE and were already implementing steps to make enhancements prior to the GAO study. The GAO findings complement the work we are doing to reduce the number of agent transfers and ensure that callers reach the CSR who can help them with their Medicare question. While 1- 800-MEDICARE is open 24 hours a day, 7 days a week, the claims contractors are not, and we have addressed the problem of calls being routed to offices that are closed. In addition, we are currently reassessing various elements of our testing requirements and training protocol. We are concerned about GAO's criteria for designating answers as "inaccurate," particularly because they consider an "incomplete" answer to be inaccurate. We train our CSRs to be attentive to the caller's needs and tailor the response to the caller's specific inquiry, as well as, the caller's interest and understanding level. We have structured our responses to make these key points, and we have provided further details regarding the GAO report below. It is important to provide some additional general background information that we believe is relevant to the study. This supplementary information provides valuable information that will help the readers of the report understand some of the broader circumstances that contributed to the findings. We then respond specifically to each of the four findings in the order they have been presented by the GAO. For ease of review, our response to each finding will address four distinct subject areas: 1. Our concurrence or non-concurrence with the finding; 2. Additional information that we believe is pertinent in understanding why the specific finding was identified; 3. Questions and/or concerns we have with the methodology used in the study including terminology issues, the need for additional information, etc.; and: 4. The steps we are taking to implement the recommendations made including timeframes where appropriate. Background Information: Based on our analysis of past call volumes and planned programmatic impacts to 1-800-MEDICARE, we anticipated and were staffing for an annual call volume of approximately seven million calls in Fiscal Year 2004. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA) and the news surrounding it caused the call volume to increase exponentially. For the month of May 2004 alone, 1-800-MEDICARE received approximately 3.8 million calls. In this single 30-day period, our call volume exceeded 50 percent of the call volume estimated for an entire year and it was not even during our regularly busy Fall ad campaign. We were not prepared to handle the extreme increase in calls received as a result of the legislation. To further illustrate the sheer explosion in call volume we experienced. we were notified in early May by MCI, our telecommunications provider, that they had to suspend delivery of calls because of concerns that the 1-800-MEDICARE calls traffic would crash their entire voice network. To meet the unprecedented high call volume, CMS rapidly worked to increase 1-800- MEDICARE capacity (both staffing and network infrastructure) and improve the level of service provided to callers. In response to this urgent and compelling need, we prepared a scope of work, negotiated a contract, hired, trained, and installed the necessary infrastructure to immediately add 1,000 additional CSRs to the I-800-MEDICARE team. This was all accomplished in approximately 15 business days. In addition, we aggressively worked with the existing 1- 800-MEDICARE contractor (Pearson GS) to recruit, hire, train, and place on the telephones additional CSRs and network routing improvements to ensure we were doing everything possible to meet the needs of our callers. To enable us to have more CSRs on the telephones faster, CMS implemented an expedited training program that focused on the use of the Prescription Drug Assistance Program (PDAP) tool, and responded to the top drug card questions with specific CSR groups. Callers with questions that went 'beyond the scope of these basic topics were transferred to a more experienced CSR for further assistance. This effort immediately raised our CSR workforce by 30 percent to 3,000 CSRs, which is now our baseline staffing level for ongoing operations. The purpose of this background section is to set the context of the call center environment in which GAO was tasked to conduct a study. It is not our intention to justify less than top quality service with circumstances that were outside of our control. It is our goal to help readers understand that we were doing everything possible to react to unprecedented call volume and some of the decisions we were making in the areas of staffing; training, content review, and network enhancements were made while considering many factors, the first of which has always been how we can get the best possible service to our callers under very difficult circumstances. GAO Recommendation: Revise routing procedures and technology to ensure that calls are not transferred or referred to claims administration contractors during non-business hours. CMS Response: We concur with the recommendation. Relevant Information: Enactment of the MMA in December 2003, resulted in the immediate legislative requirement to consolidate each of the 97 individual contractor telephone numbers into a single number (1-800-MEDICARE). It was always our goal as part of the Virtual Call Center Strategy (VCS) to rebuild the beneficiary telephone network to allow for a single entry point that callers would use to contact Medicare. The legislation that required the single 1-800 number was effective upon enactment. fhe initial planning estimates for consolidating to a single 1-800 number indicated that it would take between 12 to 18 months to consolidate the network. To respond to the legislative requirements, we rebuilt the entire network and routing plans so that a single point of entry could be routed to one of 97 different lines of business. We built and implemented the first Medicare speech automated system to help callers more easily access the desired service. We developed an extremely complex solution behind the scenes ensuring that each caller would, after answering a few simple questions, be routed to the contractor best suited to answer the caller's specific question. Due to the compressed timeframes, we were not able to implement all aspects of the original plan. As a result, some calls were routed to contractor locations after their normal hours of operation. In this situation, a message was played indicating that the contractor was closed for the day. Current Approach: We have since implemented additional routing plans that ensure callers are not transferred to a site that is closed, but it is important to note that while the base 1-800-MEDICARE CSRs are available 24 hours a day, 7 days a week, the claims processing contractor's CSRs are only available during normal business hours. With this in mind, it should be understood that while we have amended the routing plans to ensure that callers are sent to a general 1-800-MEDICARE CSR in situations where the claims processing contractor is closed, those CSRs do not have access to the claim detail that exists in the contractor's systems. The result is that while we will be answering calls with a "live" person in all instances, there will be times when we are not able to provide the information a caller needs, depending on the hour of the day. We are expanding the availability of access to claims data that will help us reduce this problem in the future. Concerns: It was never our goal to implement the specific routing solution that was in place during the study as a long-term application. We understand the impact this process has on customer service, and we would never be satisfied with that level of service on a permanent basis. However, we found it necessary to make short-term adjustments to the plans in order to comply with legislative requirements. Given the scope of work associated with consolidating the vast network of phone numbers and associated applications into a single phone number, speech routing application, and the timeframes associated with the legislative requirement, it was not technically possible to develop and implement a solution that would address all of the requirements to complete the work within the required mandate timeframe. Since implementation, we have improved the functionality of the application and the specific accomplishments we have made are outlined below under "Action Plan." More importantly, it was not our intent to implement a system where claims processing contractors would be available 24 hours a day, 7 days a week concurrent, with the implementation of a single I-800 number network. We believe that the calls identified as receiving "No Answers" in the report would more appropriately be referred to as "Calls routed to closed locations." The report indicates that there were some portions of this subset of calls that were "inadvertently disconnected." We find this portion of the report particularly troublesome, and it is one of the reasons why we immediately contacted GAO to request more detailed information on this issue. It is disheartening for CMS to learn that GAO is not willing to immediately release detailed information to CMS that would aid in our troubleshooting in what appears to be a network problem during an ongoing audit. We believe you understand from this and past audits that telephone networks are vastly complex. There are literally thousands of places where a problem with a single line of code or hardware can result in some sort of service degradation. It appears, based on the statements made in the report, that some portions of the calls are being affected by a network issue that we have not yet identified. As we explained to the GAO representative immediately upon receiving the draft report, specific call data from the audit would save extensive CMS staff resources and valuable time locating the problem independent of having your data. More importantly, it would allow CMS to respond more quickly with a solution, thus, reducing the number of callers who will now continue to have some percentage of calls "inadvertently disconnected." It is our sincere request that you provide us with the crucial information upon final completion of your report. Action Plan: We have deployed an Advanced Routing Database in the speech application that reviews the hours of operations for each of the 47 claims call centers that are now connected to 1-800-MEDICARE. The database checks to make sure that the call center is open before calls are routed to one of the claims call centers. If the call center is closed for any reason, the call will be routed to one of the general Medicare CSRs, and the CSR will try to assist the caller with general information or instruct the caller to call back during business hours when the claims system is available. We have developed new information tools for the general Medicare CSRs to use which makes it easy for them to find the correct Medicare claims center for their caller and its correct hours of operation, prior to transferring the call. We have implemented weekly business and technical process meetings with each of the call centers (both general and claims) to improve process and technologies. As a part of these meetings, the need for all CSR refresher training is identified and scheduled. We have built failsafe software into the transfer technology that will not allow the CSR to drop the caller into closed or invalid call center's queues. If the queue cannot accept the call when the CSR transfers it, the network forces the call back to the transferring CSR. This software helps to prevent CSRs from "dropping" calls and keeps the caller from getting lost in the transfer process. GAO Recommendation: Assess the current scripts for the most commonly asked questions to ensure that they are understandable to CSRs and potential callers and if not, revise them as needed and pretest new and revised scripts to ensure that CSRs can effectively use them to accurately answer callers' questions. CMS Response: We concur with the recommendation. Relevant Information: We are always interested in improving the way we develop and implement scripts in the Desktop application. However, it is inaccurate, as the report implies, to say that there is no testing of the scripts used by 1-800-MEDICARE CSRs. We based the majority of our scripts on content that has been consumer tested in the development of Medicare publications. In addition, we hold regular focus groups to identify ways to explain concepts to beneficiaries in our static publications as well as our communication messages. We often incorporate that language into scripts; thus, there has been some degree of beneficiary testing. It is also true that 1-800-MEDICARE is the front-line of our customer service process and, as such, it is the very first place people go when "hot topics" need to be addressed. Scripts are often developed to address urgent or emergent issues, and there are situations where sufficient time to fully pre-test/focus tests does not exist. Concerns: No concerns identified. Action Plan: We will continue our focused feedback sessions with CSRs that center on pre-testing scripts. CSRs participate in content workgroups to develop the MMA scripts and job aids. We continue to hold weekly meetings with the call centers to discuss how to make the scripts more user-friendly, complete, understandable, and comprehensive. Additionally, we have a desktop tool that allows CSRs to make recommended changes to scripts. This feedback is usually the result of caller feedback indicating the script is confusing, not helpful, etc. CSR feedback is compiled weekly and the content is shared with CMS staff. call center management, the learning and development team, as well as other 1-800-MEDICARE functional areas. CSR feedback is reviewed as part of the weekly meetings to determine call trends, the need for additional scripts or script modifications, identify training opportunities, and assist with overall continuous improvement at 1-800- MEDICARE. Based on feedback received, we are considering the implementation of an editorial board that would review words and phases to make recommendations such as changing "trunk strength" to "upper body strength." GAO Recommendation: Enhance testing of CSR's skills in accurately answering the most commonly asked questions using scripts and, if needed, provide additional training to improve the accuracy and completeness of their responses. CMS Response: We concur with the recommendation presented. Relevant Information: Addressed in Concerns section. Concerns: We believe the report reflects an incorrect understanding of the training requirements. The specific contractual requirement is for a CSR to "Achieve Expectations on or above two practice calls in a row." The contractor addresses this by using a series of six simulated calls, and the class trainers sit with the individual CSRs to determine, on a ease-by-case basis, their success at handling the simulated calls. If the CSR does not do well on the skill assessments on the simulated calls, whether they pass or fail, the trainers provide the CSR with additional coaching and mentoring. The intent is to provide the necessary training to ensure that the CSR is ready for real-time operations. In many of the cases where a CSR "fails" at a practice call scenario, it may be due to the CSR being nervous, and the extra tutoring helps to alleviate failure. This is not a situation where the trainers just run through the six simulated calls until the CSR gets two in a row correct, but is intended to provide the CSR with any additional training they need in areas of weakness. In the case of customer service, we disagree with the recommendation that the questions targeted for the skills other than knowledge are not as meaningful. It is very important that the CSRs can handle themselves in all areas of customer service and even if they know how to get to the right script and provide the correct answer, they will not have credibility if they do not sound and act professionally. Action Plan: We are reassessing the testing requirements currently in place to determine ways to better ensure that the CSRs are prepared to handle calls at the point that they are certified. We are adjusting the training protocol to spread testing of the CSR's ability to handle calls over the length of the training period rather than just at the conclusion and to focus on categories of calls, especially on the most frequently asked questions. This approach allows more time for the trainers to coach and mentor the CSRs prior to the next phase of their job experience, which includes side-by-side work with an experienced CSR We will also benchmark against industry standards to determine more effective approaches, ensuring that the CSRs are prepared to respond to calls upon completion of the classroom training. GAO Recommendation: Supplement current monitoring efforts to include a systematic review of the accuracy of information provided by the CSRs as a group for the most frequently asked questions and use the results to modify scripts or provide more training, as needed. CMS Response: We concur with the recommendation to pursue trend analysis. However, we cannot concur with the comment on the observations made in this section, because we do not have access to the data GAO used to arrive at their findings. (Noted below under "Concerns"): Relevant Information: We agree that more analysis could be conducted to better identify global deficiencies across our CSR base. At the same time, we believe that we have done a very good job developing a quality assurance program that focuses on the important requirements for both accuracy and customer service skills needed to answer calls from an elderly population. We require our contractor to monitor each CSR's call performance on a monthly basis using a national quality scorecard. The accuracy rate for the months July through October 2004 is as follows: July 89 percent; August 91 percent; September 90 percent; October 92 percent. Our national quality assurance scorecard measures the entire customer service experience across three broad categories-accuracy, completeness, and call action-and a subset of criteria within each of these categories. For example, accuracy is assessed on the basis of factors such as whether the CSR understood the question or whether they used the correct script(s). In order to have an independent assessment of performance we conduct "mystery shopping" through a specialized quality evaluation contractor. A response is classified as "fully responsive" only if all key points are conveyed to the caller. For example, for the mystery shopping on prescription drugs, the calls scored 85 percent or above as "fully responsive." In addition, we require our contractor to conduct 1,000 beneficiary satisfaction surveys each month to determine customer satisfaction with the services provided. For the month of July, 92 percent were satisfied with the service they received. This is consistent with other months. We also believe that we have done a very good job implementing technology to help support this effort and keep costs under control. We have not gotten to the point of regularly taking a more global review of the data we collect or looking at it from a higher perspective; (i.e., trends). Concerns: In the report, GAO identified 29 percent of the calls made to 1-800- MEDICARE resulted in "inaccurate answers." The methodology used in identifying an answer as inaccurate was "if the CSR provided information on the topic but did not provide sufficient and complete information to meet our criteria, we considered the answer to be inaccurate." Consequently, it is difficult for: IS to understand specifically what the specific criteria for a correct answer. As mentioned in he report, CMS provided GAO with a broad list of questions and various scripts that may have been used in answering the questions. It is important to note, it is not the policy of CMS to require all related scripts to be read to a caller, nor is it CMS policy to require the entire text of any single script to be read. CSRs are instructed to read only those portions of the scripts that pertain to the callers' specific inquiry. Furthermore, there are judgment decisions the CSRs must make with regard to how much information the caller really needs and their level of interest and understanding. In addition, CMS requires the CSRs to remain objective and to not provide opinions to the callers. Based on the information provided in the draft report it is not possible for CMS to identify what portion of the responses deemed "inaccurate" by GAO. We hope to receive call details and specific measurement criteria from the list of calls made to further investigate this issue. Additionally, we question the GAO's use of the word "inaccurate" and "incomplete" answers to classify a grouping of responses. We do not consider an "incomplete" answer to be an "inaccurate" answer. Again, we thank you for the opportunity to review and comment on this report. Action Plan: CMS has deployed trained staff to monitor CSR calls themselves and provide recommendations on changes and improvements for content. We have implemented ongoing analysis of trending reports from all monitoring sources to ensure that issues are identified and changes made as a result of the findings of these groups. We appreciate the feedback provided and we will implement a plan that ensures that results of our Quality Assurance (QA) activities are trended and that we have a process in place that takes the results of that trending to the next step. Each month we will focus efforts on a specific Frequently Asked Questions activity based on QA findings from our other measures. This will enable us to better ensure that the CSRs handling these questions, which make up a large majority of the issues our callers want to discuss, can do so more accurately and effectively. Attachment: ATTACHMENT A: The CMS is requesting additional information from the call sample that was conducted. This information will be extremely helpful to us in our efforts to troubleshoot and correct network problems and it will also support our efforts to improve the consistency and accuracy of the responses we provide to callers: For each of the calls in the Study we are requesting the following information: Date of call Time of call Number that the call was placed from: The phone number of the switch (in the event calls were not made from individual's homes) The name that the tester used on the call: The specific question asked How each call was scored The answer that was considered to be correct The answer that was received: In addition, it would be very helpful if the following information could be provided separately for the subset of calls that were identified as "Inadvertently disconnected" in the report: Date of call Time of call Number that the call was placed from: The phone number of the switch (in the event calls were not made from individual's homes): [End of section] Appendix IX: GAO Contact and Staff Acknowledgments: GAO Contact: Sheila K. Avruch, (202) 512-7277: Acknowledgments: Shaunessye D. Curry, Joy L. Kraybill, Krister P. Friday, Sari B. Shuman, Mary W. Reich, Ramsey L. Asaly, Alexis Chaudron, Perry G. Parsons, and Leslie Spangler made key contributions to this report. FOOTNOTES [1] These companies, known as Medicare claims administration contractors, maintain other toll-free help lines that are open during normal business hours. In June 2004, CMS began a process to discontinue listing these separate help line numbers and have 1-800-MEDICARE serve as the only number that callers would use to access information about Medicare eligibility, enrollment, benefits, coverage, and payment. As of September 30, 2004, this process was complete. [2] Pub. L. No. 108-173, § 923(d)(2), 117 Stat. 2066, 2395. [3] We made 420 calls in order to have a sample that was large enough to determine if differences in accuracy were significant. [4] http://medicare.custhelp.com/cgi-bin/medicare.cfg/php/enduser/ std_alp.php. [5] For our first objective, we focused on the accuracy of information provided by the CSRs, regardless of the contractor that managed their work. For our next two objectives, we relied on information provided by one of the two contractors--which we refer to as the primary contractor--because this contractor provided the training for all new CSRs and managed CSRs who answered 80 percent of the calls on the help line in July 2004. CMS does not use the term "primary contractor," because both companies contract directly with CMS. We are using the term primary contractor to distinguish the one that had a larger role during the time that we were conducting our calls. [6] GAO, IRS's 2002 Tax Filing Season: Returns and Refunds Processed Smoothly; Quality of Assistance Improved, GAO-03-314 (Washington, D.C.: Dec. 20, 2002); GAO, IRS Telephone Assistance: Limited Progress and Missed Opportunities to Analyze Performance in the 2001 Filing Season, GAO-02-212 (Washington, D.C.: Dec. 7, 2001). [7] In May 2004, 1-800-MEDICARE received over 3 and a half million calls, which was more calls received than in any prior month. Throughout the summer of 2004, 1-800-MEDICARE received between 1 million and 2 million calls each month. [8] There are over 670 scripts for 1-800-MEDICARE in use. [9] www.Medicare.gov. [10] For some topics, such as the prescription drug discount card, there are several scripts that deal with the topic, but generally one script provides the best information to accurately answer a specific question, according to CMS. For each of our proposed questions, CMS provided the name and number of the script that it considered as having the best information for CSRs to use in formulating a response. [11] This tool is also available on Medicare's Web site, at http:// www.medicare.gov/AssistancePrograms/home.asp. [12] AARP is an association for individuals 50 years of age and older. It provides information, such as tips for healthy living and retirement planning; advocacy on issues affecting the elderly; and services, such as supplemental health insurance. [13] In its comments on a draft of this report, CMS informed us that the written information used to develop 1-800-MEDICARE scripts often comes from Medicare publications that have been consumer tested as part of the publication preparation process. Language that has undergone some consumer testing is often incorporated into the scripts to improve clarity. [14] CMS distributes a handbook called Medicare & You to all Medicare beneficiary households. The Medicare & You handbook describes the Medicare program and its benefits. [15] The Centers for Disease Control and Prevention is an agency of the U.S. Department of Health and Human Services that provides federal leadership to develop and apply disease prevention and control, environmental health, and health promotion and education activities designed to improve the health of people of the United States. [16] The Substance Abuse and Mental Health Services Administration is an agency of the U.S. Department of Health and Human Services established to focus attention, programs, and funding on improving the lives of people with, or at risk for, mental and substance abuse disorders. [17] U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Scientific and Technical Information: Simply Put (Atlanta, Ga.: 1999) (at http://www.cdc.gov/publications.htm, downloaded Sept. 21, 2004), and U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Agency, Technical Assistance Bulletin: Pretesting is Essential; You Can Choose From Various Methods. (1994, http://www.health.org/_usercontrols/ printpage.aspx?from downloaded Sept. 21, 2004). [18] Each of our six questions was posed during evening and weekend hours, as well as during normal business hours, using the same random pattern of calling during certain hours of each day over the course of 3 weeks. However, because questions 4 and 6 were often transferred or referred more often to other contractors' help lines, these were the questions that were not answered when the contractors were closed for business. In contrast, the 1-800-MEDICARE help line did not routinely transfer questions 1, 2, 3, and 5 to other contractors. [19] For 14 transferred calls, our callers were provided with a telephone number and were able to complete the calls later. For 10 of these 14 calls completed later (71 percent), claims administration contractors' CSRs answered our questions accurately. [20] There can be more than one script on a particular topic, but often one of the scripts is the best one to answer a specific question about that topic. [21] While the primary contractor provided all training for newly hired CSRs, both the primary and secondary contractors provided continuing training for their CSRs. [22] The test measures several types of information and skills that CSRs need. It consists of 52 questions: 14 questions on choosing a script that can be used to answer a question; 10 matching questions to choose definitions of Medicare terms; 10 true or false questions where CSRs were instructed to use a script to answer a question about the Medicare program; 6 multiple choice questions on call handling etiquette; and 12 matching questions to identify the organizations to which CSRs could refer certain questions--for example, the Social Security Administration for questions about enrolling in Medicare. The guide for scoring each question on scripts indicates several scripts that could be named as possible responses, including the script that would best address the question posed. If the CSR chooses any of the scripts that are listed as possible responses, the answer is scored correctly, even if the CSR does not choose the script with the best information to answer the question. [23] The call handling exam for new CSRs consists of up to six questions. In order to successfully pass, the new CSR must adequately handle two of these calls consecutively. [24] GAO-02-212. [25] GAO-03-314. [26] The primary contractor also surveys callers to assess their satisfaction with the information they received from 1-800-MEDICARE. These customer satisfaction survey results are also reported to CMS in standard reports. [27] Incoming Calls Management Institute, Call Center Monitoring Study II Final Report (Annapolis, Md.: 2002). [28] This monitoring occurs in addition to the four calls per CSR per month that are monitored by the primary contractor, and the calls monitored by the subcontractor. [29] In addition, the subcontractor monitored about 1,400 calls in July 2004. [30] The National Medicare Education Program publishes the Medicare & You handbook and maintains other activities to communicate with Medicare beneficiaries. The 1-800-MEDICARE help line is one of the activities in the program. [31] For example, for one question related to Medicare's coverage of a piece of durable medical equipment, the study defined a fully responsive answer as one that informed the caller that a prescription would be needed for the item, specified that the beneficiary needed a supplier who accepts Medicare, provided an overview of the Medicare billing process, shared information on how to contact the contractors that process durable medical equipment claims, and indicated that supplemental insurance might pay part of the costs. [32] If CSRs provided partially correct but incomplete information that did not fully address our criteria, we scored the calls as "inaccurate." [33] This research was part of a CMS effort to determine whether CSRs were prepared to handle questions about program changes resulting from the MMA. [34] The April 2004 test calls used a scorecard that included up to nine different elements that measured accuracy, depending on the question the caller used. For example, the call could be scored on whether the CSR accurately informed the caller which Medicare beneficiaries would be eligible for a prescription drug discount card, how a beneficiary could apply for a prescription drug discount card, and when a beneficiary could start to use their prescription drug discount card. [35] For question 1, the name of either one of the two prescription drug plans with the lowest cost for the drugs was the correct answer. For questions 2 and 3, the correct answer was "yes," and for question 6, the correct answer was "no." [36] Except for a script used when transferring calls, each of the top 15 scripts accessed in May 2004 by 1-800-MEDICARE CSRs was designed to answer various questions about the prescription drug discount card program. [37] http://medicare.custhelp.com/cgi-bin/medicare.cfg/php/enduser/ std_alp.php. [38] If a CSR mentioned any prescription drug discount card that cost $182.80 per month, we considered the response to be accurate. [39] If a CSR mentioned any prescription drug discount card that cost $50.45 per month, we considered the response to be accurate. [40] TRICARE for Life provides secondary military health coverage available for Medicare-eligible uniformed services beneficiaries, their eligible family members, and survivors enrolled in Medicare part B. [41] A certificate of medical necessity is a standardized form that physicians complete to indicate that an item is needed for medical reasons. [42] The Medicare.gov Web site notes that Medicare will pay 80 percent after the $100.00 part B deductible is met, but we did not require CSRs to mention the deductible for our accuracy criteria. [43] According to the HHS Office of the Inspector General, the median Medicare reimbursement for a purchased power wheelchair was $5,297 in 2003. GAO's Mission: The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: