The Honorable E. Clay Shaw, Jr.
Chairman, Subcommittee on Social Security
Committee on Ways and Means
House of Representatives
Washington, D.C. 20515

Dear Mr. Shaw:

Thank you for your interest in the Social Security Administration's disability programs. In a January 27, 2004 letter, you requested that we review allegations made in a letter from a former medical consultant that the Chicago Regional Office may have manipulated its calculation of the Disability Determination Services net accuracy rate by discouraging medical reviewers from finding errors. You also requested an evaluation of the degree to which the Disability Determination Services net accuracy rate performance indicator is being manipulated or has the potential to be manipulated by employees of the Social Security Administration. The results of our review are presented in the enclosed report.

My office is committed to eliminating fraud, waste, and abuse in the Social Security Administration's operations and programs. If you have any questions concerning this matter, please call me or have your staff contact H. Douglas Cunningham, Assistant Inspector General for Congressional and Intra-Governmental Liaison, at (202) 358 6319.

Sincerely,

Patrick P. O'Carroll, Jr.
Acting Inspector General

CONGRESSIONAL RESPONSE REPORT

Social Security Administration
Chicago Regional Office's
Disability Determination Services
Net Accuracy Rate

A-07-04-24094

July 2004

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

Background
OBJECTIVES

The objectives of our review were to determine whether (1) evidence exists to support allegations that the Social Security Administration (SSA) Chicago Regional Office (RO) manipulated its Disability Determination Services (DDS) net accuracy rate by discouraging reviewers from finding errors and (2) the DDS net accuracy rate calculation process is vulnerable to manipulation.

BACKGROUND

SSA assures the quality of initial State disability determinations by reviewing a random sample of allowances and denials from each State. A random, stratified sample of initial allowances and denials is selected from each State by an automated sample selection process at the time of DDS input to the National DDS System. The basic design is to obtain a sample of 70 allowances and 70 denials per quarter per State. Sample cases are sent to the Office of Quality Assurance and Performance Assessment Disability Quality Branch (DQB) in each region for quality assurance review. The review process replicates the DDS' disability determination process to the extent possible. The review team consists of a DQB examiner and one or more medical consultants. Physicians performing medical reviews of disability determinations must always evaluate and rate the sufficiency of medical evidence in the file; current impairment severity and duration; and when applicable, current residual functional capacity, onset, ending and cessation dates; and diary actions.

The quality assurance review process results in the calculation of DDS net accuracy, which is defined as the percentage of correct initial disability determinations. The net accuracy rate is based on the net error rate, which is defined as the number of corrected deficient cases with changed disability decisions, plus the number of deficient cases that are not corrected within 90 days from the end of the calendar quarter, divided by the number of cases reviewed. See Appendix C for an overview of the quality assurance review process for initial disability determinations.

In a letter dated January 27, 2004, Congressman E. Clay Shaw, Jr., asked the Office of Inspector General to review the allegations made in a letter from a former medical consultant that the Chicago RO may have manipulated its calculation of the net accuracy rate by discouraging medical reviewers from finding errors in DDS disability determinations. Congressman Shaw requested an evaluation of the degree to which the DDS net accuracy rate performance indicator is being manipulated or has the potential to be manipulated by employees at SSA. See Appendix B for the scope and methodology of our review.

Results of Review
Our review did not identify evidence to support allegations that the Chicago RO manipulated its DDS net accuracy rate. Furthermore, the DDS net accuracy rate calculation does not appear to be vulnerable to manipulation because of existing management controls that are in place over the net accuracy rate calculation.

MEDICAL CONSULTANT ALLEGATIONS

In a letter to Congressman E. Clay Shaw, Jr., a medical consultant formerly under contract with the Chicago RO made five specific allegations regarding activities at the Chicago RO that could result in an inaccurate calculation of the DDS net accuracy rate. To determine if the allegations were valid, we conducted interviews with Chicago RO staff and medical consultants currently under contract with the Chicago RO. We also reviewed the policies and procedures in place at the Chicago RO for the review process resulting in the net accuracy rate calculation. The following presents the former medical consultant's allegations and the results of our review of each allegation.

ALLEGATION NUMBER 1: Medical consultants who find few or no medical errors in DDS decisions are awarded more contract hours than those who find many medical errors.

According to staff at the Chicago RO's Center for Disability, the number of contract hours awarded to a medical consultant is based on the amount of work available, the medical consultant's schedule, and the medical consultant's prior work performance in the Region. The Center for Disability staff also stated that the number of errors a medical consultant identifies in his or her reviews of disability determinations does not impact how contract hours are awarded and data on the number of errors medical consultants find in their reviews are not maintained by the RO. Since data on the number of errors medical consultants identify in their reviews does not exist, we could not determine if there was a basis for this allegation. However, during our review no evidence came to our attention that would indicate that contract hours are awarded to a medical consultant based on the number of errors identified in previous reviews of disability determinations.

ALLEGATION NUMBER 2: To avoid negative consequences for finding medical errors in DDS decisions, medical consultants select only easy and fast cases to review.

The Center for Disability staff admitted that in the past, there were some instances of medical consultants who attempted to identify and select cases that were less difficult for review. We were unable to determine why certain medical consultants attempted to select only easy and fast cases for review. However, as of Fiscal Year (FY) 2004, medical consultants do not have a choice in what case they will review, because the Disability Case Adjudication and Review System (DICARS) has a list of cases from which medical consultants must select the oldest case to review. Since the selection is electronic, the ability to select only easy or fast cases is eliminated.

ALLEGATION NUMBER 3: Medical consultants are instructed not to use certain terms or phrases in their narratives on cases.

Medical consultants are instructed to use language that is supported by medical evidence in the case file. Accordingly, the Center for Disability staff will instruct a medical consultant to remove language from a narrative when it reflects a personal opinion that is not supported by medical evidence. For example, medical consultants should not use subjective terms, such as "malingering" or faking symptoms unless they are quoting from a medical report in which the examining physician uses such a term.

ALLEGATION NUMBER 4: The medical consultant was instructed by an administrator of the Regional Office not to find medical errors.

We interviewed five medical consultants currently under contract with the Chicago RO to determine if there was a basis for this allegation. All five medical consultants stated that they had never been given instructions not to find errors.

ALLEGATION NUMBER 5: Occasionally, SSA management insists that physicians come up with a certain medical conclusion, even when the medical evidence does not support this conclusion.

We asked the five medical consultants we interviewed if they were asked to arrive at a medical conclusion that was not supported by medical evidence. All five medical consultants stated that they had never been asked to make unsupported medical conclusions. Furthermore, random samples of medical consultants' cases are selected for quality review by the Center for Disability and the review is performed by an employee that had no prior involvement in the case. With the absence of collusion, this separation of duties in the review process would identify unsupported medical conclusions. According to staff at the Center for Disability, quality reviewers and medical consultants might disagree on a medical conclusion; however, they either discuss the disagreement until it is resolved or refer the case to a lead medical consultant for resolution.

We also compared the Chicago Region's net accuracy rate to the rates for other regions for FYs 2001 through 2003. We found that the net accuracy rates for the Chicago Region did not differ significantly from the rates of other regions. In addition, the rank of the Chicago Region's net accuracy rates as compared to other regions varied during the period under review, which would discount any sustained net accuracy rate manipulation. These comparisons, along with the above facts, make it unlikely that the Chicago Region manipulated data to improve its net accuracy rate. See Appendix D for our comparison of national and regional net accuracy rates.

VULNERABILITY OF NET ACCURACY RATE TO MANIPULATION

We found that it would be difficult to manipulate the data used to calculate net accuracy because of the controls in place for the review process resulting in the net accuracy calculation. Given these controls, it would require a substantial amount of collusion among SSA employees and/or medical consultants for data to be manipulated. We found no evidence of collusion during our review. The following is a summary of the controls in place over the net accuracy rate calculation.

Cases included in the net accuracy rate calculation are reviewed by at least two staff including a DQB examiner and a medical consultant.

Cases are randomly selected for a quality review by the Center for Disability and the review is performed by a policy specialist with no prior involvement with the cases.

All cases with disability determination errors are reviewed by a DQB team leader before being returned to the DDS for correction.

DQB team leaders review a random sample of cases where no errors are identified before the cases are released for effectuation of the determination.

If disagreements about the disability determinations in cases occur throughout the review process, other medical consultants, DQB team leaders, or DQB management review the cases.

In addition to the work performed for this review, the Office of the Inspector General's Performance Indicator Audit: DDS Net Accuracy Rate-Allowances and Denials Combined (A-15-04-14074) is currently in process. This audit will address critical controls over the data generation and calculation processes for the net accuracy rate, including tests of DICARS.

Conclusion

We found no evidence to support allegations that the SSA Chicago RO manipulated its DDS net accuracy rate by discouraging reviewers from finding errors. Furthermore, the DDS net accuracy rate calculation does not appear to be vulnerable to manipulation because of existing management controls that are in place over the net accuracy rate calculation.

Appendices
Appendix A - Acronyms
Appendix B - Scope and Methodology
Appendix C - Quality Assurance Review Process for Initial Disability Determinations
Appendix D - Net Accuracy Rates - Fiscal Years 2001-2003

Appendix A
Acronyms
DDS Disability Determination Services
DICARS Disability Case Adjudication and Review System
DQB Disability Quality Branch
FY Fiscal Year
MC Medical Consultant
NDDSS National Disability Determination Services System
OIG Office of the Inspector General
OQA Office of Quality Assurance and Performance Assessment
POMS Program Operations Manual System
RO Regional Office
SSA Social Security Administration

Appendix B
Scope and Methodology
We collected and analyzed information specific to the concerns raised by Congressman E. Clay Shaw, Jr., regarding the determination of the net accuracy rate in the Chicago Region. To achieve our objectives, we:

Reviewed the previously issued Office of the Inspector General (OIG) report Performance Measure Review: Reliability of the Data Used to Measure Disability Determination Services Decisional Accuracy (A-07-99-21007) and work performed for the ongoing Performance Indicator Audit: DDS Net Accuracy Rate-Allowances and Denials Combined (A-15-04-14074).

Reviewed Program Operations Manual System (POMS) GN 04440.119, 04441.110, and 04441.806 for policies and procedures for medical consultants and POMS DI 30005.001 and 30005.005 for policies on quality reviews.

Interviewed staff at the Chicago Regional Center for Disability, Disability Quality Branch, and Contracting Office and medical consultants currently under contract to gain an understanding of how disability determination reviews are performed, medical consultants' responsibilities, and how the net accuracy rate is calculated.

Interviewed a Chicago Regional Office of General Council attorney to learn the status of current protests by the medical consultant for termination and non-renewal of contract.

The Social Security Administration operating components reviewed were the Chicago Regional Office of Quality Assurance and Performance Assessment and the Chicago Center for Disability. We performed our review in the Chicago Regional Office in Chicago, Illinois, and the OIG Office of Audit in Kansas City, Missouri, from March through June 2004. We conducted our review in accordance with Quality Standards for Inspections issued by the President's Council on Integrity and Efficiency.

Appendix C
Quality Assurance Review Process for Initial Disability Determinations

Appendix D
Region FY 2001 FY 2002 FY 2003
Rate Rank Rate Rank Rate Rank1
National Average 96.2% 96.4% 96.1%
Boston 96.8 3 96.1 6 96.7 2
New York 95.6 9 95.6 10 95.6 8
Philadelphia 96.5 5 97.4 1 96.1 5
Atlanta 96.4 6 96.1 6 96.1 5
Chicago 95.7 8 97.2 2 96.6 3
Dallas 96.6 4 96.4 5 96.5 4
Kansas City 97.7 1 97.1 3 95.9 7
Denver 96.9 2 96.6 4 97.7 1
San Francisco 95.3 10 95.7 9 94.8 10
Seattle 96.2 7 96.0 8 95.5 9
Net Accuracy Rates - Fiscal Years (FY) 2001-2003

Overview of the Office of the Inspector General

The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Executive Operations (OEO). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit

OA conducts and/or supervises financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations

OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General

OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Executive Operations

OEO supports OIG by providing information resource management and systems security. OEO also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, OEO is the focal point for OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.