This is the accessible text file for GAO report number GAO-04-14 
entitled 'SSA Disability Decision Making: Additional Steps Needed to 
Ensure Accuracy and Fairness of Decisions at the Hearing Level' which 
was released on November 12, 2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters:

United States General Accounting Office:

GAO:

November 2003:

SSA Disability Decision Making:

Additional Steps Needed to Ensure Accuracy and Fairness of Decisions at 
the Hearings Level:

GAO-04-14:

GAO Highlights:

Highlights of GAO-04-14, a report to congressional requesters

Why GAO Did This Study:

Historically, the proportion of the Social Security Administration’s 
(SSA) disability benefits claims that were approved has been lower for 
African-Americans than for whites. In 1992, GAO found that racial 
differences, largely at the Administrative Law Judge (ALJ) level, 
could not be completely explained by factors related to the decision-
making process. This report examines how race and other factors 
influence ALJ decisions and assesses SSA’s ability to ensure the 
accuracy and fairness of ALJ decisions.

What GAO Found:

GAO controlled for factors that are related to the disability decision-
making process at the Administrative Law Judge level and found: 

* no statistically significant difference in the likelihood of being 
allowed benefits between white claimants and claimants from other, non-
African-American racial/ethnic groups; and between white claimants and 
African-American claimants who were represented by attorneys;

* statistically significant differences between white and African-
American claimants who were not represented by attorneys. 
Specifically, among claimants without attorneys, African-American 
claimants were significantly less likely to be awarded benefits than 
white claimants; and 

* other factors—including sex, income, and the presence of a 
translator at a hearing—also had a statistically significant influence 
on the likelihood of benefits being allowed.

Due to the inherent limitations of statistical analysis, one cannot 
determine whether these differences by race, sex, and other factors 
are a result of discrimination, other forms of bias, or variations in 
currently unobservable claimant characteristics.

Analytical, sampling, and data weaknesses in SSA’s approach to quality 
assurance reviews limit its ability to ensure the accuracy and 
fairness of ALJ decisions. For example:

* Analytic weaknesses: SSA analyzes ALJ decisions by various factors, 
such as SSA region, but not by the claimant’s race.

* Sampling weaknesses: SSA currently excludes cases that have been 
appealed to the Appeals Council from the pool of ALJ cases that 
undergoes the quality assurance review. The exclusion of these cases 
could mean that the sample used by SSA in its quality assurance review 
is not representative of all ALJ decisions. While GAO did not find 
large differences in the sample of cases from 1997 to 2000 that it 
used for its analysis, the continued, systematic exclusion of cases 
that are under appeal could in the future result in an 
unrepresentative sample of all ALJ decisions.

* Data limitations: even if SSA wanted to conduct analyses by race/
ethnicity, it would encounter difficulties doing so in the near future 
because, since 1990, SSA significantly scaled back its collection of 
race/ethnicity data. Although GAO had sufficient race data for its 
study, the scaled back collection of race/ethnicity data will impact 
SSA’s future efforts to study ALJ benefit decisions by race. During 
GAO’s review, however, SSA decided to collect race/ethnicity data for 
persons applying for Social Security benefits.

What GAO Recommends:

GAO recommends that SSA enhance its ALJ quality assurance reviews by 
* incorporating cases that are appealed to SSA's Appeals Council in 
the quality assurance review sample, 

* conducting ongoing as well as in-depth analyses of ALJ decisions by 
race and other factors, and

* publishing these results in its biennial reports.

Further, GAO recommends that SSA 

* take action, as needed, to correct and prevent unwarranted allowance 
differences; and 

* establish an expert advisory panel to provide ongoing leadership, 
oversight, and technical assistance with respect to ALJ quality 
assurance reviews.

SSA agreed with GAO’s recommendations. 

www.gao.gov/cgi-bin/getrpt?GAO-04-14.

To view the full product, including the scope and methodology, click 
on the link above. For more information, contact Robert E. Robertson 
at (202) 512-7215 or RobertsonR@gao.gov.

[End of section]

Contents:

Letter:

Results in Brief:

Background:

Race and Other Factors Influence ALJ Decisions for Some Claimant 
Groups:

SSA's Approach to Quality Assurance Reviews Limits Its Ability to 
Ensure the Accuracy and Fairness of ALJ Decisions:

Conclusions:

Recommendations:

Agency Comments:

Appendix I: Scope and Methods:

Section 1: Databases and Information Sources:

Section 2: Data Reliability Tests:

Section 3: Weighting and Sampling Errors:

Section 4: Statistical Analysis:

Section 5: Limitations of Analysis:

Appendix II: SSA's Five-Step Sequential Evaluation Process for 
Determining Disability:

Appendix III: Comments from the Social Security Administration:

Appendix IV: GAO Contacts and Acknowledgments:

GAO Contacts:

GAO Acknowledgments:

Other Acknowledgments:

Tables:

Table 1: Variables Used in Our Model of ALJ Decision Making:

Table 2: Percentage of Claimants Allowed Benefits at the Hearings Level 
by Race and Region, 1997 to 2000:

Table 3: Data Used in Our Analyses:

Table 4: Statistically Significant Differences between Responder and 
Nonresponder Groups, as Estimated with Logistic Regression:

Table 5: Tabulations of Statistically Significant Administrative 
Factors (from Table 4) for Responders and Nonresponders:

Table 6: Results of Baseline and Final Models of ALJ Allowance 
Decisions:

Table 7: Observed and Estimated Odds Ratios by Attorney Representation 
and Race:

Table 8: Computations for Odds Ratios for Different Racial Groups That 
Are Represented by an Attorney:

Table 9: Computations for Odds Ratios for Claimants of the Same Race 
with and without Attorney Representation:

Table 10: Effect of Attorney Representation on ALJ Decisions for 
Responders and Nonresponders:

Table 11: Effect of Attorney Representation on ALJ Decisions for 
Responders and the Entire Sample:

Table 12: Effect of Attorney Representation on ALJ Decisions for 
Responders and Nonresponders, by Race:

Table 13: Effect of Attorney Representation on ALJ Decisions for 
Responders and the Entire Sample by Race:

Table 14: Summary Results of Oaxaca Decomposition:

Abbreviations:

ACAPS: Appeals Council Automated Processing System:

ALJ: Administrative Law Judge:

CCS: Office of Hearings and Appeals Case Control System:

DDHQ: Division of Disability Hearings Quality:

DDS: Disability Determination Service:

DI: Disability Insurance:

EAB: Enumeration at Birth:

HALLEX: Hearings, Appeals and Litigation Law Manual:

MEF: Master Earnings File:

NOSSCR: National Organization of Social Security Claimant 
Representatives:

OHA: Office of Hearings and Appeals:

OQA: Office of Quality Assurance and Performance Assessment:

SGA: substantial gainful activity:

SSA: Social Security Administration:

SSI: Supplemental Security Income:

United States General Accounting Office:

Washington, DC 20548:

November 12, 2003:

The Honorable Charles B. Rangel: 
Ranking Minority Member: 
Committee on Ways and Means: 
House of Representatives:

The Honorable Robert T. Matsui: 
Ranking Minority Member: 
Subcommittee on Social Security: 
Committee on Ways and Means: 
House of Representatives:

The Honorable Gene Green: 
House of Representatives:

Historically, under the Social Security Administration's (SSA) 
Disability Insurance (DI) and Supplemental Security Income (SSI) 
programs, the proportion of benefit claims that were approved for 
African-Americans has been lower than the proportion that were approved 
for whites.[Footnote 1] In 1992, GAO conducted a statistical analysis 
of disability benefit decisions and found that racial differences, 
largely at the Administrative Law Judge (ALJ) level, could not be 
completely explained by factors related to the decision-making process, 
such as certain demographic characteristics of claimants (including 
age, education, and sex) and their impairment types. In 2001, you asked 
us to examine the steps SSA had taken to correct and prevent 
unwarranted racial differences. You also asked us to examine whether 
unwarranted racial differences currently exist within these programs.

This report is the second of two reports in response to your request. 
In the first report, published in September 2002, we assessed steps SSA 
took to investigate and correct potential unwarranted differences, 
including SSA's study of racial differences in ALJ decisions.[Footnote 
2] For its study, SSA used new data--which we will refer to as enhanced 
data--developed as part of its recently established and ongoing quality 
assurance review of ALJ decisions. The enhanced data contain 
information, previously unavailable to GAO, such as an improved measure 
of severity of the claimant's impairment. In our 2002 report, we stated 
that we were unable to draw firm conclusions about racial differences 
from SSA's study because of weaknesses we identified in SSA's sampling 
and statistical methods. As a result, we recommended that SSA assess 
the degree to which its enhanced data are representative of ALJ 
disability decisions and make any needed changes to its sampling 
protocol and statistical methods, as part of its ongoing quality 
assurance review of ALJ decisions.

This report examines (1) how race and other factors influence ALJ 
decisions and (2) limitations in SSA's ability to ensure the accuracy 
and fairness of ALJ decisions. You asked us to examine racial 
differences in DI and SSI decisions at the ALJ level, including 
Hispanics and other ethnic groups. However, due to limitations with 
SSA's race/ethnicity data, our examination was limited to African-
American claimants, white claimants, and claimants from other racial/
ethnic groups.[Footnote 3]

Given our previously reported concerns about the degree to which the 
enhanced data are representative,[Footnote 4] we conducted tests at the 
beginning of this review to determine whether the enhanced data were 
sufficiently representative and reliable for our analyses.[Footnote 5] 
Because these tests established that the enhanced data were of 
sufficient quality for our analysis, we were able to analyze these data 
to determine whether racial differences currently exist in ALJ benefit 
decisions and whether differences in ALJ decisions are explained by 
factors related to the decision-making process. To do this, we analyzed 
SSA's enhanced data from 1997-2000 using statistical models of ALJ 
decision making that we constructed. Specifically, we used multivariate 
analysis to determine whether any differences by race/ethnicity could 
be statistically attributed to factors related to ALJ decision 
making.[Footnote 6] As shown in table 1, the variables we included in 
our model can be grouped into three broad sets of factors that are 
related to the decision-making process: (1) factors that represent the 
criteria used in the disability decision-making process; (2) factors 
that represent participants in the decision-making process; and (3) 
factors that are not part of the decision-making process, but may 
influence it.[Footnote 7] See appendix I for more information on our 
statistical methods.

Table 1: Variables Used in Our Model of ALJ Decision Making:

Factors representing criteria in the decision-making process: Medical 
variables: 

Factors representing criteria in the decision-making process: 
Impairments.

Factors representing criteria in the decision-making process: Severity 
of impairment.

Factors representing criteria in the decision-making process: Alcohol 
or drug abuse.

Factors representing criteria in the decision-making process: 
Consultative examination requested.

Factors representing criteria in the decision-making process: Number of 
impairments.

Factors representing criteria in the decision-making process: Number of 
severe impairments.

Factors representing criteria in the decision-making process: Residual 
functional capacity of claimant.

Factors representing criteria in the decision-making process: Mental 
residual functional capacity of claimant.

Factors representing criteria in the decision-making process: 
Nonmedical variables: 

Factors representing criteria in the decision-making process: 
Occupational type.

Factors representing criteria in the decision-making process: Years of 
employment.

Factors representing criteria in the decision-making process: 
Occupational skill level.

Factors representing criteria in the decision-making process: 
Education.

Factors representing criteria in the decision-making process: Literacy.

Factors representing criteria in the decision-making process: Age 
category.

Factors representing participants in the decision-making process: 

Factors representing participants in the decision-making process: 
Representation (by attorney or other).

Factors representing participants in the decision-making process: 
Medical expert present at hearing.

Factors representing participants in the decision-making process: 
Vocational expert present at hearing.

Factors representing participants in the decision-making process: 
Translator present at hearing.

Factors representing participants in the decision-making process: 
Claimant present at hearing.

Factors not part of the decision-making process, but may influence it: 

Factors not part of the decision-making process, but may influence it: 
Race.

Factors not part of the decision-making process, but may influence it: 
Sex.

Factors not part of the decision-making process, but may influence it: 
Earnings.

Factors not part of the decision-making process, but may influence it: 
Type of 
claim.

Factors not part of the decision-making process, but may influence it: 
Year of decision.

Factors not part of the decision-making process, but may influence it: 
Region.

Source: GAO analysis of SSA's enhanced data.

[End of table]

To obtain information on factors limiting SSA's ability to ensure the 
accuracy and fairness of ALJ decisions, we interviewed SSA officials 
and reviewed documentation concerning the agency's ongoing quality 
assurance review of ALJ decisions. We also interviewed officials within 
the Department of Health and Human Services' Centers for Medicare and 
Medicaid Services to discuss their use of SSA race data.

We performed our work from August 2002 to September 2003 in accordance 
with generally accepted government auditing standards.

Results in Brief:

When we controlled for factors that are related to the disability 
decision-making process at the hearings level, including the severity 
of the claimant's impairment, whether or not the claimant had attorney 
representation, and the claimant's age and work experience, we found no 
statistically significant differences in the likelihood of being 
allowed benefits between whites and claimants from other, non-African-
American racial/ethnic groups. We did, however, find differences 
between white and African-American claimants, but only among claimants 
who were not represented by attorneys. That is, among claimants who 
were represented by attorneys, white and African-American claimants 
were equally likely to be allowed benefits, but among claimants who 
were not represented by attorneys, African-American claimants were 
significantly less likely to be awarded benefits than white claimants. 
Moreover, claimants who were represented by persons other than 
attorneys, such as legal aides, friends or family, were more likely to 
be awarded benefits than claimants who are not represented; however, 
among claimants represented by these nonattorneys, African-Americans 
were less likely to be awarded benefits than whites. Besides race and 
attorney representation, other factors that are not part of the 
criteria used in the decision-making process also had a statistically 
significant influence on the likelihood of benefits being allowed. For 
example, male claimants, claimants with low incomes, or non-English-
speaking claimants who had a translator at a hearing were less likely 
to be awarded benefits. Due to the inherent limitations of statistical 
analysis, one cannot determine whether these differences by race, sex, 
and other factors are a result of discrimination or other forms of 
bias, or due to variations in currently unobservable claimant 
characteristics, such as a lack of detailed information on medical 
evidence needed to buttress impairment claims.

Analytical, sampling, and data weaknesses in SSA's approach to quality 
assurance reviews limit its ability to ensure the accuracy and fairness 
of ALJ decisions. As part of its ongoing quality assurance review, SSA 
analyzes ALJ decisions by various claimant characteristics such as the 
claimant's age and the region where the disability decision was issued, 
but not by the claimant's race. This analytic omission limits SSA's 
ability to identify, correct, and prevent unwarranted racial 
differences in allowance rates. In addition, weaknesses in the review's 
sampling methods present problems. For example, SSA currently excludes 
cases that have been appealed to the Appeals Council from the pool of 
ALJ cases that undergoes the quality assurance review. The exclusion of 
these cases could mean that the sample used by SSA in its quality 
assurance review is not representative of all ALJ decisions. While we 
found the sample of cases that we used for our analysis to be 
sufficiently representative, the continued, systematic exclusion of 
appealed cases could, in the future, result in an unrepresentative 
sample of all ALJ decisions. Finally, data limitations restrict SSA's 
ability to ensure the accuracy and fairness of ALJ decisions. For 
example, even if SSA wanted to conduct analyses by race/ethnicity, it 
would encounter difficulties doing so in the near future because, since 
1990, SSA has significantly scaled back its collection of race/
ethnicity data. Although we had sufficient race data for our study, the 
scaled back collection of race/ethnicity data will impact SSA's future 
efforts to study ALJ benefit decisions by race. During our review, 
however, SSA decided to collect race/ethnicity data for disability 
claimants and other individuals applying for Social Security benefits 
and has set up a task group to explore implementation issues. In 
addition, SSA officials recently informed us that they are considering 
ways to include appealed cases in their quality assurance review.

To better ensure the accuracy and fairness of ALJ decisions by race/
ethnicity and other factors not related to criteria used in the 
decision-making process, we recommend that SSA enhance its ALJ quality 
assurance reviews by: incorporating cases that are appealed to SSA's 
Appeals Council in the quality assurance review sample; conducting 
ongoing as well as in-depth analyses of ALJ decisions by race and other 
factors; and publishing these results in its biennial reports. We also 
recommend that SSA take action, as needed, to correct and prevent 
unwarranted allowance differences, and establish an expert advisory 
panel to provide ongoing leadership, oversight, and technical 
assistance with respect to ALJ quality assurance reviews.

In its written comments to our report, SSA agreed with our 
recommendations and indicated that it intends to go further as it moves 
forward with its recently proposed plan to improve the disability 
determination process. SSA's comments and its proposed plan to improve 
the disability determination process are printed in appendix III.

Background:

DI and SSI are the two largest federal programs providing cash 
assistance to people with disabilities. Established in 1956, DI 
provides monthly payments to workers with disabilities (and their 
dependents or survivors) under the age of 65 who have enough work 
experience to qualify for disability benefits. Created in 1972, SSI is 
a means-tested income assistance program that provides monthly payments 
to adults or children who are blind or who have other disabilities and 
whose income and assets fall below a certain level.[Footnote 8] To be 
considered eligible for either program as an adult, a person must be 
unable to perform any substantial gainful activity by reason of a 
medically determinable physical or mental impairment that is expected 
to result in death or that has lasted or can be expected to last for a 
continuous period of at least 12 months. Work activity is generally 
considered substantial and gainful if the person's earnings exceed a 
particular level established by statute and regulations.[Footnote 9] In 
calendar year 2002, about 5.5 million disabled workers (age 18-64) 
received about $55.5 billion in DI benefits, and about 3.8 million 
working-age individuals with disabilities received about $18.6 billion 
in SSI federal benefits.[Footnote 10]

To obtain disability benefits, a claimant must file an application 
online,[Footnote 11] by telephone or mail, or in person at any Social 
Security office. If the claimant meets the nonmedical eligibility 
criteria, the field office staff forwards the claim to the appropriate 
state Disability Determination Service (DDS) office. DDS staff--
generally a team comprised of disability examiners and medical 
consultants--review medical and other evidence provided by the 
claimant, obtaining additional evidence as needed to assess whether the 
claimant satisfies program requirements, and make the initial 
disability determination. If the claimant is not satisfied with this 
determination, the claimant may request a reconsideration of the 
decision within the same DDS.[Footnote 12] Another DDS team will review 
the documentation in the case file, as well as any new evidence the 
claimant may submit, and determine whether the claimant meets SSA's 
definition of disability. In 2002, the DDSs made 2.3 million initial 
disability determinations and over 484,000 reconsiderations.

If the claimant is not satisfied with the reconsideration, he or she 
may request a hearing before an ALJ. Within SSA's Office of Hearings 
and Appeals (OHA), there are approximately 1,150 ALJs who are located 
in 140 hearing offices across the country. The ALJ conducts a new 
review of the claimant's file, including any additional evidence the 
claimant submitted after the DDS determination. At a hearing, the ALJ 
may hear testimony from the claimant, medical experts on the claimant's 
medical condition, and vocational experts regarding whether the 
claimant could perform work he or she has done in the past or could 
perform other jobs currently available in the national 
economy.[Footnote 13] ALJs have an obligation to initiate the 
development of evidence as needed and make every effort to obtain all 
necessary evidence before the hearing. The hearings are recorded, and 
the majority of claimants are represented at these hearings by an 
attorney or a nonattorney representative, such as a legal aide, parent, 
relative, or social worker. In addition, translators may be used for 
claimants with limited proficiency in English. In fiscal year 2002, 
ALJs made over 438,000 disability decisions.

If the claimant is not satisfied with the ALJ decision, the claimant 
may request a review by SSA's Appeals Council, which is the final 
administrative appeal within SSA. The Appeals Council may grant, deny, 
or dismiss a request for review. If it agrees to review the case, the 
Appeals Council may uphold, modify, or reverse the ALJ's action or it 
may remand the case back to the ALJ level for an ALJ to hold another 
hearing and issue a new decision. In fiscal year 2002, the Appeals 
Council reviewed over 108,000 disability decisions, about 27,000 of 
which were remanded.[Footnote 14]

SSA's Office of Quality Assurance and Performance Assessment (OQA) 
conducts quality assurance reviews of ALJ decisions to promote fair and 
accurate hearing decisions. These quality assurance reviews include an 
evaluation of ALJ adjudicative and procedural issues. The findings and 
information of these reviews are included in biennial reports and 
assist the OHA in its pursuit of quality by identifying specific areas 
of concern. These findings also support the "hearings decisional 
accuracy rate" measure in SSA's annual performance plans and reports.

To conduct its quality assurance review, OQA selects a random sample 
each month from the universe of ALJ decisions, stratifying the 
selection of cases by region and decisional outcome (approval or 
denial). Then, for each selected decision, SSA requests the case file 
and a recording of the hearing proceedings from hearing offices and 
storage facilities across the country.[Footnote 15] To collect the data 
SSA uses in its review, SSA staff conducts a systematic review of each 
case, including: a review of the ALJ decision by another ALJ (i.e., a 
peer review), a review of the medical evidence provided at each level 
of adjudication performed by one or more medical consultants,[Footnote 
16] and a general review of the documentation and decision at each 
adjudicative level by a disability examiner.

The peer review of an ALJ decision includes a reviewing judge's 
assessment of whether the ALJ's ultimate decision to allow or deny 
benefits is supported by substantial evidence.[Footnote 17] These 
assessments are referred to in the quality assurance review as support 
or accuracy rates. The peer review also includes judgments about the 
fairness of the ALJ hearing, in which the reviewing judge evaluates a 
number of issues, including abuse of discretion[Footnote 18] and error 
of law.[Footnote 19] The results of the peer review, as well as the 
results of the medical and general reviews, comprise SSA's enhanced 
data.

Over the years, GAO and SSA have studied SSA's ability to administer 
its disability programs in a fair and unbiased manner. In our 1992 
report,[Footnote 20] we found that racial differences in ALJ allowance 
rates were not explained by other factors related to the disability 
decision-making process. We recommended, and SSA agreed, to further 
investigate the reasons for the racial differences at the hearings 
level and act to correct or prevent any unwarranted disparities. In 
response to our recommendations, SSA conducted its own study of ALJ 
allowance rates by race, using its enhanced data from 1992 to 1996. 
Although the results were never published, SSA officials told us that 
they found no evidence of unwarranted racial differences at the 
hearings level. In our 2002 report,[Footnote 21] we assessed the steps 
SSA had taken to study allowance rates by race, and we found that 
methodological weakness precluded us from drawing conclusions on 
whether unwarranted racial differences in ALJ allowance rates existed.

SSA's enhanced data indicate that racial differences exist in overall 
allowance rates for disability benefits at the hearings level. As shown 
in table 2, these differences in allowance rates by race exist to 
varying degrees in almost every SSA region. However, differences in 
allowance rates by race do not necessarily point to racial 
discrimination because claimants from different racial/ethnic groups 
may have other differences that influence allowance decisions.

Table 2: Percentage of Claimants Allowed Benefits at the Hearings Level 
by Race and Region, 1997 to 2000:

Region: All regions; Numbers in percent: All: 59; Numbers in percent: 
White: 63; Numbers in percent: African-American: 49; Numbers in 
percent: Other race/ethnicity: 51.

Region: Region 1 Boston; Numbers in percent: All: 73; Numbers in 
percent: White: 76; Numbers in percent: African-American: 66; Numbers 
in percent: Other race/ethnicity: 62.

Region: Region 2 New York; Numbers in percent: All: 64; Numbers in 
percent: White: 72; Numbers in percent: African-American: 51; Numbers 
in percent: Other race/ethnicity: 57.

Region: Region 3 Philadelphia; Numbers in percent: All: 60; Numbers in 
percent: White: 62; Numbers in percent: African-American: 59; Numbers 
in percent: Other race/ethnicity: 37.

Region: Region 4 Atlanta; Numbers in percent: All: 60; Numbers in 
percent: White: 65; Numbers in percent: African-American: 51; Numbers 
in percent: Other race/ethnicity: 61.

Region: Region 5 Chicago; Numbers in percent: All: 55; Numbers in 
percent: White: 59; Numbers in percent: African-American: 46; Numbers 
in percent: Other race/ethnicity: 45.

Region: Region 6 Dallas; Numbers in percent: All: 54; Numbers in 
percent: White: 61; Numbers in percent: African-American: 39; Numbers 
in percent: Other race/ethnicity: 52.

Region: Region 7 Kansas City; Numbers in percent: All: 59; Numbers in 
percent: White: 61; Numbers in percent: African-American: 51; Numbers 
in percent: Other race/ethnicity: 45.

Region: Region 8 Denver; Numbers in percent: All: 59; Numbers in 
percent: White: 61; Numbers in percent: African-American: 66; Numbers 
in percent: Other race/ethnicity: 48.

Region: Region 9 San Francisco; Numbers in percent: All: 53; Numbers in 
percent: White: 57; Numbers in percent: African-American: 49; Numbers 
in percent: Other race/ethnicity: 45.

Region: Region 10 Seattle; Numbers in percent: All: 60; Numbers in 
percent: White: 62; Numbers in percent: African-American: 53; Numbers 
in percent: Other race/ethnicity: 51.

Source: GAO analysis of weighted enhanced data.

[End of table]

Race and Other Factors Influence ALJ Decisions for Some Claimant 
Groups:

When we controlled for a comprehensive range of factors that could 
affect disability decision making by ALJs, we identified a number of 
variables, including race, which influence the likelihood that a 
claimant is allowed benefits.[Footnote 22] Specifically, we found that 
numerous variables representing medical and nonmedical criteria that 
are used in the disability decision-making process had a statistically 
significant influence on ALJ decisions. We also found that participants 
in the decision-making process, such as attorneys and translators, 
influenced ALJ decisions. In addition, our statistical model shows that 
a claimant's race affects ALJ decisions for some but not all groups of 
claimants. Finally, other factors that, like race, are not part of the 
hearings process also affect ALJ decision making. For example, male 
claimants and claimants with low incomes are less likely to be awarded 
benefits. However, as with almost all statistical analyses, we cannot 
be certain whether the differences we identified are due to unequal 
treatment, limitations in our data, or some combination of the two.

Medical and Nonmedical Criteria Affect ALJ Decision Making:

Consistent with SSA's disability decision-making process, the results 
of our statistical model show that a number of variables representing 
key criteria used in the process have a statistically significant 
effect on the likelihood of allowance. For example, claimants with 3 or 
more impairments were more likely to be allowed than claimants with 1-
2 impairments, and claimants with 1 or more severe impairments were 
more likely to be allowed than claimants with no severe impairments. 
Moreover, claimants with the physical capacity to perform light work, 
sedentary, and less than sedentary work were more likely to be allowed 
than claimants with the physical capacity to perform heavy work. 
Furthermore, claimants who did not have the mental capacity to perform 
unskilled work were more likely to be allowed than claimants with the 
mental capacity to perform such work. In addition, we found that 
claimants who were 50 years old or older were more likely to be allowed 
than claimants who were 18-24 years old. Finally, claimants with 10 or 
more years of employment were more likely to be allowed than claimants 
with less than 2 years of employment.

Participants in the Hearings Process also Influence ALJ Decisions:

Our statistical analyses also show that the presence of various 
participants in the hearings process also affects ALJ allowances. For 
example, claimants who were present at the hearing were more likely to 
be allowed than claimants who were not present at the hearing. In 
addition, claimants were less likely to be awarded benefits if a 
vocational expert testified at their hearing than claimants who did not 
have a vocational expert testify at their hearing. Also, claimants who 
had translators at the hearing (i.e., for claimants who do not speak 
English proficiently) were less likely to be awarded benefits than 
claimants who did not have translators (i.e., who presumably do speak 
English proficiently). Finally, claimants who were represented by an 
attorney or a person who is not an attorney (such as a legal aide, 
relative, or friend) were more likely to be allowed than claimants who 
had no representative.[Footnote 23]

Effect of Race on ALJ Decisions Varies among Claimant Groups:

Our statistical analyses also show that, after controlling for a range 
of factors, a claimant's race also affects ALJ decisions for some 
groups of claimants. Specifically, we found no statistically 
significant difference in the likelihood of being awarded benefits 
between white claimants and claimants from other, non-African-American 
racial/ethnic groups. However, this result is likely due to our 
controlling for the presence of translators at hearings. Before 
controlling for the presence of translators, claimants from other 
racial/ethnic groups were less likely to be awarded benefits than white 
claimants. After controlling for the presence of translators, there is 
no statistically significant effect of the other race/ethnic claimants' 
category on the likelihood of allowance. The relatively high incidence 
of translators among claimants from other racial/ethnic backgrounds 
explains why we found no statistically significant differences in the 
likelihood of being awarded benefits between whites and claimants from 
other racial/ethnic groups.[Footnote 24]

When we compared white claimants with African-American claimants, we 
found statistically significant differences in the likelihood of 
allowance, but only among claimants who had no representation.[Footnote 
25] For example, among claimants with no representation, the odds of 
being allowed benefits for African-Americans were about one-half the 
odds of being allowed for whites.[Footnote 26] In contrast, among 
claimants with attorney representation, we found no statistically 
significant difference in the likelihood of allowances between whites 
and African-Americans.[Footnote 27]

In addition, when we compared the effect of having attorney 
representation with the effect of not having attorney representation, 
we found that these effects also vary by race. That is, we found that 
the effect of attorney representation is larger for African-American 
claimants than it is for white claimants. Specifically, the odds of 
being allowed benefits for African-American claimants with attorney 
representation were more than 5 times higher than the odds of being 
allowed for African-American claimants without attorney 
representation. In comparison, the odds of being allowed benefits for 
white claimants with attorney representation were three times higher 
than the odds of being allowed benefits for white claimants with no 
representation.[Footnote 28]

Finally, we used another statistical technique--the Oaxaca 
decomposition--to analyze differences in ALJ allowances between 
African-American and white claimants. Consistent with the results from 
our other analyses, we found that, among claimants with attorney 
representation, differences between African-Americans and whites can be 
explained largely by differences in other factors included in our 
model, whereas among claimants without attorney representation, 
differences between African-Americans and whites were explained to a 
lesser degree by differences in other factors in our model.[Footnote 
29] These results are particularly important because a larger 
percentage of African-American claimants do not have attorneys (39 
percent) in comparison with white claimants (29 percent).

Although several possible explanations exist for why attorney 
representation increases a claimant's likelihood of being awarded 
benefits, we cannot empirically explain why the effect of attorney 
representation is greater for African-Americans. According to two 
attorneys affiliated with the National Organization of Social Security 
Claimant Representatives (NOSSCR), attorneys increase the claimant's 
likelihood of being awarded benefits by (1) providing assistance with 
the development of evidence over and above SSA's efforts to develop 
evidence[Footnote 30] and (2) preparing claimants to improve their 
effectiveness and credibility as witnesses. Another possible 
explanation for why attorney representation influences the likelihood 
of being awarded benefits is that attorneys often screen cases to 
select claimants with strong cases.[Footnote 31] However, given the 
data available to us, we cannot empirically explain why attorney 
representation has a stronger effect for African-American claimants 
than for white claimants.

As mentioned earlier, claimants who are represented by persons other 
than attorneys--such as legal aides, friends, or family--are also more 
likely to be allowed than claimants with no representation. When we 
conducted additional analyses on the effect these nonattorney 
representatives had on allowances by race, we found, regardless of 
race, claimants who were represented by nonattorneys had a greater 
likelihood of being awarded benefits than claimants who were not 
represented. Nevertheless, we also found that differences by race 
persisted after controlling for nonattorney representatives.[Footnote 
32]

Other Factors Not Part of the Decision-Making Process also Influence 
ALJ Allowances:

Finally, our statistical analyses found that additional factors not 
part of the decision-making process--including the claimant's earnings, 
geographical location, and sex--influence the ALJ allowance decision. 
For example, we found that claimants with higher levels of earnings 
were more likely to be awarded benefits than those who have low 
earnings levels. In particular, the odds of being allowed benefits for 
claimants who earned over $20,000 per year were 3 times higher than the 
odds of being allowed benefits for claimants who earned less than 
$5,000 per year, and the odds of being allowed for claimants who earn 
$5,000-$20,000 per year were 2 times higher than for claimants who earn 
less than $5,000 per year. In addition, the odds of being allowed 
benefits for claimants whose hearings took place in the Boston Region 
were approximately 2 times higher than for claimants whose hearings 
took place in other regions, after controlling for other 
factors.[Footnote 33] Finally, the odds of being allowed benefits for 
claimants who are men were approximately three-quarters as high as for 
female claimants.

Data Limitations Prevent Definitive Conclusions Regarding the Cause of 
Unexplained Racial Differences in ALJ Decisions:

The existence of persistent, unexplained differences by race and other 
factors not used as criteria in the decision-making process--after we 
controlled for as many factors as the data allowed--means that we 
cannot rule out the possibility that claimant groups are being treated 
unequally. However, two limitations, common to almost all multivariate 
analyses, prevent us from definitively determining whether unexplained 
differences in allowance decisions by claimant groups are due to 
discrimination or other forms of bias in the decision-making process. 
First, differences between claimant groups may be a result of a lack of 
precision in some of the variables in the model. For example, when the 
severity of a claimant's impairment is evaluated by the medical 
examiners, they are placed in one of five categories. However, the 
categories may not capture subtle differences in impairment severity. 
This is true for many of the categorical variables in the 
model.[Footnote 34] With more detailed information on severity and 
other factors, we might have been able to better explain differences by 
race. Second, differences that we see in the likelihood of being 
awarded benefits between claimant groups may be the result of a lack of 
data on certain factors that are relevant for our analysis. For 
example, data on claimants' access to medical care are not available. 
In the past, SSA developed data on the source of the claimant's medical 
care--a proxy for the quality of the medical care and a factor that 
determines the weight that is placed on a given piece of evidence. 
However, SSA told us that it stopped developing these data due to 
resource constraints. Other factors such as these, if included in the 
model, might further explain some of the differences we found in ALJ 
decisions by race, as well as other differences we found, for example, 
by sex and income.

In addition, our model's results concerning the effect of attorney 
representation on ALJ decisions might be somewhat inflated due to SSA's 
systematic exclusion of certain cases--namely, the exclusion of denied 
ALJ decisions that were appealed to the Appeals Council--from the 
enhanced data we used for our study. An upward bias of this effect 
could occur because the denied cases that were appealed (and, 
therefore, excluded from our dataset) exhibited a higher rate of 
attorney representation than the denied cases that were not appealed. 
However, further analyses suggest that our estimates of the different 
effects of attorney representation by race (that is, the larger effect 
of attorney representation for African-Americans) are not likely to be 
inflated. (See appendix I for a detailed discussion of our analyses of 
this limitation.):

SSA's Approach to Quality Assurance Reviews Limits Its Ability to 
Ensure the Accuracy and Fairness of ALJ Decisions:

Analytical, sampling, and data weaknesses in SSA's approach to quality 
assurance reviews limit its ability to ensure the accuracy and fairness 
of ALJ decisions. SSA does not analyze ALJ decisions by race, which 
limits its ability to identify, correct, and prevent unwarranted racial 
differences in allowance rates. In addition, weaknesses in the quality 
assurance review's sampling methods and data availability present 
problems.

SSA's quality assurance review of ALJ decisions includes numerous 
analyses of ALJ decisions, including analyses of support rates and 
whether an ALJ abused his or her discretion or committed an error of 
law.[Footnote 35] In addition, SSA analyzes ALJ decisions by various 
claimant characteristics such as the claimant's age and the region 
where the disability decision was issued.[Footnote 36] However, SSA 
does not currently analyze ALJ decisions by race.[Footnote 37] By not 
analyzing ALJ decisions by race as part of its ongoing quality 
assurance review, SSA is limited in its ability to identify, correct, 
and prevent unwarranted racial differences in allowance rates. At the 
time of our review, SSA had no plans to analyze decisions by race as 
part of its ongoing quality assurance review of ALJ decisions.

Even if SSA decided to analyze ALJ decisions and related data by race, 
weaknesses in the quality assurance review's sampling methods would 
present problems. Specifically, SSA is limited in its ability to 
conduct certain types of analyses by race because SSA does not take 
measures to ensure the presence of a sufficient number of claimants in 
each race/ethnicity category for its quality assurance reviews. As 
noted in our previous report,[Footnote 38] since 1997, SSA no longer 
stratifies the selection of ALJ decisions by race (i.e., by African-
American and non-African-American) when selecting a random sample of 
cases--a practice that had helped to ensure that SSA had a sufficient 
number of cases of African-American claimants in its sample to analyze 
ALJ decisions by race. Unless SSA over-samples cases for African-
Americans and claimants from other racial/ethnic groups, certain 
analyses by race/ethnicity cannot be performed. For example, due to the 
low number of African-American claimants in SSA's enhanced data, we 
were unable to analyze differences by race/ethnicity for those ALJ 
decisions that were considered to be unsupported by the reviewing 
judge. Furthermore, we were unable to analyze by race whether the ALJ 
followed the appropriate procedures in deciding whether the claimant 
was eligible for disability benefits.[Footnote 39] Because these 
analyses for African-American cases would rely on a relatively small 
number of decisions, conclusions related to race could be statistically 
unreliable.

SSA also excludes cases that are appealed to the Appeals Council from 
its quality assurance review--a sampling weakness that affects SSA's 
entire quality assurance review process. SSA estimates that about 75 
percent of ALJ denials are appealed. By excluding such cases, SSA may 
be running the risk of using a nonrepresentative sample in its analyses 
of ALJ decisions and, consequently, drawing incorrect conclusions about 
the accuracy and fairness of ALJ decisions, although we did not find 
large differences in the sample we used for our analysis.[Footnote 40] 
For example, cases are often appealed on the basis of an alleged error 
of law or abuse of discretion; therefore, SSA may be omitting cases 
with information that could be valuable in assessing the fairness of 
ALJ decisions.

According to SSA officials, SSA does not include appealed cases in its 
ALJ quality assurance review because generally SSA has yet to render a 
final decision for them. SSA believes that the Appeals Council decision 
could be inappropriately influenced by information resulting from the 
quality assurance review of these "live" cases. However, SSA officials 
informed us that they are considering ways to include appealed cases in 
their ALJ quality assurance review for which final decisions have been 
rendered.[Footnote 41] According to SSA officials, this would require 
establishing a special control system so that SSA can recover the files 
and tapes after the cases have been reviewed at the Appeals Council and 
have received a final decision.[Footnote 42] SSA officials said this 
approach would also require removing any information regarding the 
final decision from the files, so that the reviewing judge can assess 
the cases without being influenced by this additional information. One 
concern that SSA has about reviewing appealed cases that have received 
a final decision is the 1-to 2-year time lag before the quality 
assurance review could take place.[Footnote 43] SSA officials informed 
us that reviewing cases 1 to 2 years after the original ALJ decision 
could affect the quality of the data and the effectiveness of the 
quality assurance review process.[Footnote 44] Another concern that SSA 
has regarding this approach is that reviewing judges would know which 
cases were appealed to the Appeals Council and might analyze appealed 
cases differently from those cases that were not appealed.

In addition to having analytical and sampling weaknesses, SSA's quality 
assurance reviews do not collect certain types of data that could be 
useful in conducting its analyses of ALJ decisions. For example, SSA 
does not collect information on the types and sources of medical 
evidence in the claimant's file. Types of medical evidence could 
include treatment records, narrative reports, results of laboratory or 
clinical tests, and frequency of medical visits, and sources of medical 
evidence could include treating physician, other specialist, hospital 
(inpatient), and clinic or hospital (outpatient). This kind of 
information, which was collected by SSA in the past, but is no longer 
collected, could be used to study the impact of various types and 
sources of medical evidence on the likelihood that a claimant would be 
awarded benefits. For example, as part of its quality assurance review, 
SSA would be able to analyze the relationship between claimants' access 
to health care (as measured by the presence of a treating physician or 
the number or length of doctor visits) and ALJ decisions to allow or 
deny benefits. SSA would also be able to determine whether the extent 
of medical evidence in the claimant's file is affected by attorney 
representation, or the race, sex, or income of the claimant.

Additionally, since 1990, SSA has significantly scaled back its 
collection of race/ethnicity data, leaving gaps for certain claimant 
groups. As we noted in our previous report,[Footnote 45] SSA requests 
information on race/ethnicity from individuals who complete a form to 
request a new or replacement Social Security card. The race/ethnicity 
field on this form is a voluntary field and the data collected are 
self-reported. Although this process is still in place, only a small 
portion of SSNs is issued in this manner today. Since 1990, SSA has 
been assigning SSNs to newborns through its Enumeration at Birth (EAB) 
program, and SSA does not collect race/ethnicity data through the EAB 
program. In fiscal year 2002, approximately 90 percent of the 4.2 
million original SSN cards issued to U.S. citizens were through the EAB 
program. Consequently, SSA has not collected race data for those 
individuals who obtained their SSNs through the EAB program and, under 
its current approach, SSA would not generally collect these data in the 
future.[Footnote 46] As future generations obtain their SSNs through 
the EAB program, the number and proportion of claimants for whom SSA 
lacks race/ethnicity data are likely to increase.

This lack of race data has implications on SSA's ability--and the 
ability of other federal agencies that rely on SSA for race/ethnicity 
data--to conduct certain types of analyses by race/ethnicity. Although 
we had sufficient race data for our study,[Footnote 47] SSA's future 
ability to identify, correct, and prevent racial differences in ALJ 
decisions will be hampered by this growing lack of data for claimants 
who received their SSNs through the EAB program. This growing lack of 
data will also affect the ability of other federal agencies that rely 
on SSA for race/ethnicity data, such as the Centers for Medicare and 
Medicaid Services, to conduct research and produce reports to ensure 
the fairness of their programs.

During our review, SSA decided to collect race/ethnicity data on 
individuals applying for disability or other Social Security benefits 
at the time of application. Previously, SSA did not collect race data 
at the point of application for disability benefits since race is not a 
criterion in the disability determination process. However, during our 
review, SSA decided to collect data on race/ethnicity because, 
according to SSA officials, the agency now views collecting and 
analyzing these data as important for research purposes and to ensure 
the race neutrality of its programs. SSA recently set up a task group 
to explore implementation issues. Even though this decision to collect 
race information has been made, SSA has not set a start date, and SSA 
officials anticipate that implementation of this endeavor will be a 
lengthy process.

Conclusions:

Our analyses of SSA's enhanced data from its quality assurance reviews 
show that for claimants who are not represented by attorneys, there are 
differences in the likelihood of being awarded benefits between 
African-Americans and whites that cannot be explained by other factors 
related to the disability decision-making process. Although our 
empirical results cannot be used as proof that discrimination or some 
other form of bias exists, the results also do not rule out this 
possibility. As such, our findings raise important program integrity 
issues for SSA in terms of its ability to ensure that disability 
decisions are made accurately and fairly. Relatedly, the results of our 
analyses raise questions regarding the role and influence that attorney 
and nonattorney representatives have in the decision-making process; 
although SSA does not require claimants to have representation, the 
results of our analysis show that claimants with representation are 
more likely to be awarded benefits than those without representation. 
The lower likelihood of being awarded benefits for other claimant 
groups, including non-English-speaking claimants with translators, 
claimants with low income, and claimants who are men, also raise 
questions about the fairness of SSA's disability decision-making 
process. These findings point to the need for SSA's continued efforts 
to understand racial and other differences in ALJ allowances. While SSA 
may not have control over the sources of some of these differences, 
understanding the sources of these differences is the key to taking the 
necessary steps to demonstrate the neutrality of its decision-making 
process and to eliminate and prevent unwarranted differences in 
allowance rates.

SSA's approach to quality assurance reviews has limited its ability to 
understand these differences and take appropriate action, if necessary, 
in several ways. For example, because SSA does not over-sample cases 
for African-Americans and claimants from other racial/ethnic groups and 
analyze the ALJ decisions by race, it cannot determine whether 
inaccuracies in ALJ decision making, such as errors of law and abuses 
of discretion, occur with the same likelihood for claimants of 
different racial/ethnic backgrounds. Additionally, by not including 
cases appealed to the Appeals Council with those that undergo an ALJ 
quality review, SSA's sample is potentially nonrepresentative of all 
ALJ decisions. Moreover, the agency misses an opportunity to analyze 
precisely those cases that are more likely to have had an alleged error 
of law or abuse of discretion by the ALJ. Finally, SSA no longer 
collects data on type and source of medical evidence that would allow 
for more careful analyses of the accuracy and fairness of ALJ 
decisions. Although SSA has significantly scaled back its collection of 
race/ethnicity data since 1990, we applaud the agency's recent decision 
to begin collecting these data at the point of application for 
disability and other benefits, which will help to fill some of the gaps 
in its race/ethnicity data.

Recommendations:

To improve SSA's ability to ensure the accuracy and fairness of ALJ 
decisions, we recommend that the agency conduct ongoing analyses of ALJ 
decisions by race/ethnicity, as well as by other claimant groups (such 
as claimants with attorneys and nonattorneys, with translators, with 
low incomes, from certain regions and claimants who are men). In doing 
so, it should take the following steps to enhance its approach to 
quality assurance reviews:

* Collect data on the types and sources of medical evidence in the 
claimant's file to better understand the agency's and attorney's role 
in the development of evidence.

* Analyze differences in support (accuracy) rates, in addition to 
differences in allowance decisions.

* Over-sample the selection of ALJ decisions by African-American 
claimants and, to the extent possible, other racial/ethnic groups to 
ensure that SSA has a sufficient number of cases to conduct analyses of 
ALJ decisions by race.

* Publish methods used and results as part of its biennial reporting on 
the findings of its disability hearings quality review process.

* If needed, take actions to correct and prevent any unwarranted 
differences in allowance and support rates among racial/ethnic and 
other claimant groups.

To further ensure the accuracy and fairness of ALJ decisions for 
various claimant groups, we recommend that SSA conduct in-depth 
investigations of cases (e.g., case studies) to better understand 
differences in ALJ allowances for certain claimant groups, including 
claimants with and without an attorney. The results of these 
investigations should also be published in the biennial reports. If 
needed, SSA should take actions to correct and prevent any unwarranted 
differences in allowance rates among these claimant groups.

To ensure that SSA uses a sample that is representative of all ALJ 
decisions in its quality assurance review, we recommend that the agency 
restructure its sampling process to incorporate cases that are appealed 
to SSA's Appeals Council in the quality assurance review sample. These 
appealed cases should be analyzed together with, rather than separate 
from, the rest of SSA's quality assurance sample.

In light of the methodological complexities associated with analyzing 
ALJ decisions, we recommend that SSA establish an advisory panel 
comprised of external experts in a range of disciplines--including 
statistics/econometrics, design methodology, law, medicine, vocational 
training, and disability--to provide leadership, oversight, and 
technical assistance with respect to conducting these and other quality 
assurance reviews of ALJ decisions.

Agency Comments:

We provided a draft of this report to SSA for comment. In its written 
comments, SSA said that our report was useful and timely and agreed 
with all of our recommendations. SSA also indicated that it intends to 
go further. For example, SSA noted that, as part of its overall plan to 
improve the disability determination process, it intends to look at all 
factors that may produce adverse impacts based on race, ethnicity, 
national origin, or gender. In addition, SSA is currently developing 
recommendations on how to collect meaningful data on race and 
ethnicity. SSA's comments, as well as its recently proposed plan for 
improving the disability determination process, are printed in appendix 
III.

We are sending copies of this report to the Social Security 
Administration, appropriate congressional committees, and other 
interested parties. We will also make copies available to others on 
request. In addition, the report will be available at no charge on 
GAO's Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please 
call me or Carol Dawn Petersen, Assistant Director, at (202) 512-7215. 
Staff acknowledgments are listed in appendix IV.

Robert E. Robertson: 
Director, Education, Workforce, and Income Security Issues: 

Signed by Robert E. Robertson:

[End of section]

Appendix I: Scope and Methods:

To determine whether decisions by Administrative Law Judges (ALJs) to 
allow disability claims were affected by the race of the claimant, we 
developed a model of ALJ decision making that tested for racial 
differences after controlling for other factors related to the 
disability decision-making process. These factors included (1) factors 
that represent criteria in the decision-making process; (2) factors 
that represent participants in the decision-making process; and (3) 
factors that are not part of, but may influence, the decision-making 
process. To conduct our analysis, we employed logistic regression 
models and Oaxaca decomposition methods. We used data from the Social 
Security Administration's (SSA) quality assurance review at the 
hearings level, which we refer to as the enhanced data. The enhanced 
data contain detailed information--some of which was previously 
unavailable to GAO--on medical and vocational factors for a sample of 
7,908 SSA claimants.

Prior to constructing these models, we conducted analyses related to 
data quality. Given our previously reported concerns about the degree 
to which the enhanced data are representative,[Footnote 48] we 
conducted tests to determine whether the enhanced data were 
sufficiently representative and reliable for our analyses. 
Specifically, in these analyses, we sought to determine (1) whether the 
more detailed medical and vocational information included in the 
enhanced data set were sufficiently important to justify using this 
restricted sample of claimants and (2) whether the sample of claimants 
for which the enhanced data were available was representative of the 
broader population of claimants.

We developed our analyses and models in consultation with GAO 
methodologists, expert consultants, and SSA officials.[Footnote 49]

This appendix is organized into five sections: Section 1 describes the 
data that were used in the analysis of potential racial disparities, as 
well as data that were used in the analyses of data quality. Section 2 
describes analyses and results related to our tests of data quality and 
reliability. Section 3 provides background on the weighting scheme used 
in the analysis, as well as details on sampling errors. Section 4 
describes the variables that were included in our baseline and final 
models and presents the results of these final models and the Oaxaca 
decomposition analysis. Finally, Section 5 presents the limitations of 
our analyses.

Section 1: Databases and Information Sources:

We used two types of SSA data to conduct our analyses: (1) the enhanced 
data, which were derived from a sample of SSA claimants, and (2) 
administrative data, which were derived from the universe of claimants.

The enhanced data are compiled by the Division of Disability Hearings 
Quality (DDHQ) within SSA's Office of Quality Assurance (OQA). These 
data are compiled as part of an ongoing quality assurance review of the 
decision-making accuracy of ALJs. The review involves an examination of 
the initial, reconsideration, and hearings level decisions by a medical 
consultant, a disability examiner, and an ALJ.

The administrative data were obtained from several sources. For each 
adjudicative level (the initial and reconsideration, hearings, and 
Appeals Council levels), SSA has an electronic file that contains a 
limited amount of data for each claimant. In addition to these three 
datasets, we used earnings data from SSA's Master Earnings File (MEF).

We used these data for the various analyses that are described more 
fully in later sections. In brief, we used the enhanced data for our 
"severity analysis," which sought to determine whether the enhanced 
data contained variables that were better measures of the claimant's 
medical severity than the variables contained in SSA's administrative 
files. We used the administrative data for our "nonresponder analysis," 
which sought to determine whether the enhanced data were 
representative. Based on the results of the severity and nonresponder 
analyses, we decided to use the enhanced data for our analysis of 
potential racial disparities.

Table 3 presents the datasets that we used in our analyses, the 
decision-making level to which the particular dataset pertains, the 
analyses for which we used the particular dataset, and the years of 
data and the specific variables that were used in our analyses.

Table 3: Data Used in Our Analyses:

Dataset: Enhanced data; Decision-making levels to which data generally 
pertain: Hearings level[A]; Analyses conducted: Final analysis and 
severity analysis; Years used in analyses: Oct. 1997-Sept. 2000; 
Information that was used in analyses: Claimant's impairments, severity 
of impairments, alcohol or drug abuse, consultative exam requested, 
number of impairments, number of severe impairments, residual 
functional capacity of claimant, mental residual functional capacity of 
claimant, occupational type, years of employment, occupational skill 
level, years of education, literacy, age, type of representation, other 
hearing participants (vocational expert, medical expert, translator, 
and claimant), sex, race, claim type, year of decision, region, and the 
allowance decision at the hearing level.

Dataset: 831 data[B]; Decision-making levels to which data generally 
pertain: Initial and reconsideration levels; Analyses conducted: 
Nonresponder analysis; Years used in analyses: 1990-2000; Information 
that was used in analyses: Claimant's age, sex, race, body systems 
affected by the impairment(s) alleged at the initial and 
reconsideration levels, occupational years, years of education, whether 
the claimant obtained a consultative exam, and claim type.

Dataset: Office of Hearings and Appeals Case Control System (CCS) 
data[B]; Decision-making levels to which data generally pertain: 
Hearings level; Analyses conducted: Nonresponder analysis; Years used 
in analyses: Oct. 1997-Sept. 2000; Information that was used in 
analyses: Claimant's body system affected by the impairment(s) alleged 
at the hearing level, type of representation, other hearing 
participants (vocational expert, medical expert, translator and 
claimant), and the allowance decision at the hearing level.

Dataset: Appeals Council Automated Processing System (ACAPS)[B]; 
Decision-making levels to which data generally pertain: Appeals Council 
level; Analyses conducted: Nonresponder analysis; Years used in 
analyses: 1997-2002; Information that was used in analyses: Indicator 
of whether claimant appealed the allowance decision at the hearing 
level and allowance decision at the Appeals Council level.

Dataset: Master Earnings File[B]; Decision-making levels to which data 
generally pertain: N/A; Analyses conducted: Final analysis; Years used 
in analyses: 1948-2002; Information that was used in analyses: Yearly 
individual earnings.

Source: Social Security Administration.

[A] The enhanced data also contain variables pertaining to conditions 
or actions taken at the initial and reconsideration levels for a sample 
of claimants who have appealed to an Administrative Law Judge.

[B] The use of this database was restricted to only those observations 
that had matches with the SSNs that were included in the enhanced data 
or in the sample from which the enhanced data were developed.

[End of table]

Section 2: Data Reliability Tests:

To ensure that the SSA data were sufficiently reliable for our 
analyses, we conducted detailed data reliability assessments of the 
five datasets that we used. We restricted these assessments, however, 
to the specific variables and records that were pertinent to our 
analyses. We found that all of the datasets were sufficiently reliable 
for use in our analyses.

Enhanced Data:

Our reliability assessment of the enhanced data included two steps. 
First, to assess the general reliability of the enhanced data that we 
used in our analysis, we interviewed officials from SSA's DDHQ about 
procedures to ensure the enhanced data's reliability. On the basis of 
discussions with DDHQ officials, we concluded that careful data entry 
controls and processing procedures are applied in maintaining the 
reliability of the enhanced data. Second, to assess the completeness of 
the enhanced data that we used in our analyses, we conducted frequency 
analysis of relevant fields. On the basis of the results of our 
frequency tests of relevant data elements and our interviews with SSA 
officials, we concluded that the enhanced data were sufficiently 
complete and accurate for use in our final analyses.[Footnote 50]

SSA Administrative Files:

Our assessment of the reliability of the relevant data from SSA's 
administrative files (831, CCS, and ACAPS) also involved several steps. 
For each dataset, we assessed the general reliability of relevant data 
(i.e., the specific variables and records that we would use in our 
analyses) by interviewing SSA officials on their processes and 
procedures to ensure data quality. To determine the completeness of the 
data, we conducted frequency analyses of relevant fields. Finally, to 
assess the accuracy of the relevant fields, we matched the enhanced 
data with the data from the administrative files and compared the 
values of the fields common to both data sets.

On the basis of our review of existing information, we concluded that, 
while not optimal, adequate quality controls are in place to ensure the 
reliability of the specific variables from SSA's administrative files 
that we used in our analysis, and the results of our frequency tests 
and our examination of matched data confirmed that we had sufficiently 
complete and accurate data for use in our nonresponder 
analyses.[Footnote 51]

With respect to earnings data from the MEF, SSA provided us with 
complete earnings data for each person included in the enhanced data. 
We were unable to test the accuracy of earnings data from the MEF 
because comparable data were not available in the enhanced data. 
However, SSA's OQA annually reviews the accuracy of the MEF earnings 
data by extracting individual earnings from the reports submitted by 
employers and self-employed individuals and by then comparing the 
reported earnings to earnings posted to the MEF. To further ensure the 
accuracy of these data, SSA also now mails Social Security statements 
to individuals who have earnings and are age 25 years or older to 
inform individuals about their earnings.

Additional Tests of Enhanced Data:

For our final analyses, the enhanced data have some significant 
advantages over SSA's administrative files. Most importantly, the 
enhanced data contain information on medical severity[Footnote 52] that 
are not available in SSA's administrative files and were not available 
to GAO when our agency issued a report in 1992 concerning similar 
analyses.[Footnote 53] Data on medical severity are important because 
severity is a key factor in the disability allowance decision. This and 
other variables in the enhanced dataset are developed from a sample of 
hearings claimants. However, as highlighted in our 2002 report, we were 
concerned that the sample from which the enhanced data are developed 
had the potential for being unrepresentative of the population of 
hearings claimants.[Footnote 54]

The enhanced data may not be representative because SSA uses only a 
fraction of the files that it selects for its sample of ALJ decisions. 
SSA selects the sample for the enhanced data using an automated system 
that selects a stratified random sample every month from the population 
of claimants who had a hearing.[Footnote 55] However, over the period 
that we examined (1997-2000), roughly 50 percent of the files that were 
selected to be in the sample were not obtained. There were three 
primary reasons for why files were not obtained:

* The files were still in use because claimants appealed the ALJ 
decision to the next level, that is, to the Appeals Council.[Footnote 
56]

* The files were misplaced or misfiled.

* The files were still in use because there were still pending payment 
decisions for cases that were allowed.

In addition, not all of the files that were obtained underwent the 
three reviews needed to be included in our sample (i.e., reviews by an 
ALJ, a medical consultant, and a disability examiner). According to SSA 
officials we interviewed, this was due to time and budget constraints. 
After the monthly sample was selected, DDHQ requested the files from 
various storage facilities and regional offices. As the files came in, 
they were chosen to be reviewed by a medical team on a "first come, 
first serve" basis--that is, files were selected until a sufficient 
number (as deemed by DDHQ) of files for a given time period was 
reached. The remaining files were not reviewed by a medical team. 
Additionally, some of the files that were supposed to be reviewed by an 
ALJ were not reviewed. In the end, of the 50,022 that were sampled from 
1997 through 2000, only 9,082 files underwent all three reviews. For 
purposes of exposition, we will call the sample of 9,082 files that 
underwent all three reviews the "responders" and the sample of files 
that were not obtained the "nonresponders."[Footnote 57]

Given our concerns about the degree to which the enhanced data were 
representative, before we decided to use the data, we needed to 
determine (1) whether the additional information contained in the 
enhanced data were critical to our analyses (in terms of obtaining the 
best possible estimates of the variables in our model of ALJ 
decisions)[Footnote 58] and, if so, (2) whether the enhanced data were 
representative of the population of claimants at the hearings level. To 
answer these questions, we conducted (1) a "severity analysis" to 
assess whether the additional information contained in the enhanced 
data were critical to our analyses and (2) a "nonresponder analysis" to 
test whether the enhanced data are representative. We developed these 
statistical tests in consultation with our methodologists, our external 
expert consultants, and SSA officials. The results of these analyses 
indicated that the enhanced data were critical for our study and were 
of sufficient quality for analyses of ALJ allowance decisions.

Severity analysis:

The goal of our severity analysis was to determine which data would 
allow us to obtain the best possible estimates of the variables in our 
model of ALJ decisions. Ensuring that we obtain such estimates requires 
that we use data that are as precise as possible (i.e., those that best 
capture the actual characteristics of the claimant and the case). 
Imprecision in the measurement of variables that are statistically 
significantly related to the disability determination process could 
result in estimates of the differences between racial categories in 
allowances that are inappropriately larger or smaller than the real 
difference.

To determine whether variables in the enhanced data more precisely 
measured severity and other factors that influence ALJ decisions than 
variables in the 831 and CCS data, we conducted our severity 
analysis.[Footnote 59] The specific objective of this test was to 
determine (using regression analysis) whether the severity data in the 
enhanced data increased the explanatory power of the model. If it did 
not, we could use the severity data from SSA's administrative files, 
which are available for all claimants, thus avoiding any problems of 
representativeness.

To conduct our severity analysis, we compared two models of the ALJ's 
disability decision (that is, the dependent variable is the ALJ's 
decision to allow or deny disability benefits) for the same group of 
claimants. Specifically:

* Model A contained only those independent variables from the enhanced 
data that are also available in SSA's 831 and CCS files.[Footnote 60]

* Model B contained all of the independent variables in Model A, plus 
several variables that are only available in the enhanced data, 
including variables that measure medical severity at the hearings level 
(impairment severity, number of impairments, number of severe 
impairments, and residual functional capacity) as well as variables 
that measure the occupational skill level of the claimant and whether 
the claimant is literate.[Footnote 61]

To determine whether the additional variables in the enhanced data 
improved our ability to explain allowance rates, we used logistic 
regression analysis to estimate both of these models. We then compared 
the predictive power of each model and the significance of the 
additional variables in Model B.

In summary, we found that Model A (which excluded the additional 
variables that are available in the enhanced data) explained roughly 27 
percent of the variation in allowances, while Model B (which included 
those additional variables) explained over 40 percent. The results of 
this analysis show that the additional variables that are included in 
Model B increase the overall explanatory power of the model. 
Furthermore, the additional variables in Model B--such as the degree of 
medical severity, the number of impairments, the number of severe 
impairments, and measures of the claimant's residual functional 
capacity and mental residual functional capacity--were all highly, 
statistically significant predictors of the ALJ allowance decision.

Nonresponder analysis:

To determine whether the enhanced data were sufficiently 
representative, we conducted our nonresponder analysis, which tested 
whether the responders' cases (those that were included in SSA's 
enhanced data) were statistically significantly different from the 
nonresponders' cases (those that were excluded from SSA's enhanced 
data). It is important to note that we can only compare the responders 
and nonresponders on characteristics that are observable (that is, for 
which data are available).[Footnote 62] Since we are controlling for 
many of these same variables in our final model, differences we see in 
observable characteristics in our nonresponder analysis are not 
critical in and of themselves. However, if few differences exist 
between responders and nonresponders in observable characteristics, it 
is more likely (though not guaranteed) that few differences exist 
between them in unobservable characteristics. Thus, if the nonresponder 
analysis reveals little or no differences between the two groups we are 
afforded some measure of confidence that the two groups are similar in 
unobservable characteristics.

Our nonresponder analysis consisted of a series of tests to compare 
responders and nonresponders with respect to (1) the allowance decision 
and (2) characteristics that are related to the allowance decision, 
including claimant characteristics and characteristics related to 
administrative processes. To conduct these tests we used data available 
from SSA's administrative files (831, CCS, and ACAPS).[Footnote 63] We 
conducted both regression analyses and bivariate tests. Regression 
methods and related test statistics were used to estimate differences 
between responders and nonresponders after simultaneously controlling 
for other factors that could influence nonresponse. Chi-squared tests 
and t-tests were used to evaluate the differences in specific 
characteristics when other characteristics were ignored. These 
differences were estimated first for responders and nonresponders 
overall, and then for responders and nonresponders within categories of 
race, and then for responders and nonresponders within categories of 
claimants who were allowed or denied at the hearings level.

The regression analysis showed no statistically significant differences 
between responders and nonresponders in many factors that are related 
to the decision-making process. Specifically, responders were not 
statistically significantly different from nonresponders in most 
medical, vocational, and demographic characteristics including body 
system, age, sex, and race. However, the results of the regression also 
showed that responders differed from nonresponders in some 
administrative characteristics. Specifically, claimants who had 
attorney or nonattorney representation or who had a medical expert 
testify at the hearing, or had consultative exams were significantly 
less likely to be responders. We also found small, but statistically 
significant, differences in the year of the decision and the region. 
Table 4 summarizes the results of the nonresponder regression analysis, 
and presents these comparisons for (1) all responders and 
nonresponders, (2) African-American responders and African-American 
nonresponders, and (3) white responders and white nonresponders.

Table 4: Statistically Significant Differences between Responder and 
Nonresponder Groups, as Estimated with Logistic Regression:

Variable or variable groups in the model: Medical, vocational, and 
demographic characteristics: 

Variable or variable groups in the model: Body system categories[A]; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
No[B]; Statistically significant differences between: African-
American responders and African-American nonresponders: Medical, 
vocational, and demographic characteristics: No; Statistically 
significant differences between: White responders and white 
nonresponders: Medical, vocational, and demographic characteristics: 
No.

Variable or variable groups in the model: Age group categories; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
No; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: No.

Variable or variable groups in the model: Sex; Statistically 
significant differences between: All responders and all nonresponders: 
Medical, vocational, and demographic characteristics: No; 
Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: No.

Variable or variable groups in the model: African-American; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
No; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: Not applicable; Statistically 
significant differences between: White responders and white 
nonresponders: Medical, vocational, and demographic characteristics: 
Not applicable.

Variable or variable groups in the model: Years of education 
categories; Statistically significant differences between: All 
responders and all nonresponders: Medical, vocational, and demographic 
characteristics: No[C]; Statistically significant differences 
between: African-American responders and African-American 
nonresponders: Medical, vocational, and demographic characteristics: 
No[D]; Statistically significant differences between: White 
responders and white nonresponders: Medical, vocational, and 
demographic characteristics: No.

Variable or variable groups in the model: Administrative 
characteristics: 

Variable or variable groups in the model: Attorney representation; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
Yes; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: Yes.

Variable or variable groups in the model: Nonattorney representation; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
Yes; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: Yes.

Variable or variable groups in the model: Medical expert at hearing; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
Yes; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: Yes.

Variable or variable groups in the model: Translator at hearing; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
No; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: No.

Variable or variable groups in the model: Vocational expert at hearing; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
No; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: No.

Variable or variable groups in the model: Supplemental Security Income 
(SSI) claim; Statistically significant differences between: All 
responders and all nonresponders: Medical, vocational, and demographic 
characteristics: No; Statistically significant differences between: 
African-American responders and African-American nonresponders: 
Medical, vocational, and demographic characteristics: No; 
Statistically significant differences between: White responders and 
white nonresponders: Medical, vocational, and demographic 
characteristics: No.

Variable or variable groups in the model: Consultative examination; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
Yes; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: Yes; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: Yes.

Variable or variable groups in the model: Year of decision; 
Statistically significant differences between: All responders and all 
nonresponders: Medical, vocational, and demographic characteristics: 
Yes; Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: Yes; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: Yes.

Variable or variable groups in the model: Region; Statistically 
significant differences between: All responders and all nonresponders: 
Medical, vocational, and demographic characteristics: Yes; 
Statistically significant differences between: African-American 
responders and African-American nonresponders: Medical, vocational, 
and demographic characteristics: No; Statistically significant 
differences between: White responders and white nonresponders: 
Medical, vocational, and demographic characteristics: Yes.

Source: GAO analysis of 831 and CCS data.

Note: Dependent variable is 1 if the claimant is a responder and 0 if 
the claimant is a nonresponder.

[A] Body system categories represent the body system that was affected 
by the claimant's impairment.

[B] Although the test for the effect of all of the body system 
categories combined was not significant, the category for all 
respiratory disorders was significant at the 95-percent confidence 
level for this sample.

[C] Although the test for all of the education categories combined was 
not significant, the category for less than 9 years of education was 
significant at the 95-percent confidence level for this sample.

[D] Although the test for all of the education categories combined was 
not significant, the category for between 12 and 16 years of education 
was significant at the 95-percent confidence level for this sample.

[End of table]

To further explore the extent of the differences we identified in the 
regression analysis, we conducted a series of statistical tests of 
cross tabulations. The results of these tests confirm that--with 
respect to the claimant's body system, age, sex, and race--the 
responders did not differ significantly from the nonresponders. The 
results also indicate that the statistically significant differences 
between responders and nonresponders in allowances and several 
administrative variables were not large in magnitude. Table 5 shows 
that responders differed from nonresponders with respect to 
statistically significant administrative factors from table 3 by 0 to 4 
percentage points.

Table 5: Tabulations of Statistically Significant Administrative 
Factors (from Table 4) for Responders and Nonresponders:

Variable: Attorney representation; Percent of responders in this 
category: 70; Percent of nonresponders in this category: 73.

Variable: Nonattorney representation; Percent of responders in this 
category: 11; Percent of nonresponders in this category: 11.

Variable: Medical expert at hearing; Percent of responders in this 
category: 15; Percent of nonresponders in this category: 16.

Variable: Consultative examination requested; Percent of responders in 
this category: 70; Percent of nonresponders in this category: 73.

Year of decision: 

Year of decision: 1997; Percent of responders in this category: 8; 
Percent of nonresponders in this category: 8.

Year of decision: 1998; Percent of responders in this category: 36; 
Percent of nonresponders in this category: 33.

Year of decision: 1999; Percent of responders in this category: 33; 
Percent of nonresponders in this category: 33.

Year of decision: 2000; Percent of responders in this category: 22; 
Percent of nonresponders in this category: 26.

Region: 

Region: 1. Boston; Percent of responders in this category: 12; 
Percent of nonresponders in this category: 10.

Region: 2. New York; Percent of responders in this category: 10; 
Percent of nonresponders in this category: 10.

Region: 3. Philadelphia; Percent of responders in this category: 10; 
Percent of nonresponders in this category: 10.

Region: 4. Atlanta; Percent of responders in this category: 9; 
Percent of nonresponders in this category: 10.

Region: 5. Chicago; Percent of responders in this category: 11; 
Percent of nonresponders in this category: 10.

Region: 6. Dallas; Percent of responders in this category: 10; 
Percent of nonresponders in this category: 10.

Region: 7. Kansas; Percent of responders in this category: 11; 
Percent of nonresponders in this category: 10.

Region: 8. Denver; Percent of responders in this category: 10; 
Percent of nonresponders in this category: 10.

Region: 9. San Francisco; Percent of responders in this category: 9; 
Percent of nonresponders in this category: 10.

Region: 10. Seattle; Percent of responders in this category: 9; 
Percent of nonresponders in this category: 11.

Source: GAO analysis of 831 and CCS data.

[End of table]

When we repeated the above analysis for subgroups of the sample--
African-American claimants, non-African-American claimants, claimants 
who were allowed benefits, and claimants who were denied benefits--our 
findings were generally consistent across most subgroups. That is, when 
we compared responders and nonresponders who were African-American, 
non-African-American, and who were allowed benefits, we found virtually 
no differences in demographic, medical, and vocational characteristics, 
and only small differences in administrative characteristics.

However, among the sample of claimants who were denied benefits, we 
found a substantial difference in the rates of attorney representation 
among responders and nonresponders. Specifically, 59 percent of 
responders who were denied benefits were represented by attorneys and 
67 percent of nonresponders who were denied benefits were represented 
by attorneys. This means that claimants who were denied benefits and 
had attorneys are underrepresented in the sample. Such under-
representation could result in inflated estimates of the effect of 
attorney representation on allowances. Further analysis of denied 
responders and nonresponders by race did not reveal variations in the 
differences in attorney representation between responders and 
nonresponders by race. (See below for our further analysis of this 
effect by race.) Therefore, we are confident that, even though denied 
claimants with attorneys are under-represented overall, our finding 
indicating that the effect of attorney representation is greater for 
African-American claimants than for white claimants is valid.

Ultimately, the small differences we found between responders and 
nonresponders on only administrative factors, and the similarity of the 
differences in responders and nonresponders for African-Americans and 
whites, makes us reasonably confident that our estimates of the effects 
of the factors on ALJ decisions are not severely biased by nonresponse. 
At the same time, the statistical significance of the associations 
between nonresponse and a number of administrative characteristics as 
well as the cumulative effect of a number of small differences between 
responders and nonresponders may be nontrivial. [Footnote 64]

Section 3: Weighting and Sampling Errors:

We conducted all of our analyses of the enhanced data using probability 
weights because the enhanced data were based on a stratified sample 
rather than the universe of hearings claimants. The weight for each 
claimant equals the inverse probability of the claimant being selected 
into the sample. To control for the effect of the stratified sampling 
scheme on the estimates, we conducted all of our regression analysis 
using computer software that adjusts the estimates according to the 
weighting scheme.

Because the analysis was based on a sample, the reported estimates have 
sampling errors associated with them.[Footnote 65] Sampling errors for 
the estimates of allowance rates for whites, African-Americans, and 
claimants from other racial/ethnic groups were calculated at the 95-
percent confidence level. This means that in 95 out of 100 chances, the 
actual percentage would fall within the range defined by the estimate, 
plus or minus the sampling error. For example, the estimate that 63 
percent of claims filed by whites were allowed at the hearing level has 
a sampling error of 2 percent. This means that a 95-percent chance 
exists, or we can be 95-percent confident, that the actual percentage 
falls between 61 percent and 65 percent. Similarly, for each variable 
in our logistic regression model, a standard error was computed that 
reflects the precision of the estimated odds ratio. The odds ratio for 
each variable in the logistic regressions was considered to be 
significantly different from 1.0 (1.0 implies no difference in the 
odds) when the 95-percent confidence interval around the estimate of 
the odds ratio did not contain 1.0. For example, the 95-percent 
confidence interval for the variable indicating that a translator was 
present at the hearing was 0.39 to 0.90. This interval did not contain 
1.00 and, therefore, the translator variable is considered 
statistically significant.

Section 4: Statistical Analysis:

To choose the appropriate variables for our model of ALJ decision 
making, we reviewed pertinent literature and consulted with SSA 
officials and outside experts.[Footnote 66] The final model included 
variables that are either measures or approximate measures for (1) 
factors that represent criteria used in decision-making process, (2) 
factors that represent participants in the decision-making process, (3) 
factors that are not part of the decision-making process but may have 
an influence on it, and (4) interaction variables reflecting the 
relationship between factors that are not criteria used in the 
decision-making process.

A number of variables in our model are measures for medical and 
nonmedical criteria used in 4 of the 5 steps of the disability 
decision-making process.[Footnote 67] Specifically, the medical 
factors that we controlled for included type of medical impairment 
(such as disorders of the back and musculoskeletal disorders), the 
degree of impairment severity, alcohol or drug abuse alleged,[Footnote 
68] consultative examination requested, number of impairments, number 
of severe impairments, residual functional capacity, and mental 
residual functional capacity. The nonmedical factors that we controlled 
for included occupational categories (blue collar, white collar, and 
service sector), years employed, occupational skill level, educational 
level, literacy, and age.

We also controlled for factors that represent participants in the 
decision-making process. These variables include whether the claimant 
was represented by an attorney or a nonattorney, such as a relative, 
legal aide, or friend; whether a medical and/or vocational expert 
testified at the hearing; whether a translator attended the hearing, 
and whether the claimant attended the hearing.

Finally, we controlled for factors that are not part of the decision-
making process, but for which we have reason to believe may influence 
the disability decision-making process. These variables include the 
claimant's claim type,[Footnote 69] the year of the hearing decision, 
and the SSA region.[Footnote 70] Other factors that we controlled for 
include demographic factors such as sex, race, and earnings.[Footnote 
71] Although these factors are not part of the ALJ decision-making 
process,[Footnote 72] we included these variables in our analysis to 
find out whether they are related to ALJ allowance decisions.

After estimating our initial model, we found several variables that did 
not represent criteria but that had a statistically significant 
influence on ALJ decisions. To investigate whether the effects of these 
variables on ALJ decisions differed by the claimant's race, we 
incorporated interaction terms into our model and tested their 
significance, both simultaneously and sequentially. Specifically, to 
test whether racial groups are treated differently when they are 
represented by attorneys, we included an interaction term between race 
and attorney representation. Similarly, we included an interaction term 
to test whether racial groups are treated differently when they are 
represented by persons other than attorneys. We also included 
interaction terms between race and the following variables: sex, 
earnings, translator, year of the decision,[Footnote 73] and region.

Logistic Regression:

We used logistic regression to estimate the model--an appropriate 
technique when the dependent variable is binary, or has two categories, 
such as benefits being allowed or denied.

On the basis of our initial analyses, we found that the interaction 
term for race and attorney representation was the only statistically 
significant interaction term in the model. We removed the remaining 
insignificant interaction terms from the model because removing them 
had little effect on our estimates of the variables left in the model. 
We did not, however, remove insignificant variables that were not 
interaction terms from our models since our primary objective was to 
estimate the effect of race "net" of other factors we believed could 
potentially influence the allowance decision, regardless of how small 
or statistically insignificant they were.

The results of two of our models--our baseline model and our final 
model containing the significant interaction term--are presented in 
table 6. The first numerical column in table 6 presents the percentage 
of claimants within each variable category. The second and third 
columns present odds ratios that are estimated for each variable in our 
baseline and final models, respectively.[Footnote 74] The 
interpretation of the odds ratio for a particular variable depends on 
whether the variable is a dummy variable or a categorical variable. For 
dummy variables, a statistically significant odds ratio that is 
greater/less than 1.00 indicates that claimants with that 
characteristic are more/less likely to be allowed than claimants 
without it. For categorical variables, a statistically significant odds 
ratio that is greater/less than 1.00 indicates that claimants in that 
category are more/less likely to be allowed than the claimants in the 
comparison category.[Footnote 75]

Table 6: Results of Baseline and Final Models of ALJ Allowance 
Decisions:

Categories for explanatory variables: Factors that represent criteria 
in the decision-making process: 

Categories for explanatory variables: Medical criteria: Impairments 
(dummy variables): 

Explanatory variables: Impairments (dummy variables): Disorders of the 
back; Weighted percent of claimants in this category: 31%; Predicted 
odds ratio for baseline model: 0.83; Predicted odds ratio for final 
model: 0.83.

Explanatory variables: Osteoarthritis and allied disorders; Weighted 
percent of claimants in this category: 10%; Predicted odds ratio for 
baseline model: 0.84; Predicted odds ratio for final model: 0.84.

Explanatory variables: Other musculoskeletal disorders; Weighted 
percent of claimants in this category: 18%; Predicted odds ratio for 
baseline model: 0.63**; Predicted odds ratio for final model: 0.64**.

Explanatory variables: Mental retardation; Weighted percent of 
claimants in this category: 1%; Predicted odds ratio for baseline 
model: 0.83; Predicted odds ratio for final model: 0.80.

Explanatory variables: Mood disorders; Weighted percent of claimants 
in this category: 24%; Predicted odds ratio for baseline model: 0.92; 
Predicted odds ratio for final model: 0.92.

Explanatory variables: Schizophrenia; Weighted percent of claimants in 
this category: 2%; Predicted odds ratio for baseline model: 0.97; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Other mental disorders; Weighted percent of 
claimants in this category: 17%; Predicted odds ratio for baseline 
model: 0.59**; Predicted odds ratio for final model: 0.59**.

Explanatory variables: Diabetes; Weighted percent of claimants in this 
category: 9%; Predicted odds ratio for baseline model: 1.16; Predicted 
odds ratio for final model: 1.14.

Explanatory variables: Other endocrine disorders; Weighted percent of 
claimants in this category: 4%; Predicted odds ratio for baseline 
model: 1.13; Predicted odds ratio for final model: 1.11.

Explanatory variables: Ischemic heart; Weighted percent of claimants 
in this category: 4%; Predicted odds ratio for baseline model: 1.17; 
Predicted odds ratio for final model: 1.17.

Explanatory variables: Hypertension; Weighted percent of claimants in 
this category: 5%; Predicted odds ratio for baseline model: 0.58**; 
Predicted odds ratio for final model: 0.57**.

Explanatory variables: Other cardiovascular disorders; Weighted 
percent of claimants in this category: 4%; Predicted odds ratio for 
baseline model: 0.92; Predicted odds ratio for final model: 0.93.

Explanatory variables: Neurological disorders; Weighted percent of 
claimants in this category: 14%; Predicted odds ratio for baseline 
model: 1.11; Predicted odds ratio for final model: 1.11.

Explanatory variables: Respiratory disorders; Weighted percent of 
claimants in this category: 7%; Predicted odds ratio for baseline 
model: 0.93; Predicted odds ratio for final model: 0.93.

Explanatory variables: Neoplasms; Weighted percent of claimants in 
this category: 2%; Predicted odds ratio for baseline model: 2.94**; 
Predicted odds ratio for final model: 2.85**.

Explanatory variables: Other disorders; Weighted percent of claimants 
in this category: 17%; Predicted odds ratio for baseline model: 1.39*; 
Predicted odds ratio for final model: 1.39*.

Categories for explanatory variables: Medical criteria: Severity of 
impairment (categorical variable): 

Explanatory variables: Not severe; Weighted percent of claimants in 
this category: 11%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Moderate; Weighted percent of claimants in this 
category: 55%; Predicted odds ratio for baseline model: 1.30; 
Predicted odds ratio for final model: 1.26.

Explanatory variables: Moderately severe; Weighted percent of 
claimants in this category: 20%; Predicted odds ratio for baseline 
model: 2.52**; Predicted odds ratio for final model: 2.46**.

Explanatory variables: Meets listing; Weighted percent of claimants in 
this category: 11%; Predicted odds ratio for baseline model: 49.31**; 
Predicted odds ratio for final model: 48.97**.

Explanatory variables: Insufficient medical evidence; Weighted percent 
of claimants in this category: 3%; Predicted odds ratio for baseline 
model: 3.71**; Predicted odds ratio for final model: 3.65**.

Categories for explanatory variables: Medical criteria: Medical 
criteria: Drug abuse (dummy variable): 

Explanatory variables: Alcohol or drug abuse; Weighted percent of 
claimants in this category: 1%; Predicted odds ratio for baseline 
model: 0.62; Predicted odds ratio for final model: 0.62.

Categories for explanatory variables: Medical criteria: Source of 
medical care (dummy variable): 

Explanatory variables: Consultative examination requested; Weighted 
percent of claimants in this category: 15%; Predicted odds ratio for 
baseline model: 1.07; Predicted odds ratio for final model: 1.06.

Categories for explanatory variables: Medical criteria: Number of 
impairments (categorical variable): 

Explanatory variables: 1-2 impairments; Weighted percent of claimants 
in this category: 36%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: 3-4 impairments; Weighted percent of claimants 
in this category: 39%; Predicted odds ratio for baseline model: 
1.49**; Predicted odds ratio for final model: 1.49**.

Explanatory variables: 5 or more impairments; Weighted percent of 
claimants in this category: 25%; Predicted odds ratio for baseline 
model: 2.08**; Predicted odds ratio for final model: 2.08**.

Categories for explanatory variables: Medical criteria: Number of 
severe impairments categorical variable): 

Explanatory variables: No severe impairments; Weighted percent of 
claimants in this category: 14%; Predicted odds ratio for baseline 
model: 1.00; Predicted odds ratio for final model: 1.00.

Explanatory variables: 1 severe impairment; Weighted percent of 
claimants in this category: 47%; Predicted odds ratio for baseline 
model: 1.77*; Predicted odds ratio for final model: 1.81*.

Explanatory variables: 2 severe impairments; Weighted percent of 
claimants in this category: 26%; Predicted odds ratio for baseline 
model: 2.33**; Predicted odds ratio for final model: 2.40**.

Explanatory variables: 3 or 4 severe impairments; Weighted percent of 
claimants in this category: 13%; Predicted odds ratio for baseline 
model: 2.36**; Predicted odds ratio for final model: 2.43**.

Categories for explanatory variables: Medical criteria: Residual 
functional capacity (categorical variable): 

Explanatory variables: Heavy or medium; Weighted percent of claimants 
in this category: 17%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Light (nonexertional restrictions); Weighted 
percent of claimants in this category: 26%; Predicted odds ratio for 
baseline model: 1.89**; Predicted odds ratio for final model: 1.91**.

Explanatory variables: Light (exertional restrictions); Weighted 
percent of claimants in this category: 7%; Predicted odds ratio for 
baseline model: 3.53**; Predicted odds ratio for final model: 3.49**.

Explanatory variables: Sedentary; Weighted percent of claimants in 
this category: 9%; Predicted odds ratio for baseline model: 2.42**; 
Predicted odds ratio for final model: 2.42**.

Explanatory variables: Less than sedentary; Weighted percent of 
claimants in this category: 8%; Predicted odds ratio for baseline 
model: 13.69**; Predicted odds ratio for final model: 13.74**.

Explanatory variables: Not applicable (mental RFC or not severe); 
Weighted percent of claimants in this category: 29%; Predicted odds 
ratio for baseline model: 1.30; Predicted odds ratio for final model: 
1.31.

Explanatory variables: Not determinable; Weighted percent of claimants 
in this category: 4%; Predicted odds ratio for baseline model: 1.80*; 
Predicted odds ratio for final model: 1.81*.

Categories for explanatory variables: Medical criteria: Mental 
residual functional capacity (dummy variable): 

Explanatory variables: Does not meet mental demands of unskilled work; 
Weighted percent of claimants in this category: 8%; Predicted odds 
ratio for baseline model: 30.97**; Predicted odds ratio for final 
model: 31.97**.

Categories for explanatory variables: Nonmedical criteria: 

Categories for explanatory variables: Nonmedical criteria: 
Occupational categories (categorical variable)[A]:  

Explanatory variables: White collar; Weighted percent of claimants in 
this category: 28%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Service sector; Weighted percent of claimants 
in this category: 23%; Predicted odds ratio for baseline model: 0.97; 
Predicted odds ratio for final model: 0.96.

Explanatory variables: Blue collar; Weighted percent of claimants in 
this category: 37%; Predicted odds ratio for baseline model: 1.06; 
Predicted odds ratio for final model: 1.07.

Explanatory variables: No occupation; Weighted percent of claimants in 
this category: 11%; Predicted odds ratio for baseline model: 1.08; 
Predicted odds ratio for final model: 1.09.

Explanatory variables: Occupation not determinable; Weighted percent 
of claimants in this category: 1%; Predicted odds ratio for baseline 
model: 2.52; Predicted odds ratio for final model: 2.60.

Categories for explanatory variables: Nonmedical criteria: Years of 
employment (categorical variable): 

Explanatory variables: Less than 2 years of employment; Weighted 
percent of claimants in this category: 22%; Predicted odds ratio for 
baseline model: 1.00; Predicted odds ratio for final model: 1.00.

Explanatory variables: 2-4 years of employment; Weighted percent of 
claimants in this category: 21%; Predicted odds ratio for baseline 
model: 1.26; Predicted odds ratio for final model: 1.25.

Explanatory variables: 5-9 years of employment; Weighted percent of 
claimants in this category: 22%; Predicted odds ratio for baseline 
model: 1.34; Predicted odds ratio for final model: 1.34.

Explanatory variables: 10 or more years of employment; Weighted 
percent of claimants in this category: 32%; Predicted odds ratio for 
baseline model: 1.56**; Predicted odds ratio for final model: 1.56**.

Explanatory variables: Not determinable; Weighted percent of claimants 
in this category: 3%; Predicted odds ratio for baseline model: 0.73; 
Predicted odds ratio for final model: 0.74.

Categories for explanatory variables: Nonmedical criteria: 
Occupational skill level (categorical variable): 

Explanatory variables: Skilled; Weighted percent of claimants in this 
category: 30%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Semiskilled; Weighted percent of claimants in 
this category: 37%; Predicted odds ratio for baseline model: 0.88; 
Predicted odds ratio for final model: 0.88.

Explanatory variables: Unskilled or has no skill; Weighted percent of 
claimants in this category: 32%; Predicted odds ratio for baseline 
model: 0.84; Predicted odds ratio for final model: 0.85.

Explanatory variables: No skill information available; Weighted 
percent of claimants in this category: 1%; Predicted odds ratio for 
baseline model: 1.07; Predicted odds ratio for final model: 1.04.

Categories for explanatory variables: Nonmedical criteria: Education 
(categorical variable): 

Explanatory variables: Under 6 years of education; Weighted percent of 
claimants in this category: 5%; Predicted odds ratio for baseline 
model: 1.00; Predicted odds ratio for final model: 1.00.

Explanatory variables: 6-11 years of education; Weighted percent of 
claimants in this category: 31%; Predicted odds ratio for baseline 
model: 1.00; Predicted odds ratio for final model: 0.99.

Explanatory variables: 12 years of education; Weighted percent of 
claimants in this category: 45%; Predicted odds ratio for baseline 
model: 0.92; Predicted odds ratio for final model: 0.91.

Explanatory variables: Greater than 12 years of education; Weighted 
percent of claimants in this category: 18%; Predicted odds ratio for 
baseline model: 1.02; Predicted odds ratio for final model: 1.02.

Explanatory variables: Not determinable; Weighted percent of claimants 
in this category: 0.3%; Predicted odds ratio for baseline model: 0.91; 
Predicted odds ratio for final model: 0.95.

Categories for explanatory variables: Nonmedical criteria: Literacy 
(categorical variable): 

Explanatory variables: Literate; Weighted percent of claimants in this 
category: 96%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Illiterate; Weighted percent of claimants in 
this category: 3%; Predicted odds ratio for baseline model: 1.20; 
Predicted odds ratio for final model: 1.19.

Explanatory variables: Literacy not determinable; Weighted percent of 
claimants in this category: 1%; Predicted odds ratio for baseline 
model: 1.25; Predicted odds ratio for final model: 1.22.

Categories for explanatory variables: Age category[B] (categorical 
variable): 

Explanatory variables: 18-24 years old; Weighted percent of claimants 
in this category: 2%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: 25-44 years old; Weighted percent of claimants 
in this category: 44%; Predicted odds ratio for baseline model: 1.13; 
Predicted odds ratio for final model: 1.14.

Explanatory variables: 45-49 years old; Weighted percent of claimants 
in this category: 21%; Predicted odds ratio for baseline model: 1.28; 
Predicted odds ratio for final model: 1.29.

Explanatory variables: 50-54 years old; Weighted percent of claimants 
in this category: 20%; Predicted odds ratio for baseline model: 
2.28**; Predicted odds ratio for final model: 2.31**.

Explanatory variables: 55 years old or over; Weighted percent of 
claimants in this category: 13%; Predicted odds ratio for baseline 
model: 2.18**; Predicted odds ratio for final model: 2.19**.

Categories for explanatory variables: Factors that represent 
participants in the decision-making process: 

Categories for explanatory variables: Representation (categorical 
variable): 

Explanatory variables: No representation; Weighted percent of claimants in this category: 21%; Predicted odds ratio for baseline model: 1.00; Predicted odds ratio for final model: 1.00.

Explanatory variables: Attorney representation[C]; Weighted percent of claimants in this category: 67%; Predicted odds ratio for baseline model: 3.31**; Predicted odds ratio for final model: 2.93**.

Explanatory variables: Other representation; Weighted percent of claimants in this category: 12%; Predicted odds ratio for baseline model: 2.78**; Predicted odds ratio for final model: 2.75**.

Categories for explanatory variables: Other hearing participants (dummy variables); 

Explanatory variables: Medical expert; Weighted percent of claimants in this category: 13%; Predicted odds ratio for baseline model: 1.01; Predicted odds ratio for final model: 1.00.

Explanatory variables: Vocational expert; Weighted percent of claimants in this category: 47%; Predicted odds ratio for baseline model: 0.41**; Predicted odds ratio for final model: 0.41**.

Explanatory variables: Translator; Weighted percent of claimants in this category: 4%; Predicted odds ratio for baseline model: 0.59*; Predicted odds ratio for final model: 0.59*.

Explanatory variables: Claimant present at hearing; Weighted percent of claimants in this category: 99%; Predicted odds ratio for baseline model: 2.51**; Predicted odds ratio for final model: 2.55**.

Categories for explanatory variables: Factors that are not part of the 
decision-making process: 

Categories for explanatory variables: Sex (dummy variable): 


Explanatory variables: Male; Weighted percent of claimants in this 
category: 47%; Predicted odds ratio for baseline model: 0.73**; 
Predicted odds ratio for final model: 0.72**.

Categories for explanatory variables: Race (categorical variable): 

Explanatory variables: White; Weighted percent of claimants in this 
category: 65%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Explanatory variables: Other racial/ethnic groups; Weighted percent of 
claimants in this category: 11%; Predicted odds ratio for baseline 
model: 0.84; Predicted odds ratio for final model: 0.90.

Explanatory variables: African-American[D]; Weighted percent of 
claimants in this category: 24%; Predicted odds ratio for baseline 
model: 0.73**; Predicted odds ratio for final model: 0.50**.

Categories for explanatory variables: Earnings[E] (categorical 
variable): 

Explanatory variables: Less than $5,000 per year; Weighted percent of 
claimants in this category: 49%; Predicted odds ratio for baseline 
model: 1.00; Predicted odds ratio for final model: 1.00.

Explanatory variables: $5,000-$20,000 per year; Weighted percent of 
claimants in this category: 37%; Predicted odds ratio for baseline 
model: 1.96**; Predicted odds ratio for final model: 1.97**.

Explanatory variables: Greater than $20,000; Weighted percent of 
claimants in this category: 14%; Predicted odds ratio for baseline 
model: 3.24**; Predicted odds ratio for final model: 3.22**.

Categories for explanatory variables: Claim type (categorical 
variable):  

Explanatory variables: Supplemental Security Income (SSI); Weighted 
percent of claimants in this category: 27%; Predicted odds ratio for 
baseline model: 1.00; Predicted odds ratio for final model: 1.00.

Explanatory variables: Concurrent claim; Weighted percent of claimants 
in this category: 34%; Predicted odds ratio for baseline model: 1.15; 
Predicted odds ratio for final model: 1.16.

Explanatory variables: Disability Insurance (DI); Weighted percent of 
claimants in this category: 39%; Predicted odds ratio for baseline 
model: 1.12; Predicted odds ratio for final model: 1.13.

Categories for explanatory variables: Year of decision (categorical 
variable):  

Explanatory variables: 1997; Weighted percent of claimants in this 
category: 9%; Predicted odds ratio for baseline model: 1.00; Predicted 
odds ratio for final model: 1.00.

Explanatory variables: 1998; Weighted percent of claimants in this 
category: 39%; Predicted odds ratio for baseline model: 1.22; 
Predicted odds ratio for final model: 1.23.

Explanatory variables: 1999; Weighted percent of claimants in this 
category: 33%; Predicted odds ratio for baseline model: 1.33; 
Predicted odds ratio for final model: 1.33.

Explanatory variables: 2000; Weighted percent of claimants in this 
category: 19%; Predicted odds ratio for baseline model: 1.35; 
Predicted odds ratio for final model: 1.35.

Categories for explanatory variables: Region (categorical variable): 

Explanatory variables: 1. Boston; Weighted percent of claimants in 
this category: 3%; Predicted odds ratio for baseline model: 2.32**; 
Predicted odds ratio for final model: 2.31**.

Explanatory variables: 2. New York; Weighted percent of claimants in 
this category: 12%; Predicted odds ratio for baseline model: 1.10; 
Predicted odds ratio for final model: 1.11.

Explanatory variables: 3. Philadelphia; Weighted percent of claimants 
in this category: 11%; Predicted odds ratio for baseline model: 1.15; 
Predicted odds ratio for final model: 1.15.

Explanatory variables: 4. Atlanta; Weighted percent of claimants in 
this category: 26%; Predicted odds ratio for baseline model: 1.02; 
Predicted odds ratio for final model: 1.02.

Explanatory variables: 5. Chicago; Weighted percent of claimants in 
this category: 14%; Predicted odds ratio for baseline model: 1.08; 
Predicted odds ratio for final model: 1.08.

Explanatory variables: 6. Dallas; Weighted percent of claimants in 
this category: 14%; Predicted odds ratio for baseline model: 0.94; 
Predicted odds ratio for final model: 0.93.

Explanatory variables: 7. Kansas; Weighted percent of claimants in 
this category: 4%; Predicted odds ratio for baseline model: 1.05; 
Predicted odds ratio for final model: 1.05.

Explanatory variables: 8. Denver; Weighted percent of claimants in 
this category: 3%; Predicted odds ratio for baseline model: 1.06; 
Predicted odds ratio for final model: 1.05.

Explanatory variables: 9. San Francisco; Weighted percent of claimants 
in this category: 12%; Predicted odds ratio for baseline model: 0.89; 
Predicted odds ratio for final model: 0.88.

Explanatory variables: 10. Seattle; Weighted percent of claimants in 
this category: 3%; Predicted odds ratio for baseline model: 1.00; 
Predicted odds ratio for final model: 1.00.

Categories for explanatory variables: Interaction variables: 

Categories for explanatory variables: Race/attorney interaction term 
(dummy variables): 

Explanatory variables: Categories for explanatory variables: White 
claimant with attorney; Weighted percent of claimants in this 
category: 46%; Predicted odds ratio for baseline model: N/A; Predicted 
odds ratio for final model: 1.00.

Explanatory variables: Claimant from other racial/ethnic group with 
attorney; Weighted percent of claimants in this category: 6%; 
Predicted odds ratio for baseline model: N/A; Predicted odds ratio for 
final model: .87.

Explanatory variables: African-American claimant with attorney; 
Weighted percent of claimants in this category: 14%; Predicted odds 
ratio for baseline model: N/A; Predicted odds ratio for final model: 
1.76**.

Source: GAO analysis of weighted enhanced data.

Notes: The dependent variable is 1 if the claimant is allowed and 0 if 
the claimant is not allowed. Variables with an odds ratio of 1.00 
represent the excluded category.

* Indicates that the variable is statistically significant at the 95-
percent confidence level.

** Indicates that the variable is statistically significant at the 99-
percent confidence level.

[A] White collar includes professional, technical, or managerial and 
clerical and sales occupations. Service includes service occupations. 
Blue collar includes all other occupations.

[B] Age reflects the age of the claimant on the hearing date.

[C] In the baseline model, the variable for attorney representation 
indicates that, on average, the odds of allowance for claimants with 
attorney representation are 3.3 times higher than those for claimants 
with no representation. In the final model, the variable for attorney 
representation indicates that the odds of allowance for white claimants 
with attorney representation are 2.93 times higher than the odds of 
allowance for white claimants without attorney representation. The 
interpretation of the variable for attorney representation changes in 
the final model because interaction terms between race and attorney 
representation have been included in the final model. Section 4 
explains the interpretation of the interaction terms in greater detail.

[D] In the baseline model, the variable for African-Americans indicates 
that, on average, the odds of allowance for African-Americans are 0.73 
times as high as the odds of allowance for white claimants. In the 
final model, the variable for African-American indicates that the odds 
of allowance for African-Americans without attorneys are 0.50 times as 
high as the odds of allowance for white claimants without attorneys. 
The interpretation of the variable for race changes in the final model 
because interaction terms between race and attorney representation have 
been included in the model. Section 4 explains the interpretation of 
the interaction terms in greater detail.

[E] Earnings are computed as an average of the claimant's earnings for 
the 5 years preceding the hearings level decision date.

[End of table]

Due to the presence of the interaction term between attorney 
representation and race in the final model, one cannot interpret the 
effect of race and attorney representation independent of each other. 
Tables 7, 8, and 9 show how to derive and interpret odds ratios for 
different race and attorney representation subgroups. Table 7 shows 
that, first, the odds of allowance are computed for every race 
subgroup. The odds of allowance are equal to the number of claims 
allowed divided by the number of claims denied for a particular group. 
For example, using the weighted enhanced data, we find that among white 
claimants who were not represented by an attorney, 54,981 were allowed 
and 57,667 were denied. Thus, the odds of being allowed for a white 
claimant that was not represented by an attorney were 0.95 (54,981/
57,667).

The observed odds ratio compares the odds of one group against another. 
The ratio is computed by dividing the odds of allowance of one group by 
the odds of allowance for another group. For example, the odds of 
allowance for African-American and white claimants who were not 
represented were 0.49 and 0.95, respectively. Thus, the observed odds 
ratio of an African-American claimant who was not represented compared 
with a white claimant who was not represented was 0.52 (0.49/0.95). The 
column entitled observed odds ratios presents these ratios for each 
group, as they compare to whites. Both the odds of allowance and the 
observed odds ratio are computed without controlling for other factors 
that influence the allowance decision.

If we control for the other factors that influence the allowance 
decision using regression analysis, we can estimate the odds ratios 
"net" of the influence of other factors--the estimated odds ratio. 
These are presented in the last column of table 7 and come from the 
estimated odds ratios from the final model in table 6. Specifically, 
the last column of table 7 shows that the estimated odds ratio for 
claimants from other racial/ethnic groups who are not represented by an 
attorney is 0.90, which is not significantly different from 1.00. This 
means that after controlling for other factors, the likelihood of 
allowance for claimants from other racial/ethnic groups without an 
attorney is not significantly different from the likelihood of 
allowance for white claimants who are not represented by attorneys (the 
comparison group). In contrast, the odds ratio for African-Americans 
without attorneys is statistically significantly different from 1.00. 
The estimated odds ratio of 0.50 means that the odds of being allowed 
benefits for African-Americans without attorneys are one-half as high 
as the odds of being allowed benefits for whites without attorneys. 
Among claimants who are represented by attorneys, the estimated odds 
ratios for claimants from other racial/ethnic groups and for African-
American claimants are not statistically significantly different from 
1.00 in comparison with white claimants. This means that among 
claimants who are represented by attorneys, the likelihood of allowance 
does not differ significantly by race.

Table 7: Observed and Estimated Odds Ratios by Attorney Representation 
and Race:

Race: Not represented by an attorney: 

Race: White; Allowed: 54,981; Denied: 57,668; Total: 112,649; Odds of 
allowance: 0.95; Observed odds ratios: 1.00; Estimated odds ratios: 
1.00.

Race: Other racial/ethnic background; Allowed: 11,196; Denied: 17,491; 
Total: 28,687; Odds of allowance: 0.64; Observed odds ratios: 0.67; 
Estimated odds ratios: 0.90.

Race: African-American; Allowed: 18,281; Denied: 37,028; Total: 55,309; 
Odds of allowance: 0.49; Observed odds ratios: 0.52; Estimated odds 
ratios: 0.50*.

Race: Represented by an attorney: 

Race: White; Allowed: 191,225; Denied: 86,046; Total: 277,271; Odds of 
allowance: 2.22; Observed odds ratios: 1.00; Estimated odds ratios: 
1.00.

Race: Other racial/ethnic background; Allowed: 23,390; Denied: 15,326; 
Total: 38,716; Odds of allowance: 1.53; Observed odds ratios: 0.69; 
Estimated odds ratios: 0.78.

Race: African-American; Allowed: 50,932; Denied: 34,590; Total: 85,522; 
Odds of allowance: 1.47; Observed odds ratios: 0.66; Estimated odds 
ratios: 0.88.

Source: GAO analysis of weighted enhanced data.

*Statistically different from 1.00.

[End of table]

The last column of table 7 also shows the effect of race among 
claimants who have attorneys. Using the estimated odds ratios from our 
final model, table 8 shows how to compute these odds ratios. They are 
computed by multiplying the odds ratio for the race variable[Footnote 
76] by the odds ratio for the attorney/race interaction variable from 
the final model (reported in table 6). For example, to derive the odds 
ratio for African-American claimants with attorneys compared with white 
claimants with attorneys, we multiplied the odds ratio for African-
American claimants (0.50) by the odds ratio for the interaction 
variable between African-Americans and attorney representation (1.76).

Table 8: Computations for Odds Ratios for Different Racial Groups That 
Are Represented by an Attorney:

Race: White; Odds ratio for race effect: 1.00; Multiplied by: Odds 
ratio for race/attorney interaction term: 1.00; Equals the: Odds ratio 
for claimants with attorneys who are a certain race relative to white 
claimants with attorneys: 1.00.

Race: Other racial/ethnic background; Odds ratio for race effect: 
0.90; Multiplied by: Odds ratio for race/attorney interaction term: 
0.87; Equals the: Odds ratio for claimants with attorneys who are a 
certain race relative to white claimants with attorneys: 0.78[A].

Race: African-American; Odds ratio for race effect: 0.50; Multiplied 
by: Odds ratio for race/attorney interaction term: 1.76; Equals the: 
Odds ratio for claimants with attorneys who are a certain race 
relative to white claimants with attorneys: 0.88[A].

Source: GAO analysis of weighted enhanced data.

[A] Not statistically different from 1.00.

[End of table]

Taken alone, the odds ratio for the interaction variable for African-
Americans with attorney representation (1.76) indicates that the effect 
of attorney representation is bigger for African-American claimants 
than for whites. Specifically, the odds of being allowed benefits for 
African-Americans with attorney representation are 1.76 times higher 
than the odds of being allowed benefits for white claimants with 
attorney representation. However, this does not mean that African-
American claimants with attorneys have higher odds of allowance than 
white claimants with attorneys. Since African-Americans without 
attorneys start with lower odds of allowance (0.50 times) than white 
claimants without attorneys, the additional impact of attorneys for 
African-Americans does not boost their odds of allowance above the odds 
of allowance for white claimants with attorneys.[Footnote 77]

Using the estimated odds ratios from our final model, table 9 shows how 
to compute the effect of attorney representation within a particular 
race group--to compare the odds of allowance between claimants of the 
same race who have attorneys with those that do not have attorneys. For 
example, to derive the odds ratio for African-American claimants with 
attorneys compared with African-American claimants without attorneys, 
we multiply the odds ratio for attorney representation (2.93) by the 
odds ratio for the interaction variable between African-Americans and 
attorney representation (1.76). The product (5.16) means that the odds 
of being allowed benefits for African-American claimants with attorneys 
are 5.16 times higher than the odds of being allowed benefits for 
African-American claimants without attorneys. In contrast, the odds of 
being allowed benefits for white claimants with attorneys are 2.93 
times higher than the odds of being allowed benefits for white 
claimants without attorneys.

Table 9: Computations for Odds Ratios for Claimants of the Same Race 
with and without Attorney Representation:

Race: White; Odds ratio for attorney representation: 2.93; Multiplied 
by: Odds ratio for race/attorney interaction term: 1.00; Equals the: 
Odds ratio for claimants with attorneys who are a certain race 
relative to claimants without attorneys from the same race: 2.93*.

Race: Other racial/ethnic background; Odds ratio for attorney 
representation: 2.93; Multiplied by: Odds ratio for race/attorney 
interaction term: 0.87; Equals the: Odds ratio for claimants with 
attorneys who are a certain race relative to claimants without 
attorneys from the same race: 2.55*.

Race: African-American; Odds ratio for attorney representation: 2.93; 
Multiplied by: Odds ratio for race/attorney interaction term: 1.76; 
Equals the: Odds ratio for claimants with attorneys who are a certain 
race relative to claimants without attorneys from the same race: 
5.16*.

Source: GAO analysis of weighted enhanced data.

*Statistically different from 1.00.

[End of table]

In addition, the average effect of attorney representation is measured 
with the odds ratio for the attorney representation variable in the 
baseline model (before the interaction terms were added). Table 6 shows 
that, on average, the odds of being allowed benefits for claimants with 
attorney representation are 3.3 times higher than the odds of being 
allowed benefits for claimants without attorney representation.

Due to the lower rates of attorney representation among denied 
claimants in our sample, our estimate of the effect of attorney 
representation may be inflated. Specifically, we found that the rate of 
attorney representation was lower among responders who were denied 
benefits (59 percent) than among nonresponders who were denied benefits 
(66 percent).[Footnote 78] This difference in rates of attorney 
representation between denied responders and denied nonresponders could 
result in an overestimation of the effect of attorney representation on 
ALJ decisions. This can be shown with an analysis comparing the 
influence of attorney representation on ALJ decisions for responders 
and nonresponders. Table 10 shows that among the responders, the odds 
of allowance for claimants with and without attorneys were 1.97 and 
0.69, respectively. The observed odds ratio comparing responders with 
attorneys to responders without attorneys is 2.88--which means that, 
the odds of allowance for responders with attorneys were 2.88 times 
higher than the odds of allowance for responders without attorneys. 
Similarly, among the nonresponders, the odds of allowance for claimants 
with and without attorneys were 1.75 and 0.87, respectively. The 
observed odds ratio comparing nonresponders with attorneys to 
nonresponders without attorneys is 1.90. When we compare the size of 
the effect of attorney representation for these two groups--that is, 
2.88 for responders compared with 1.90 for nonresponders--we find that 
the effect of attorney representation is 1.51 times higher among 
responders than among nonresponders. Consequently, we conclude that, by 
analyzing only responders, we are overestimating or inflating the 
effect of attorney representation.

Table 10: Effect of Attorney Representation on ALJ Decisions for 
Responders and Nonresponders:

Attorney representation: Responder: 

Attorney representation: Has attorney; Allowed: 71,259; Denied: 36,092; 
Odds of allowance: 1.97; Observed odds ratio of allowance: 2.88; Ratio 
of odds ratios: 1.51.

Attorney representation: No attorney; Allowed: 17,442; Denied: 25,427; 
Odds of allowance: 0.69; Observed odds ratio of allowance: [Empty]; 
Ratio of odds ratios: [Empty].

Attorney representation: Nonresponder: 

Attorney representation: Has attorney; Allowed: 325,249; Denied: 
196,796; Odds of allowance: 1.65; Observed odds ratio of allowance: 
1.90; Ratio of odds ratios: [Empty].

Attorney representation: No attorney; Allowed: 87,825; Denied: 101,085; 
Odds of allowance: 0.87; Observed odds ratio of allowance: [Empty]; 
Ratio of odds ratios: [Empty].

Source: GAO analysis of weighted CCS data.

[End of table]

A precise estimate of how greatly the size of the effect of attorney 
representation is inflated by nonresponse would require complete 
information about nonresponders, which we lack. Our best estimate 
without more complete information on nonresponders is that the actual 
effect of attorney representation in our sample of responders is higher 
than in the entire sample (including responders and nonresponders), by 
a factor of about 1.4. (See table 11.):

Table 11: Effect of Attorney Representation on ALJ Decisions for 
Responders and the Entire Sample:

Attorney representation: Responder: 

Attorney representation: Has attorney; Allowed: 71,259; Denied: 36,092; 
Odds of allowance: 1.97; Observed odds ratio of allowance: 2.88; Ratio 
of odds ratios: 1.41.

Attorney representation: No attorney; Allowed: 17,442; Denied: 25,427; 
Odds of allowance: 0.69; Observed odds ratio of allowance: [Empty]; 
Ratio of odds ratios: [Empty].

Attorney representation: Entire Sample: 

Attorney representation: Has attorney; Allowed: 396,508; Denied: 
232,888; Odds of allowance: 1.70; Observed odds ratio of allowance: 
2.05; Ratio of odds ratios: [Empty].

Attorney representation: No attorney; Allowed: 105,267; Denied: 
126,512; Odds of allowance: 0.83; Observed odds ratio of allowance: 
[Empty]; Ratio of odds ratios: [Empty].

Source: GAO analysis of weighted CCS data.

[End of table]

In order to determine the extent to which this overestimation affects 
our finding that African-American claimants without attorneys were less 
likely to be allowed than white claimants without attorneys, we 
compared the effect of attorney representation on allowance decisions 
for responders and nonresponders by race. As shown in table 12, among 
African-Americans claimants, the observed odds ratio for responders 
with attorneys versus responders without attorneys is 3.40 (in other 
words, the odds of allowance for responders with attorneys were 3.40 
times higher than the odds of allowance for responders without 
attorneys), whereas the observed odds ratio for nonresponders is 2.06 
(that is, the odds of allowance for nonresponders with attorneys were 
2.06 times higher than the odds of allowance for nonresponders without 
attorneys). The ratio of these two effects is 1.65. In other words, for 
African-American claimants, the effect of attorney representation is 
1.65 times higher for responders than for nonresponders. When we do a 
similar computation for white claimants, we find that the effect of 
attorney representation is 1.60 times higher for responders than for 
nonresponders. The relatively small difference between 1.65 and 1.60 
leads us to conclude that the over-estimation of attorney 
representation does not vary by race.

Table 12: Effect of Attorney Representation on ALJ Decisions for 
Responders and Nonresponders, by Race:

(Continued From Previous Page)

African-American claimants: 

Responder; Attorney representation: Has attorney; Allowed: 16,223; 
Denied: 8,150; Odds of allowance: 1.99; Observed odds ratio of 
allowance: 3.40; Ratio of odds ratios: 1.65.

Attorney representation: Nonresponder: No attorney; Allowed: 
Nonresponder: 3,499; Denied: Nonresponder: 5,973; Odds of allowance: 
Nonresponder: 0.59; Observed odds ratio of allowance: Nonresponder: 
[Empty]; Ratio of odds ratios: Nonresponder: [Empty].

Nonresponder; Attorney representation: Has attorney; Allowed: 75,505; 
Denied: 45,700; Odds of allowance: 1.65; Observed odds ratio of 
allowance: 2.06; Ratio of odds ratios: [Empty].

Attorney representation: No attorney; Allowed: 19,954; Denied: 24,862; 
Odds of allowance: 0.80; Observed odds ratio of allowance: [Empty]; 
Ratio of odds ratios: [Empty].

White claimants: 

Responder; Attorney representation: Has attorney; Allowed: 47,147; 
Denied: 23,648; Odds of allowance: 1.99; Observed odds ratio of 
allowance: 2.92; Ratio of odds ratios: 1.60.

Attorney representation: Nonresponder: No attorney; Allowed: 
Nonresponder: 10,991; Denied: Nonresponder: 16,116; Odds of allowance: 
Nonresponder: 0.68; Observed odds ratio of allowance: Nonresponder: 
[Empty]; Ratio of odds ratios: Nonresponder: [Empty].

Nonresponder; Attorney representation: Has attorney; Allowed: 211,805; 
Denied: 128,031; Odds of allowance: 1.65; Observed odds ratio of 
allowance: 1.83; Ratio of odds ratios: [Empty].

Attorney representation: Attorney representation: No attorney; 
Allowed: Allowed: 55,668; Denied: Denied: 61,478; Odds of allowance: 
Odds of allowance: 0.91; Observed odds ratio of allowance: Observed 
odds ratio of allowance: [Empty]; Ratio of odds ratios: Ratio of odds 
ratios: [Empty].

Source: GAO analysis of weighted CCS data.

[End of table]

Table 13 shows that the over-estimation of attorney representation also 
does not vary by race when we compare responders to the entire sample 
of responders and nonresponders.

Table 13: Effect of Attorney Representation on ALJ Decisions for 
Responders and the Entire Sample by Race:

African-American claimants: 

Responder; Attorney representation: Has attorney; Allowed: 16,223; 
Denied: 8,150; Odds of allowance: 1.99; Observed odds ratio of 
allowance: 3.40; Ratio of odds ratios: 1.52.

Attorney representation: Entire sample: No attorney; Allowed: Entire 
sample: 3,499; Denied: Entire sample: 5,973; Odds of allowance: Entire 
sample: 0.59; Observed odds ratio of allowance: Entire sample: [Empty]; 
Ratio of odds ratios: Entire sample: [Empty].

Entire sample; Attorney representation: Has attorney; Allowed: 91,728; 
Denied: 53,850; Odds of allowance: 1.70; Observed odds ratio of 
allowance: 2.24; Ratio of odds ratios: [Empty].

Attorney representation: No attorney; Allowed: 23,453; Denied: 30,835; 
Odds of allowance: 0.76; Observed odds ratio of allowance: [Empty]; 
Ratio of odds ratios: [Empty].

White claimants: 

Responder; Attorney representation: Has attorney; Allowed: 47,147; 
Denied: 23,648; Odds of allowance: 1.99; Observed odds ratio of 
allowance: 2.92; Ratio of odds ratios: 1.47.

Attorney representation: Entire sample: No attorney; Allowed: Entire 
sample: 10,991; Denied: Entire sample: 16,116; Odds of allowance: 
Entire sample: 0.68; Observed odds ratio of allowance: Entire sample: 
[Empty]; Ratio of odds ratios: Entire sample: [Empty].

Entire sample; Attorney representation: Has attorney; Allowed: 258,952; 
Denied: 151,679; Odds of allowance: 1.71; Observed odds ratio of 
allowance: 1.99; Ratio of odds ratios: [Empty].

Attorney representation: Attorney representation: No attorney; 
Allowed: Allowed: 66,659; Denied: Denied: 77,594; Odds of allowance: 
Odds of allowance: 0.86; Observed odds ratio of allowance: Observed 
odds ratio of allowance: [Empty]; Ratio of odds ratios: Ratio of odds 
ratios: [Empty].

Source: GAO analysis of weighted CCS data.

[End of table]

Based on this analysis, we conclude that (1) our estimates of the 
effect of having an attorney on the likelihood to be allowed may be 
inflated, but (2) our estimates of the relative effects of attorney 
representation by race on the likelihood to be allowed should not be 
biased.

Oaxaca decomposition:

To further test whether differences in allowance rates between African-
American and white claimants are the result of differences in their 
race or in other characteristics, we employed a statistical technique-
-the Oaxaca decomposition--that is commonly used in analyses of 
discrimination.[Footnote 79] The goal of this technique is to separate 
the difference in allowance rates between African-Americans and whites 
into two components: one that results from differences in 
characteristics between African-Americans and whites and the second 
that results from differential treatment by race.

Several steps were taken to develop the results for our final Oaxaca 
decomposition analysis:

* First, we estimated two versions of our baseline model--one with only 
the African-American claimants in the sample and one with only the 
white claimants in the sample. This step provided us with two sets of 
estimated regression coefficients--one set of coefficients for African-
Americans and the other set for whites.

* Second, we applied the estimated coefficients from the model for 
African-Americans to the values of each variable for African-Americans 
to produce a probability of allowance for African-Americans. We did the 
same with the estimated coefficients for whites and the values of each 
variable for whites to produce a probability of allowance for whites. 
These estimated probabilities of allowance are similar to the allowance 
rates for African-Americans and whites based on observed (or actual) 
data; but, because the probabilities are predicted, they deviate 
slightly from the observed allowance rates.

* Third, we used the coefficients from the model of whites and the 
actual values for each variable for African-Americans to produce a new 
probability of allowance. This probability reflects what the 
probability of allowance would have been for African-Americans had they 
been treated the same as whites in the allowance decision.

For our final Oaxaca decomposition analysis, we compared the results of 
the steps above. Specifically, we compared (1) the African-American 
probability of allowance predicted using the African-American model, 
with (2) the African-American probability of allowance predicted using 
the white model, with (3) the white probability of allowance predicted 
using the white model. To the extent that the African-American 
probability of allowance predicted using the white model departs from 
the white probability of allowance predicted using the white model, we 
can conclude that the difference between African-Americans and whites 
can be explained by differences in characteristics. To the extent that 
the African-American probability predicted using the white model 
departs from that predicted using the African-American model, we 
conclude that (1) the two models reflect different treatment of 
African-Americans and whites and (2) the difference between African-
Americans and whites cannot be fully explained by differences in 
characteristics. We performed these analyses on (1) the entire sample 
of claimants, (2) the sample of claimants with attorney representation, 
and (3) the sample of claimants without attorney representation. Table 
14 presents the results of these analyses for each sample.

Table 14: Summary Results of Oaxaca Decomposition:

Entire sample; Predicted allowance rate for: African-Americans (with 
African-American coefficients): 49%; Predicted allowance rate for: 
African-Americans (with white coefficients): 53%; Predicted allowance 
rate for: Whites (with African-American coefficients): 59%; Predicted 
allowance rate for: Whites (with white coefficients): 63%; 
Percentage of explained disparities[A]: 71%; Percentage due to unequal 
treatment and/or factors not controlled for in model: 29%.

Claimants with attorneys; Predicted allowance rate for: African-
Americans (with African-American coefficients): 60%; Predicted 
allowance rate for: African-Americans (with white coefficients): 62%; 
Predicted allowance rate for: Whites (with African-American 
coefficients): 68%; Predicted allowance rate for: Whites (with white 
coefficients): 69%; Percentage of explained disparities[A]: 
78%; Percentage due to unequal treatment and/or factors not controlled 
for in model: 22%.

Claimants without attorneys; Predicted allowance rate for: African-
Americans (with African-American coefficients): 34%; Predicted 
allowance rate for: African-Americans (with white coefficients): 40%; 
Predicted allowance rate for: Whites (with African-American 
coefficients): 43%; Predicted allowance rate for: Whites (with white 
coefficients): 49%; Percentage of explained disparities[A]: 
60%; Percentage due to unequal treatment and/or factors not controlled 
for in model: 40%.

Source: GAO analysis of weighted enhanced data.

[A] The percentage of explained disparities is computed by dividing the 
difference between the predicted allowance rate for whites (with white 
coefficients) and the predicted allowance rates for African-Americans 
(with white coefficients), by the difference between the predicted 
allowance rate for whites (with white coefficients) and the predicted 
allowance rate for African-Americans (with African-American 
coefficients). For example, for the entire sample, the computation is 
(63%-53%/63%-49%)=71%.

[End of table]

The results of the Oaxaca decomposition show that most of the 
difference between African-Americans and whites can be explained by 
differences in their characteristics. Specifically, we found that using 
the entire sample, 71 percent of the difference in predicted allowance 
rates between whites and African-Americans is due to differences in the 
characteristics of African-Americans and whites. The remaining 29 
percent is due to either unequal treatment in the disability decision-
making process or to factors that are not controlled for in the model 
or to some combination of the two.

The results of the two subsamples can be interpreted in the same way as 
the results from the entire sample. Specifically, the results for the 
sample of claimants with attorneys show that 78 percent of the 
difference in predicted allowance rates between whites and African-
Americans is due to differences in characteristics between African-
Americans and whites. The remaining 22 percent is due to either unequal 
treatment in the disability decision-making process or to factors that 
are not controlled for in the model or to some combination of the two. 
In addition, when we use the sample of claimants without attorney 
representation, we find that less of the difference between African-
Americans and whites is explained by differences in characteristics (as 
compared with the entire sample or the sample of claimants with 
attorneys). Specifically, the results show that 60 percent of the 
difference in predicted allowance rates between whites and African-
Americans is due to differences in characteristics. The remaining 40 
percent is due to either unequal treatment or to factors that are not 
controlled for in the model or to some combination of the two. The 
results of this technique buttress the conclusions we draw from our 
final model, that is, among claimants without attorney representation, 
substantial differences between African-Americans and whites cannot be 
explained by differences in other factors.

Section 5: Limitations of Analysis:

Due to inherent limitations with our data and methods, we cannot 
definitively determine whether unexplained differences in allowance 
rates by race are due to unequal treatment during the decision-making 
process.

First, many of the variables we used in our analyses had some degree of 
measurement error, and this can be a potentially serious problem when 
continuous variables are redefined and collapsed into categorical 
variables. For example, the severity of the claimant's impairments 
ranges along a very broad continuum. However, the data available for 
these analyses rank the severity of claimant's impairments and place 
them in a limited number of categories. Within a particular category, 
however, there may be subtle and important variations in severity that 
are completely unmeasured. Second, some variables were measured 
imprecisely. For example, the earnings variable was derived using the 
average of employment income earned by the claimant during the 5 years 
previous to the hearings decision. This earnings variable did not 
include investment income or earnings from other family members. Hence, 
it does not necessarily reflect the claimant's total household income, 
data that were not available.

Third, several factors, for which data were not available, could not be 
controlled for in our model. For example, we were unable to control for 
the extent to which claimants may differ in their access to and quality 
of healthcare. Differences in access to and quality of healthcare are 
reflected in, and thus related to, the quality of medical evidence in 
the claimant's file--an important component of the decision-making 
process. Credibility is also a key factor in the ALJ disability 
decision-making process. However, we did not include a proxy for 
credibility in our model because we did not have an independent 
assessment of the claimant's credibility.[Footnote 80]

Finally, the choice of whether or not to appeal has a theoretical 
potential to affect the analysis. However, due to a lack of data at the 
initial level, we were unable to estimate, or control for, the 
claimant's likelihood of appealing to the ALJ level.

Improving the precision of some of the variables that were included in 
our model and including additional variables to control for other 
factors might have improved our ability to account for the variation in 
ALJ decisions. Although these limitations could have resulted in biased 
estimates of our coefficients, the enhanced data we used were the best 
data available for examining potential racial disparities in ALJ 
disability decision making.

[End of section]

Appendix II: SSA's Five-Step Sequential Evaluation Process for 
Determining Disability:

SSA's regulations provide for disability evaluation under a procedure 
known as the "sequential evaluation process." For adult claimants, this 
process requires a sequential review of the claimant's current work 
activity, the severity of his or her impairment(s), and if necessary, 
the claimant's residual functional capacity, his or her past work, and 
his or her age, education, and work experience.[Footnote 81]

Step 1. Is the claimant working? If the claimant is working and the 
claimant's average monthly countable earnings are above the substantial 
gainful activity (SGA) level,[Footnote 82] SSA will find the claimant 
not disabled, regardless of the claimant's medical condition, age, 
education, and work experience, and deny the claim. If the claimant's 
average monthly countable earnings are at or less than the SGA level, 
SSA will look at the claimant's medical condition (step 2).

Step 2. Is the claimant's condition "severe?" The claimant's impairment 
must significantly limit his or her physical or mental ability to do 
basic work activities, such as walking, sitting, seeing, and 
remembering. If it does not, SSA will deny the claim, regardless of the 
claimant's age, education, and work experience. If it does, SSA will 
look further at the claimant's medical condition (step 3).

Step 3. Is the claimant's medical condition in the list of "disabling" 
impairments? If the claimant has an impairment that meets the duration 
requirement and is on SSA's listing of impairments,[Footnote 83] the 
claimant is considered "disabled" without considering age, education, 
and work experience. If the medical condition is not on the list, SSA 
considers whether the condition is of equal severity to an impairment 
on SSA's list. If so, the claim is approved. If not, SSA considers 
additional factors (step 4).

Step 4. Can the claimant perform past relevant work? If the medical 
condition is severe, but not at the same or equal severity as an 
impairment on SSA's list, then SSA will review the claimant's residual 
functional capacity, and the physical and mental demands of work 
performed in the past. If the claimant can do work performed 
previously, SSA will deny the claim. If not, SSA considers other 
factors (step 5).

Step 5. Can the claimant perform other types of work? If the claimant 
cannot perform past work, SSA will consider the claimant's residual 
function capacity, age, education, and past work experience to 
determine whether he or she can perform other work that is available in 
the national economy. If the claimant cannot perform other work, SSA 
will approve the claim. If the claimant can perform other work, SSA 
will deny the claim.

[End of section]

Appendix III: Comments from the Social Security Administration:

This flowchart is printed on pages 70-72.

SOCIAL SECURITY:

The Commissioner:

October 14, 2003:

Mr. Robert E. Robertson Director, Education, Workforce, and Income 
Security Issues U.S. General Accounting Office Room 5T57:

441 G Street, NW Washington, D.C. 20548:

Dear Mr. Robertson:

Thank you for the opportunity to review the draft report, "SSA 
Disability Decision Making: Additional Steps Needed to Ensure Accuracy 
and Fairness of Decisions at the Hearings Level." The draft report is 
useful and timely and fits into our overall goals of fairness and 
accuracy. On September 25, 2003, I testified before the House Ways and 
Means Subcommittee on Social Security and presented my approach to 
improve the disability determination process. The proposed process 
would shorten decision times, pay benefits much earlier to people who 
are obviously disabled and test new incentives for people with 
disabilities who wish to remain in, or return to, the workforce. I have 
enclosed a copy of my testimony as well as flow charts depicting the 
approach I described.

I agree with the recommendations in the report but intend to go 
further. As part of our overall plan to improve the disability 
determination process, we intend to look at all factors that may 
produce adverse impacts based on race, ethnicity, national origin or 
gender. And, we plan to introduce an in-line quality review process 
that will, among other things, help us to assess those impacts at all 
stages in the process. Finally, a few months ago, we convened an Agency 
workgroup tasked with developing recommendations on how we can collect 
meaningful data on race and ethnicity so we will have the information 
we need to analyze any adverse effects of our program policies and 
rules.

If you have any questions, please have your staff contact Candace 
Skurnik, Director, Audit Management Liaison Staff at (410) 965-4636.

Sincerely,

Jo Anne B. Barnhart:

Signed by Jo Anne B. Barnhart: 

Enclosures:

SOCIAL SECURITY ADMINISTRATION	BALTIMORE MD 21235-0001:

Social Security Testimony Before Congress:

Testimony: 

Mr. Chairman,

I want to thank you and the entire Subcommittee for your continuing 
support for the people and programs of the Social Security 
Administration, and most especially for your interest in and commitment 
to improving the disability process. I also want to thank you for 
holding this hearing which provides the opportunity for me to describe 
my approach for improving the Social Security and Supplemental Security 
Income disability process. Our disability programs are critically 
important in the lives of almost 13 million of Americans. Claimants and 
their families expect and deserve fair, accurate, consistent, and 
timely decisions.

EDIB is a major agency initiative that will move all components 
involved in disability claims adjudication and review to an electronic 
business process through the use of an electronic disability folder. 
Implementation of an electronic disability folder is essential for 
process improvements. Therefore, structurally, my long-term strategy 
for achieving process improvements is predicated on successful 
implementation of our electronic disability system.

In designing my approach to improve the overall disability 
determination process, I was guided by three questions the President 
posed during our first meeting to discuss the disability programs.

* Why does it take so long to make a disability decision?

* Why can't people who are obviously disabled get a decision 
immediately?

* Why would anyone want to go back to work after going through such a 
long process to receive benefits?

I realized that designing an approach to fully address the central and 
important issues raised by the President required a focus on two over-
arching operational goals: (1) to make the right decision as early in 
the process as 
possible; and (2) to foster return to work at all stages of the 
process. I also decided to focus on improvements that could be 
effectuated by regulation and to ensure that no SSA employee would be 
adversely affected by my approach. My reference to SSA employees 
includes state Disability Determination Service (DDS) employees and 
Administrative Law Judges (ALJs).

As I developed my approach for improvement, I met with and talked to 
many people --SSA employees and other interested organizations , 
individually and in small and large groups --to listen to their 
concerns about the current process at both the initial and appeals 
levels and their recommendations for improvement. I became convinced 
that improvements must be looked at from a system-wide perspective and, 
to be successful, perspectives from all parts of the system must be 
considered. I believe an open and collaborative process is critically 
important to the development of disability process improvements.

To that end, members of my staff and I visited our regional offices, 
field offices, hearing offices, and State Disability Determination 
Services, and private disability insurers to identify and discuss 
possible improvements to the current process.

Finally, a number of organizations provided written recommendations for 
changing the disability process. Most recently, the Social Security 
Advisory Board issued a report prepared by outside experts making 
recommendations for process change. My approach for changing the 
disability process was developed after a careful review of these 
discussions and written recommendations. As we move ahead, I look 
forward to working within the Administration and with Congress, as well 
as interested organizations and advocacy groups. I would now like to 
highlight some of the major and recurring recommendations made by these 
various parties.

The need for additional resources to eliminate the backlog and reduce 
the lengthy processing time was a common theme. This important issue is 
being addressed through my Service Delivery Plan, starting with the 
President's FY 2004 budget submission which is currently before 
Congress. Another important and often heard concern was the necessity 
of improving the quality of the administrative record.

DDSs expressed concerns about receiving incomplete applications from 
the field office; ALJs expressed concerns about the quality of the 
adjudicated record they receive and emphasized the extensive pre-
hearing work required to thoroughly and adequately present the case for 
their consideration.

In addition, the number of remands by the Appeals Council and the 
Federal Courts make clear the need for fully documenting the 
administrative hearing record.

Applying policy consistently in terms of: 1) the DDS decision and ALJ 
decision; 2) variations among state DDSs; and 3) variations among 
individual ALJs --was of great concern. Concerns related to the 
effectiveness of the existing regional quality control reviews and ALJ 
peer review were also expressed. Staff from the Judicial Conference 
expressed strong concern that the process assure quality prior to the 
appeal of cases to the Federal Courts.

ALJs and claimant advocacy and claimant representative organizations 
strongly recommended retaining the de novo hearing before an ALJ. 
Department of Justice litigators and the Judicial Conference stressed 
the importance of timely case retrieval, transcription, and 
transmission. Early screening and analysis of cases to make expedited 
decisions for clear cases of disability was emphasized time and again 
as was the need to remove barriers to returning to work.

My approach for disability process improvement is designed to address 
these concerns. It incorporates some of the significant features of the 
current disability process. For example, initial claims for disability 
will continue to be handled by SSA's field offices. The State 
Disability Determination Services will continue to adjudicate claims 
for benefits, and Administrative Law Judges will continue to conduct 
hearings and issue decisions. My approach envisions some significant 
differences.

I intend to propose a quick decision step at the very earliest stages 
of the claims process for people who are obviously disabled. Cases will 
be sorted based on disabling conditions for early identification and 
expedited action. Examples of such claimants would be those with ALS, 
aggressive cancers, and end-stage renal disease. Once a disability 
claim has been 
completed at an SSA field office, these Quick Decision claims would be 
adjudicated in Regional Expert Review Units across the country, without 
going to a State Disability Determination Service.

This approach would have the two-fold benefit of allowing the claimant 
to receive a decision as soon as possible, and allowing the State DDSs 
to devote resources to more complex claims.

Centralized medical expertise within the Regional Expert Review Units 
would be available to disability decision makers at all levels, 
including the DDSs and the Office of Hearings and Appeals (OHA). These 
units would be organized around clinical specialties such as 
musculoskeletal, neurological, cardiac, and psychiatric. Most of these 
units would be established in SSA's regional offices.

The initial claims not adjudicated through the Quick Decision process 
would be decided by the DDSs. However, I would also propose some 
changes in the initial claims process that would require changes in the 
way DDSs are operating. An in-line quality review process managed by 
the DDSs and a centralized quality control unit would replace the 
current SSA quality control system. I believe a shift to inline quality 
review would provide greater opportunities for identifying problem 
areas and implementing corrective actions and related training.

The Disability Prototype would be terminated and the DDS 
Reconsideration step would be eliminated. Medical expertise would be 
provided to the DDSs by the Regional Expert Review units that I 
described earlier.

State DDS examiners would be required to fully document and explain the 
basis for their determination. More complete documentation should 
result in more accurate initial decisions. The increased time required 
to accomplish this would be supported by redirecting DDS resources 
freed up by the Quick Decision cases being handled by the expert units, 
the elimination of the Reconsideration step, and the shift in medical 
expertise responsibilities to the regional units.

A Reviewing Official (RO) position would be created to evaluate claims 
at the next stage of the process. If a claimant files a request for 
review of the DDS 
determination, the claim would be reviewed by an SSA Reviewing 
Official. The RO, who would be an attorney, would be authorized to 
issue an allowance decision or to concur in the DDS denial of the 
claim.	If the claim is not allowed by the RO, the RO will prepare either 
a Recommended Disallowance or a Pre-Hearing Report. A Recommended 
Disallowance would be prepared if the RO believes that the evidence in 
the record shows that the claimant is ineligible for benefits.	It would 
set forth in detail the reasons the claim should be denied.

A Pre-Hearing Report would be prepared if the RO believes that the 
evidence in the record is insufficient to show that the claimant is 
eligible for benefits but also fails to show that the claimant is 
ineligible for benefits. The report would outline the evidence needed 
to fully support the claim. Disparity in decisions at the DDS level has 
been a long-standing issue and the SSA Reviewing Official and creation 
of Regional Expert Medical Units would promote consistency of decisions 
at an earlier stage in the process.

If requested by a claimant whose claim has been denied by an RO, an ALJ 
would conduct a de novo administrative hearing. The record would be 
closed following the ALJ hearing. If, following the conclusion of the 
hearing, the ALJ determines that a claim accompanied by a Recommended 
Disallowance should be allowed, the ALJ would describe in detail in the 
written opinion the basis for rejecting the RO's Recommended 
Disallowance.

If, following the conclusion of the hearing, the ALJ determines that a 
claim accompanied by a Pre-Hearing Report should be allowed, the ALJ 
would describe the evidence gathered during the hearing that responds 
to the description of the evidence needed to successfully support the 
claim contained in the Pre-hearing Report.

Because of the consistent finding that the Appeals Council review adds 
processing time and generally supports the ALJ decision, the Appeals 
Council stage of the current process would be eliminated.

Quality control for disability claims would be centralized with end-of-
line reviews and ALJ oversight. If an ALJ decision is not reviewed by 
the centralized quality control staff, the decision of the ALJ will 
become a final agency action. If the centralized quality control review 
disagrees with an allowance or disallowance 
determination made by an ALJ, the claim would be referred to an 
Oversight Panel for determination of the claim. The Oversight Panel 
would consist of two Administrative Law Judges and one Administrative 
Appeals Judge.

If the Oversight Panel affirms the ALJ's decision, it becomes the final 
agency action. If the Panel reverses the ALJ's decision, the oversight 
Panel decision becomes the final agency action. As is currently the 
case, claimants would be able to appeal any final agency action to a 
Federal Court.

At the same time these changes were being implemented to improve the 
process, we plan to conduct several demonstration projects aimed at 
helping people with disabilities return to work. These projects would 
support the President's New Freedom Initiative and provide work 
incentives and opportunities earlier in the process.

I believe these changes and demonstrations will address the major 
concerns I highlighted earlier. I also believe they offer a number of 
important improvements:

* People who are obviously disabled will receive quick decisions.

* Adjudicative accountability will be reinforced at every step in the 
process.

* Processing time will be reduced by at least 25%. . Decisional 
consistency and accuracy will be increased.

* Barriers for those who can and want to work would be removed.

Describing my approach for improving the process is the first step of 
what I believe must be --and will work to make --a collaborative 
process. I will work within the Administration, with Congress, the 
State Disability Determination Services and interested organizations 
and advocacy groups before putting pen to paper to write regulations. I 
said earlier, and I say again that to be successful, perspectives from 
all parts of the system must be considered.

Later today, I will conduct a briefing for Congressional staff of the 
Ways and Means and Senate Finance Committees. I will also brief SSA and 
DDS management. In addition, next week I will provide a 
video tape of the management briefing describing my approach for 
improvement to all SSA regional, field, and hearing offices, State 
Disability Determination Services, and headquarters and regional office 
employees involved in the disability program.

Tomorrow, I will be conducting briefings for representatives of SSA 
employee unions and interested organizations and advocacy groups, and I 
will schedule meetings to provide an opportunity for those 
representatives to express their views and provide assistance in 
working through details, as the final package of process improvements 
is fully developed.

I believe that if we work together, we will create a disability system 
that responds to the challenge inherent in the President's questions. 
We will look beyond the status quo to the possibility of what can be. 
We will achieve our ultimate goal of providing accurate, timely service 
for the American people.

Note: A flowchart describing the process is available in pdf format.

A New Approach to SSA Disability Determination:

[See PDF for image]

[End of figure]

Reviewing Official-ALJ Work Flow: 

[See PDF for image]

[End of figure]

Work Opportunity Demonstrations: 

[See PDF for image]

[End of figure]

[End of section]

Appendix IV: GAO Contacts and Acknowledgments:

GAO Contacts:

Robert E. Robertson, (202) 512-7215 Carol Dawn Petersen, (202) 512-
7215:

GAO Acknowledgments:

In addition to those named above, the following GAO staff made 
significant contributions to this report: Mark de la Rosa, Erin 
Godtland, Michele Grgich, Stephen S. Langley III, and Ann T. Walker, 
Education, Workforce, and Income Security Issues; Doug Sloane, Applied 
Research and Methods. Also contributing to the report were: Gene 
Kuehneman and Jill Yost, Education, Workforce, and Income Security 
Issues; Jessica Botsford, Richard Burkard, David Plocher, and Dayna 
Shah, General Counsel; Wendy Turenne and Shana Wallace, Applied 
Research and Methods; Scott Farrow, Chief Economist; Robert Parker, 
Chief Statistician; Ron Stroman, Office of Opportunity and Inclusion.

Other Acknowledgments:

We contracted with the following individuals for technical assistance:

* Judith Hellerstein, Associate Professor of Economics, Department of 
Economics, University of Maryland.

* Joseph Kadane, University Professor and Professor of Statistics and 
Social Sciences, Department of Statistics, Carnegie-Mellon University.

* Brent Kreider, Associate Professor of Economics, Department of 
Economics, Iowa State University.

* Kajal Lahiri, Professor of Economics, and Professor of Health Policy, 
Management and Behavior, Department of Economics, University at Albany, 
State University of New York.

FOOTNOTES

[1] In 1992, GAO reported that DI allowance rates between 1961 and 1985 
and SSI allowance rates between 1971 and 1989 were consistently lower 
for African-Americans than whites. See U.S. General Accounting Office, 
Social Security: Racial Difference in Disability Decisions Warrants 
Further Investigation, GAO/HRD-92-56 (Washington, D.C.: Apr. 21, 1992).

[2] See U.S. General Accounting Office, SSA Disability Decision Making: 
Additional Measures Would Enhance Agency's Ability to Determine Whether 
Racial Bias Exists, GAO-02-831 (Washington, D.C.: Sept. 9, 2002).

[3] Changes in coding schemes over time limit our ability to analyze 
Hispanic and other ethnic groups separately. Prior to 1980, race data 
were collected for three categories: white, black, or other. In 1980, 
SSA adopted new codes: "White," "Black," "Hispanic," "Asian or Pacific 
Islander," and "American Indian or Alaskan Native." Because much of the 
race data were collected before 1980, and were not recoded into the new 
categories, "Hispanic," "Asian or Pacific Islander," or "American 
Indian or Alaskan Native," we were unable to conduct our analyses using 
these new categories.

[4] GAO-02-831.

[5] In conducting these tests, we compared the enhanced data with data 
from SSA's administrative files. See appendix I.

[6] To construct the models, we reviewed pertinent literature and 
consulted with SSA officials and outside experts.

[7] After estimating our initial model of factors affecting ALJ 
decisions using logistic regression analysis, we identified race, 
attorney representative, and several other factors that are not part of 
the criteria used in the decision-making process but that had a 
statistically significant influence on allowance decisions. We 
constructed additional models that included combinations of these 
variables to determine the influence of these variables on allowance 
decisions. One of these interaction variables--controlling for African-
American claimants that had attorney representation--had a 
statistically significant influence on allowance decisions and was, 
therefore, included in our final model. To further analyze the 
relationship between race and attorney representation on allowance 
decisions, we employed a statistical technique--the Oaxaca 
decomposition--that is commonly used in analyses of discrimination. See 
appendix I for a description of this analysis.

[8] SSI also provides income assistance to the aged who have income and 
assets below a certain level.

[9] The Social Security commissioner has the authority to set the 
substantial and gainful activities level for individuals who have 
disabilities other than blindness. In December 2000, SSA finalized a 
rule calling for the annual indexing of the nonblind level to the 
average wage index of all employees in the United States. The current 
nonblind level is set at $800 per month. The level for individuals who 
are blind is set by statute and is also indexed to the average wage 
index. Currently, the level for blind individuals is $1,330 of 
countable earnings.

[10] DI beneficiaries with low income and assets can also receive SSI 
benefits. Of the 5.5 million DI beneficiaries, about .8 million also 
received SSI in 2002. Thus, there was a total of 8.5 million working-
age beneficiaries in 2002, with 9 percent receiving both DI and SSI.

[11] SSA permits DI, but not SSI, claimants to file for benefits on-
line.

[12] While most claimants may request a reconsideration, at the time of 
our study, SSA was testing an initiative that eliminates the 
reconsideration step from the DDS decision-making process. In her 
September 2003 testimony before Congress, SSA's Commissioner proposed 
eliminating reconsideration as part of a large set of revisions to the 
disability decision-making process.

[13] According to SSA's Hearings, Appeals and Litigation Law Manual 
(HALLEX), Sec. I-2-5-30, the ALJ decides whether the testimony of a 
medical or vocational expert is needed at a hearing.

[14] If the claimant is not satisfied with the Appeals Council 
decision, the claimant may appeal to a federal district court. The 
claimant can continue legal appeals to the U.S. Circuit Court of 
Appeals and ultimately to the Supreme Court of the United States.

[15] Obtaining this documentation is complicated by the fact that files 
are stored in different locations, depending on whether the case 
involved an SSI or DI claim, and whether the ALJ decision was an 
allowance or denial. For fiscal years 1999 and 2000, SSA obtained files 
and tapes for 48 percent of the 33,484 records sampled. The case file 
contains the application for benefits, disability information provided 
by the claimant, DDS determinations, claimant's appointment of an 
attorney/representative (if applicable), appeal request documentation, 
medical evidence furnished at each level of the appeal, and the ALJ 
decision. For ALJ allowance decisions, the file will also contain 
documentation of benefit computation and payment.

[16] The number of medical consultants used depends on the number and 
type of impairments alleged by the claimant.

[17] In the peer review process, ALJs use the standard of substantial 
evidence that means that the ALJ should not overturn a decision if the 
relevant evidence is what a reasonable mind might accept as adequate to 
support a conclusion. In the original ALJ hearings process, ALJs use a 
higher standard of preponderance of evidence that means that more than 
half of the evidence must support a particular conclusion. 

[18] According to SSA's HALLEX, Sec. I-3-3-2, abuse of discretion in a 
judgment or conclusion involves an ALJ acting in a manner that is 
imprudent, incautious, unwise, against precedent, and clearly against 
logic. 

[19] According to SSA's HALLEX, Sec. I-3-3-3, error of law covers six 
broad issues: (1) misinterpretation of law or regulations; (2) 
misapplication of the law, regulations, or rulings to the facts; (3) 
failure to consider pertinent provisions of law, regulations, or 
rulings; (4) failure to make a finding of fact, or to give reasons for 
making a finding of fact, on an issue properly before the ALJ; (5) a 
procedural error that affects due process (e.g., improper notice of 
hearing, failure to notify the claimant of the right to question 
witnesses; and (6) failure to rule on an objection raised at the 
hearing.

[20] GAO/HRD-92-56.

[21] GAO-02-831.

[22] The complete results of our model are presented in appendix I.

[23] The category for nonattorney may include representatives from 
legal aid organizations, which could include attorneys as well as 
nonattorneys.

[24] About 25 percent of the claimants from the other racial/ethnic 
group had translators at their hearings, and our analyses also show 
that claimants who had translators at the hearing were less likely to 
be awarded benefits than claimants who did not have translators.

[25] This discussion pertains only to claimants with no representation 
as compared with claimants with attorney representation, and does not 
pertain to claimants with nonattorney representatives such as legal 
aides, relatives, and friends. Additional analyses showed that among 
claimants with nonattorney representatives, African-Americans were 
less likely to be awarded benefits than whites. However, this result 
may be due to the low number of observations for claimants with 
nonattorneys.

[26] The odds on claims being allowed are related to, but not quite the 
same as, the probability of claims being allowed. Suppose that among 
whites, 200 claims were allowed among a total of 300 filed. While the 
probability of claims being allowed is estimated by dividing the number 
of claims allowed by the number of all claims (i.e., 200/300= 0.66), 
odds are estimated by dividing the number of claims allowed by the 
number of claims not allowed (i.e., 200/100 = 2). If we found that 
among African-Americans, 50 out of 100 claims were allowed, we would 
calculate the odds of allowance to be 50/50 = 1.00, and the odds ratio 
of African-Americans to whites would be 1.00/2.00 = 0.5. This implies 
that the odds for African-Americans were only one-half those of whites. 
While probabilities (P) and odds (O) are mathematically related (O = P/
[1-P]), odds have certain advantages over probabilities for these 
statistical purposes, which is why we employ them. 

[27] See appendix I for an explanation as to why this interaction term 
was created and an explanation of how the specific result was 
calculated.

[28] The effect of attorney representation for other race/ethnicity 
claimants is not significantly different than for white claimants. 

[29] See appendix I for a description and the results of our Oaxaca 
decomposition analysis.

[30] Attorneys' efforts to obtain medical evidence might result in 
better medical evidence than that obtained by SSA earlier in the 
decision-making process because, for example: (1) attorneys often use 
request forms that are tailored to the disability criteria and the 
claimant's impairments to solicit specific information on the 
claimant's medical history from medical providers and (2) attorneys pay 
more for medical records than SSA. 

[31] We were told by attorneys affiliated with NOSSCR that attorneys 
typically screen their claimants to assess the strength of the 
claimant's case. If the attorney believes the evidence does not support 
an argument for the claimant's disability, as defined in SSA's 
guidelines, the attorney is not likely to take the case. This may mean 
that claimants with attorneys have stronger cases and are more likely 
to be approved for benefits regardless of the additional assistance 
provided by the attorney. Relatedly, ALJs--who may be aware that 
attorneys choose stronger cases--may be more likely to view a claimant 
with an attorney as having an impairment with such severity so as to 
qualify the claimant for benefits.

[32] Additional analyses showed that among claimants with nonattorney 
representatives, African-Americans were less likely to be awarded 
benefits than whites. However, this result may be due to the low number 
of observations for claimants with nonattorneys.

[33] The current model compares claimants in the Boston Region with 
claimants in the New York Region (the reference category). However, 
when we use any other region as the reference category, claimants from 
the Boston Region are always significantly more likely to be awarded 
benefits than claimants from the reference region.

[34] These variables include number of impairments, number of severe 
impairments, physical and mental capacity, type of impairment, 
occupational years, age, occupational categories, occupational skill 
level, education, literacy, and earnings. 

[35] The quality assurance review of ALJ decisions includes analyses of 
the accuracy of ALJ decisions, in which the reviewing ALJs assess 
whether the original ALJ's ultimate decision to allow or deny is 
supported by substantial evidence--which is referred to in the quality 
assurance review as support rates. This review also includes analyses 
of the fairness of ALJ hearings in which the reviewing ALJs evaluate a 
multitude of issues, including abuse of discretion and error of law.

[36] SSA's analysis of ALJ decisions is limited to descriptive 
statistics; SSA does not use multivariate techniques--i.e., control for 
other factors simultaneously--in its analysis of ALJ decisions.

[37] In addition to not analyzing AJJ decisions by race, SSA does not 
analyze ALJ decisions by sex or income.

[38] GAO-02-831.

[39] In SSA's enhanced data that we used for our analysis, only 10 
percent of the cases represented unsupported ALJ decisions, and only 13 
percent of these were for African-Americans.

[40] As described in appendix I, we compared the characteristics of 
claimants in SSA's enhanced data with the characteristics of claimants 
that were originally sampled for but, for various reasons, were not 
included in the enhanced data, and did not find large differences 
between the two claimant groups. However, our results might be due to 
the particular cases sampled and/or not included for various reasons 
during the time period. 

[41] SSA currently envisions selecting several hundred cases that were 
originally excluded from the sample and reviewing them after the agency 
has reached a final decision.

[42] A case is considered final by the agency when a claimant has 
exhausted his or her right to appeal, and either SSA or the federal 
courts have rendered a final decision. For example, a decision is 
considered final when the Appeals Council dismisses cases or upholds, 
modifies, or reverses the ALJ's action. If the Appeals Council remands 
the case back to the ALJ level, the case is not considered final until 
the ALJ decides on the case. Appeals to the federal court system would 
further delay the final decision.

[43] For example, claimants have 60 days to appeal the ALJ decision to 
the Appeals Council, after which the average number of days for 
processing and deciding a case at the Appeals Council level is about 
225 days. It takes, on average, an additional 250 days to reach a final 
decision for cases that are remanded by the Appeals Council back to the 
ALJ.

[44] The quality of data could be affected when policies and guidance 
change over time. For example, reviewing ALJs may be using policies and 
guidance that were not applicable when the original ALJ decided on a 
case. For corrective action to be effective, it should be taken in a 
timely manner. For example, if a belated quality assurance review finds 
that a certain region does not make accurate and fair decisions for a 
substantial number of its cases, corrective action might occur long 
after the problem occurred.

[45] GAO-02-831.

[46] Under current procedures, SSA is unlikely to subsequently obtain 
information on race and ethnicity for individuals assigned SSNs at 
birth unless those individuals apply for a new or replacement Social 
Security card, due to a change in name or a lost card.

[47] Since SSA's EAB program began in 1990, and our study used a sample 
of adult disability claimants from 1997-2000, most claimants in our 
sample preceded the EAB program. As a result, we had race data for most 
of the claimants in our sample.

[48] See U.S. General Accounting Office, SSA Disability Decision 
Making: Additional Measures Would Enhance Agency's Ability to Determine 
Whether Racial Bias Exists, GAO-02-831 (Washington, D.C.: Sept. 9, 
2002).

[49] We are grateful to four outside experts who assisted us with this 
study. They are Judith Hellerstein, Associate Professor of Economics at 
the University of Maryland; Joseph Kadane, Professor of Statistics and 
Social Sciences at Carnegie-Mellon University; Brent Kreider, Associate 
Professor of Economics at Iowa State University; and Kajal Lahiri, 
Professor of Economics at the University at Albany, State University of 
New York. We take full responsibility for any errors.

[50] See below for a discussion of the representativeness of the 
enhanced data.

[51] In conducting these tests, we found that only one data field 
(occupation from the 831 administrative file) did not pass all 3 of 
these tests and was, therefore, excluded from the subsequent 
nonresponder analyses.

[52] The data on medical severity in the enhanced data are developed 
during DDHQ's disability examiner/medical consultant review--a process 
that is independent from SSA's disability decision-making process. The 
medical severity variables are proxies for information that the judge 
would have seen during the hearing, but are not developed by the judge. 
Thus, they are appropriate for use in a regression estimating the 
judge's allowance decision.

[53] U.S. General Accounting Office, Social Security: Racial Difference 
in Disability Decisions Warrants Further Investigation, GAO/HRD-92-56 
(Washington, D.C.: Apr. 21, 1992).

[54] GAO-02-831.

[55] Specifically, 140 decisions from each region were selected per 
month. Of the 140 decisions, 70 were denials and 70 were allowances.

[56] This usually occurs for cases that were denied, but can also occur 
for allowances such as when the claimant disputes the date of onset.

[57] The nonresponders also include the sample of files that were 
obtained, but did not undergo all three reviews.

[58] By best possible estimates, we mean unbiased estimates, combined 
with small standard errors.

[59] Other factors that are available in the enhanced data, but are not 
available in the administrative data, include variables on the 
claimant's occupational skill level and whether the claimant is 
literate.

[60] The enhanced data contain variables that are equivalent (or very 
similar) to the variables in SSA's administrative files, such as 
occupation, but are likely to be more complete and accurate than 
administrative data, per our data reliability assessments. We used the 
enhanced data for this analysis so that we would capture only the added 
value of the variables that are available in the enhanced data in our 
comparison. If we had used the 831 and CCS data in Model A and the 
enhanced data in Model B, then Model B might also capture the effect of 
the higher quality of the enhanced data.

[61] This model was the preliminary model of the ALJ decision-making 
process, from which our final model was derived.

[62] Specifically, the variables that we compared include demographic 
factors such as age, sex, and race; vocational factors such as years 
employed and years of education; medical variables such as the body 
system involved in the claimant's impairment (at the DDS level and at 
the ALJ level) and whether they had a consultative exam; and 
administrative variables including claim type, hearing participants 
(attorney representation, nonattorney representation, vocational 
expert present, medical expert present), ALJ allowance decision, the 
final allowance decision (including Appeals Council decision if 
claimant was denied at the ALJ level and appealed to the Appeals 
Council), and regulation basis codes (indicating the step of sequential 
disability decision-making process at which claimant was allowed or 
denied).

[63] We did not use the enhanced data to conduct this analysis because 
they were not available for nonresponders. Had we used the enhanced 
data for nonresponders and SSA's administrative data for nonresponders, 
it would have been difficult to separate the differences between 
responders and nonresponders in characteristics with the differences 
between the enhanced data and SSA's administrative data in quality.

[64] We conducted the nonresponder analysis with and without 
probability weights. The results of both sets of analysis were 
consistent. 

[65] A sampling error is a variation that occurs by chance when a 
model/analysis relies on a sample that was surveyed rather than the 
entire population. The size of the sampling error reflects the 
precision of the estimate--the smaller the sampling error, the more 
precise the estimate.

[66] Four outside experts reviewed our methods and preliminary results 
and provided us with helpful feedback. They are Judith Hellerstein, 
Associate Professor of Economics at the University of Maryland; Joseph 
Kadane, Professor of Statistics and Social Sciences at Carnegie-Mellon 
University; Brent Kreider, Associate Professor of Economics at Iowa 
State University; and Kajal Lahiri, Professor of Economics at the 
University at Albany, State University of New York.

[67] See appendix II for a description of the 5-step decision-making 
process.

[68] In 1996, the Contract With America Advancement Act provided that 
individuals could not be found disabled for purposes of DI or SSI if 
drug addiction or alcoholism was a "contributing factor material to the 
determination of disability." Drug addicts and alcoholics who were 
disabled as a result of other causes would still be eligible.

[69] Claim type includes SSI claims, DI claims, and concurrent claims 
for both SSI and DI.

[70] The year of the decision might capture changes in decision making 
that have occurred over time due to changes in national policy or in 
the economic health of the country. In addition, region might capture 
regional differences in culture, social norms, court decisions or 
geographic variation in SSA's practices. In "A Structural Model of 
Social Security's Disability Determination Process," in The Review of 
Economics and Statistics, May 2001, 83(2): 348-61, Jianting Hu, Kajal 
Lahiri, Denton R. Vaughan, and Bernard Wixon found evidence that 
allowance rates at the initial level differed significantly by region 
at Step 2 and 4 of the disability decision-making process. In 
"Disability Insurance: Applications, Awards, and Lifetime Opportunity 
Costs," Journal of Labor Economics, Oct. 1999, 784-827, Brent Kreider 
found a significant relationship between region allowance rates and the 
likelihood of allowance for an individual claimant.

[71] GAO/HEHS-94-94 found significant differences in allowance 
decisions at the initial level by sex. GAO/HRD-92-56 found significant 
differences in allowance decisions at the hearings level by race. 
Additionally, in "A Structural Model of Social Security's Disability 
Determination Process," in The Review of Economics and Statistics, May 
2001, 83(2): 348-61, Jianting Hu, Kajal Lahiri, Denton R. Vaughan, and 
Bernard Wixon found that sex and race played a statistically 
significant role in Step 2 of the decision-making process. In SSA's 
initial comments on our analysis, they suggested that we incorporate a 
variable that controls for the claimant's earnings into our model.

[72] Although earnings are used in Step 1 of the decision-making 
process to determine whether the claimant's earnings exceed the limit 
required for eligibility (and to determine whether the claim type is 
SSI or DI), earnings are not considered in Steps 2-5, which pertain to 
the ALJ disability decision-making process.

[73] Although we had no compelling theoretical or empirical reason for 
testing this particular interaction, we believed it would be useful to 
determine whether any racial differences that we found in our initial 
model were larger at the beginning of the 4-year period for which we 
had data than they were at the end of the 4-year period.

[74] Odds (O) are mathematically related to but not the same as 
probabilities (P), that is O = P/[1-P]. For further explanation of how 
to interpret odds and odds ratios, see text after table 6.

[75] Comparison categories can be identified because they have an odds 
ratio of exactly 1.00 and in our report, with the exception of region, 
are presented first among the categories of a variable.

[76] Due to the presence of interaction terms between race and attorney 
representation in the final model, the odds ratio for the race variable 
in the final model represents the odds ratio for claimants of a 
particular race who do not have attorney representation.

[77] The odds ratio for the interaction variable for claimants from 
other racial/ethnic groups with attorney representation is not 
significant. This indicates that the effect of attorney representation 
on the odds of allowance for claimants from other racial/ethnic 
backgrounds is not significantly different from the effect of attorney 
representation on the odds of allowance for white claimants.

[78] This difference probably results from SSA's systematic exclusion 
of cases that are appealed to the Appeals Council from the enhanced 
data. According to attorneys that represent SSA claimants, attorneys 
usually advise claimants who are denied at the ALJ level to appeal to 
the Appeals Council. Therefore, claimants who are denied at the ALJ 
level and appeal to the Appeals Council are likely to have higher rates 
of attorney representation than claimants who are denied at the ALJ 
level and do not appeal. 

[79] For details on this technique see "Male-Female Wage Differentials 
in Urban Labor Markets," by Ronald Oaxaca, in International Economic 
Review, Volume 14, Issue 3 (Oct. 1973), 693-709.

[80] The original ALJ's assessment of the claimant's credibility cannot 
be used as an independent variable because it is too highly correlated 
with the final allowance decision and could distort other results in 
our model.

[81] For children applying for SSI, the process requires sequential 
review of the child's current work activity (if any), the severity of 
his or her impairment(s), and an assessment of whether his or her 
impairment(s) results in marked and severe functional limitations.

[82] The 2003 SGA level for claimants who are not blind is $800. The 
2003 SGA level for persons who are blind is $1,330.

[83] SSA's Listing of Impairments describes, for each major body 
system, impairments that are considered severe enough to prevent an 
adult person from doing any gainful activity (or in the case of 
children under age 18 applying for SSI, cause marked and severe 
functional limitations). Most of the listed impairments are permanent 
or expected to result in death, or a specific statement of duration is 
made. For all others, the evidence must show that the impairment has 
lasted or is expected to last for a continuous period of at least 12 
months. The criteria in the Listing of Impairments are applicable to 
evaluation of claims for disability benefits under both the Social 
Security DI and SSI programs.

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: