Table of Contents | Previous | Next |
The Evaluation
This section provides some key information about the WRP evaluation, including its research design, environment, samples, and data sources.
The Evaluation’s Design
Components of the study. This report presents results from the three major components of the WRP evaluation:
-
Impact analysis. This part of the study provided estimates of the effects of WRP on employment rates and earnings, public assistance receipt, family income, and other outcomes relative to the welfare system that preceded it.
-
Implementation analysis. This component of the study examined how WRP’s policies were operated by staff in the six research districts.
-
Benefit-cost analysis. This analysis used data from the impact study, along with fiscal data, to compare the financial benefits and costs generated by WRP for both eligible families and the government budget.
Research design for the impact analysis. The impact analysis was based on a random assignment research design. Between July 1994 and December 1996, cash assistance applicants and recipients throughout Vermont were assigned, at random, to one of three groups:9
-
WRP group (60 percent). Members of this group were subject to all the elements of WRP described earlier, including both the time-triggered work requirement and the financial work incentives.
-
WRP Incentives Only group (20 percent). Members of this group received WRP’s enhanced financial incentives but were not subject to its work requirement.
-
ANFC group (20 percent). Members of this group remained subject to the welfare rules that existed before WRP.
Members of all three groups had the same access to employment and training, case management, and support services through Reach Up. Table 1 summarizes the policies that applied to each of the three research groups.
Because people were assigned to a group at random, there were no systematic differences among the three groups’ members when they entered the study. In addition, all three groups experienced the same general economic and social conditions during the study. Thus, any differences that emerged among the groups during the study’s follow-up period can reliably be attributed to WRP; these differences are known as the program’s impacts, or effects.
As discussed earlier, the key elements of WRP can be grouped into two categories: (1) financial incentives to promote and reward work and (2) the time-triggered work requirement. The three-group design allows the evaluation to decompose the program’s overall impact. Specifically:
- Comparing the WRP group with the ANFC group shows the combined impact of WRP’s incentives and work requirement relative to the traditional welfare system.
| WRP Group | WRP Incentives Only Group | ANFC Group | |
|---|---|---|---|
| Work and Reach Up participation requirements | |||
| Single-parent families | Parent required to work after 30 months of cash assistance receipt; mandatory job search after 28 months | No participation mandate or work requirement | No participation mandate or work requirement |
| Two-parent familiesa | Immediate Reach Up participation mandate for principal earner; required to seek and accept employment at all times; full-time work requirement after 15 months of cash assistance receipt | Immediate Reach Up participation mandate for principal earner; required to seek and accept employment at all times | Immediate Reach Up participation mandate for principal earner; required to seek and accept employment at all times |
| Financial/eligibility rules and transitional benefits | |||
| Earned income disregard | First $150 plus 25% of remaining earnings disregarded | Same as WRP group | First $120 plus 33% of remaining earnings disregarded for first 4 months of employment; first $120 disregarded in Months 5 to 12; first $90 disregarded thereafter |
| Value of vehicle excluded in counting assets | One vehicle excluded regardless of valueb | Same as WRP group | $1,500 |
| Other asset rules | Assets derived from earnings disregardedc | Same as WRP group | Assets derived from earnings counted against asset limits |
| Child support rules | All child support passed through to the custodial parent; amount in excess of $50 a month counted against the cash assistance grantd | Same as WRP group | First $50 a month in child support passed through to the custodial parent and not counted against cash assistance grant; state retained the rest |
| Medical assistance for families leaving welfare for work | Three years of transitional Medicaid; coverage beyond that point depended on eligibility for other programse | Same as WRP group | One year of transitional Medicaid; coverage beyond that point depended on eligibility for other programse |
| Child care assistance for families leaving welfare for workf | Subsidies for all types of care continued as long as family remained financially eligible | Same as WRP group | Subsidies for licensed or registered care continued as long as family remained financially eligible; subsidies for “unregulated” arrangements continued for only one yearg |
| Cash assistance eligibility for two-parent families | Nonfinancial eligibility criteria similar for single-parent and two-parent families | Same as WRP group | Two-parent families subject to special nonfinancial eligibility requirements (e.g., principal earner had to work less than 100 hours per month) |
| Family composition rules | Unmarried couples who lived together and had a child in common treated as one family unit, along with any other children in the household | Same as WRP group | Unmarried couples who lived together and had a child in common treated as two family units as long as the principal earning parent remained employed 100 hours or more per month |
| SOURCE: Welfare Restructuring Project (WRP)
policies from Vermont’s Department of Social Welfare.
NOTES:aThese rules were for two-parent families with an able-bodied primary wage-earner (ANFC-UP cases). Different rules applied to two-parent families with an incapacitated parent; the able-bodied parent in these families was subject to rules that are similar to those for single parents.(back) bThe enhanced vehicle exclusion also applied in determining eligibility for Food Stamps for families who had received cash assistance subsequent to random assignment, and it continued to apply to Food Stamps after the family was no longer receiving cash assistance.(back) cThe disregard applied to assets accumulated after receiving cash assistance benefits subsequent to random assignment. It also applied in determining Food Stamp eligibility for families who had received cash assistance subsequent to random assignment, and it continued to apply to Food Stamps after the family was no longer receiving cash assistance.(back) dThe $50 pass-through also applied to Food Stamps for families who received cash assistance subsequent to random assignment, and it continued to apply after the family was no longer receiving cash assistance.(back) eFamilies in all three groups could receive subsidized health coverage through other programs (e.g., Vermont Health Access Plan and Dr. Dynasaur); there were no differences across groups in eligibility rules for these programs.(back) fThis table describes the child care rules in place through June 1998.(back) gFamilies in the ANFC group using unregulated arrangements could continue to receive subsidies beyond the one-year transitional child care period if they met income guidelines and the provider became registered or licensed by the Department of Social and Rehabilitation Services (SRS), a requirement for receiving payments for SRS’s child care program, or the family switched from an unregulated provider to a licensed or registered provider of child care.(back) |
-
Comparing the WRP Incentives Only group with the ANFC group shows the impact of WRP’s financial incentives alone, not accompanied by the work requirement.
-
Comparing the WRP group with the WRP Incentives Only group shows the impact of adding the work requirement to the financial incentives.10
Most of the findings presented in this report are based on the first comparison, and they thus show the effects of the full WRP package of services and requirements. Results based on the second and third comparisons are discussed briefly and are presented in the appendices (see Box 1).
It is worth noting that the WRP research sample includes a very broad share of the cash assistance caseload. In most previous studies of welfare reform initiatives, certain categories of exempt cases — for example, parents of very young children — were screened out before random assignment and did not become part of the research sample. In contrast, Vermont chose to include almost all cash assistance applicants and recipients in the study (and in WRP) and to identify exemptions at the point that recipients approached the time-triggered work requirement. This characteristic of the WRP sample is important to consider in making comparisons across studies.
In addition, for cash assistance applicants, random assignment took place early in the application process, before staff knew whether the application would be approved or denied. Thus, some individuals (about 5 percent) in the three research groups never received cash assistance during the follow-up period. Conducting random assignment at this early point gave the study a better chance of capturing the full impact of WRP; for example, the program may have affected the number of people who completed their application or who were approved for benefits. In fact, this is likely, because WRP included changes in the welfare eligibility rules. At the same time, the early point of random assignment means that some people in the WRP group and in the WRP Incentives Only group had only very limited contact with the program’s new policies.
Random assignment is generally recognized as the most reliable way to determine what difference, if any, a new program makes. Nevertheless, a few factors should be considered when interpreting the evaluation’s results. The earlier discussion noted that Vermont’s approach to welfare reform is different in several ways from the approach advocated by the 1996 federal law (and from reforms enacted in neighboring states) and that it was impossible to isolate sample members from the intense publicity generated by national welfare reform throughout the study period. Thus, it seems likely that some members of the ANFC group may have been affected by this general message even if they understood that they were not subject to WRP’s specific rules and requirements. Similarly, members of the two WRP groups may have been confused about the policies that applied to them, because they may have heard that the federal law includes different policies. The broad new “message” about welfare may have affected the number of people who applied for benefits, but the research design cannot measure such a change.11 Finally, as discussed below, many of the broad changes in Vermont’s welfare system in the 1990s applied to all three research groups. In sum, the evaluation’s results represent a conservative estimate of the model’s potential.
The Evaluation’s Environment and the Target Population
When assessing WRP’s effects, it is helpful to consider the environment in which the program was studied as well as the composition of the research sample.
The State of Vermont. Table 2 provides some basic information about the State of Vermont (and, for comparison, about the United States). As the table shows, Vermont is a small, mostly rural state with a racially homogenous population: 98 percent of its residents are white. In 1998, it ranked 49th among the 50 states in population, and its poverty rate was lower than the nation’s average. Vermont’s economy was exceptionally healthy: As the table shows, the state’s unemployment rate remained below the national average throughout the study period.
Vermont’s welfare grant levels are among the highest in the nation. In January 1997, the maximum welfare grant for a family of three with no other income was $640.12 Like most other states, Vermont experienced a significant decline in its welfare caseload in the late 1990s. The statewide caseload declined from about 9,900 in 1994 (the year that WRP began operating) to less than 6,000 in 2000 — a decrease of 39 percent. (Appendix Table A.1 presents Vermont’s caseload size for selected years.)
The research districts. As noted earlier, MDRC’s evaluation included data from all 12 welfare districts in Vermont but focused in detail on 6 of them, which are referred to as the research districts.13 The Burlington district includes Vermont’s largest city and serves about one-fifth of the state’s welfare caseload. The Barre, Rutland, and St. Albans districts include smaller cities or towns, while the Newport and Springfield districts are more rural. Together, the research districts include about two-thirds of Vermont’s cash assistance caseload. (Appendix Table A.1 provides some basic information on each of the research districts.)
| Characteristic | Vermont | United States |
|---|---|---|
| Total population (1998) | 590,883 | 270,298,524 |
| Rank among the 50 states (1998) | 49 | N/A |
| White population (1998) (%) | 98.4 | 82.5 |
| Rural population (1990) (%) | 67.9 | 24.8 |
| Median household income (1998) ($) | 36,196 | 37,779 |
| Poverty rate (1998) (%) | 9.9 | 12.7 |
| Annual average unemployment rate (%) | ||
| 1994 | 4.7 | 6.1 |
| 1995 | 4.3 | 5.6 |
| 1996 | 4.6 | 5.4 |
| 1997 | 4.0 | 4.9 |
| 1998 | 3.4 | 4.5 |
| 1999 | 3.0 | 4.2 |
| 2000 | 2.9 | 4.0 |
| 2001 | 3.6 | 4.8 |
| Nonfarm employment by industry (1995) (%) | ||
| Manufacturing | 16.7 | 15.8 |
| Services | 29.3 | 28.2 |
| Transportation and public utilities | 4.4 | 5.3 |
| Government | 16.7 | 16.5 |
| Wholesale and retail sale | 23.7 | 23.5 |
| Construction | 4.4 | 4.4 |
| Finance, insurance, real estate | 4.4 | 5.8 |
| SOURCES: Gaquin and DeBrandt, 2000; Statistical
Abstract of the United States, 1996, 1999, 2000 (state rank, rural
population, poverty rate, and nonfarm employment by industry); U.S.
Department of Labor, Bureau of Labor Statistics (unemployment rates).
NOTE: N/A indicates that data are not applicable. |
Characteristics of the research sample. Table 3 shows selected characteristics and the attitudes and opinions of the single-parent sample members when they entered the evaluation. These data were drawn from the Background Information Forms (BIFs) and the Private Opinion Survey (POS), which were completed just before random assignment. As the table shows, nearly all the single-parent sample members are women, and their average age at the time of random assignment was 31. Most sample members had small families (the average number of children was 1.8), but more than one-third had at least one child under age 3.
The baseline data also show that Vermont’s cash assistance caseload was not as disadvantaged as recipients in many other states. Nearly three-fourths of the sample members had at least a high school diploma at the point of random assignment. Almost all sample members (92 percent) had at least some work experience. Most sample members, however, had little recent work experience: Just over half had not worked in the year before the study.14
Overall, responses to the POS indicate that most sample members expressed negative views of welfare and expected to be working and off welfare relatively quickly. For example, more than four-fifths said that they expected to be working one year later, and only one-fourth said that they expected to be receiving welfare at that point. At the same time, however, the responses indicate that many sample members were concerned about their ability to support their families through work. Over 75 percent reported that they faced at least one of five specific barriers to employment listed on the survey. Many sample members were concerned about the financial trade-offs involved in going to work, which can be particularly onerous in a state like Vermont that pays relatively high welfare grants. For example, more than 60 percent of the survey respondents said that being on welfare provided for their family better than working could.15
| Characteristic | Report Sample |
|---|---|
| Selected characteristics | |
| Gender/sex (%) | |
| Female | 93.3 |
| Male | 6.7 |
| Average age (years) | 30.8 |
| Average number of children | 1.8 |
| Age of youngest child (%) | |
| Under 3a | 36.9 |
| 3-5 | 22.8 |
| 6-12 | 29.7 |
| 13-18 | 10.6 |
| Ever worked (%) | 91.7 |
| Ever worked full time for 6 months or more for one employerb (%) | 61.6 |
| Has a diploma or GEDc(%) | 73.1 |
| Client-reported barriers to employment | |
| Among those not currently employed, the percentage who agreed or agreed a lot that they could not work part time right now for the following reasons:d | |
| No way to get there every day | 40.6 |
| Cannot arrange for child care | 39.6 |
| A health or emotional problem, or a family member with a health or emotional problem | 32.8 |
| Too many family problems | 27.5 |
| Already have too much to do during the day | 25.5 |
| Any of the above five reasons | 75.7 |
| Client-reported expectations regarding employment | |
| Percentage who agreed or agreed a lot that: | |
| It will probably take them more than a year to get a full-time job and get off welfare | 58.6 |
| They would take a full-time job today, even if the job paid less than welfare | 25.7 |
| If they got a job, they could find someone they trusted to take care of their children | 79.3 |
| A year from now they expect to be working | 82.4 |
| A year from now they expect to be receiving welfare | 26.6 |
| Client-reported attitudes toward welfare | |
| Percentage who agreed or agreed a lot with the following statements: | |
| I feel that people look down on me for being on welfare | 67.8 |
| I am ashamed to admit to people that I am on welfare | 60.6 |
| Right now, being on welfare provides for my family better than I could by working | 60.7 |
| I think it is better for my family that I stay on welfare than work at a job | 17.9 |
| Sample sizee | 5,469 |
| SOURCES: MDRC calculations using data from
Background Information Forms and the Private Opinion Survey (POS).
NOTES: In most of the attitude and opinion item groupings, individuals could agree or agree a lot with more than one statement. Therefore, distributions may add up to more than 100 percent. Invalid or missing values are not included in individual variable distributions. Rounding may cause slight discrepancies in calculating sums and differences. aIncludes sample members pregnant with their first child.(back) bFull-time employment is defined as 30 hours or more per week.(back) cThe General Educational Development (GED) certificate is given to those who pass the GED test and is intended to signify knowledge of basic high school subjects.(back) dPart-time employment is defined as a minimum of 10 hours per week.(back) eThe sample size includes the 159 sample members who chose not to fill out a POS.(back) |
Data Sources, Samples, and Time Frame for the Analysis
Data sources. The WRP evaluation drew on a wide variety of data sources, including administrative records of public assistance payments, records of earnings reported to the unemployment insurance (UI) system, and an in-depth survey of sample members. Box 2 describes these and the evaluation’s other data sources.
|
Box 2
Data Sources Used in This Report Baseline data. Two brief baseline information forms — the Background Information Form (BIF) and the Private Opinion Survey (POS) — were completed just before each member of the research sample was randomly assigned. Field research. MDRC staff visited each of the research districts approximately yearly between 1994 and 2001. During these visits, MDRC staff interviewed WRP line workers and supervisors, and they observed program activities. Staff surveys. Written surveys were administered to virtually all welfare eligibility specialists and Reach Up workers in the research districts in mid-1996. A total of 82 eligibility workers and 72 Reach Up workers completed surveys (more than 90 percent of each staff). Computerized administrative records. DSW provided computerized administrative records, including:
42-Month Client Survey. A total of 1,872 sample
members (1,256 single parents and 616 respondents from two-parent
families) were interviewed by a subcontractor in 1998 and 1999,
approximately 42 months after each person’s random assignment
date. The survey achieved an 80 percent response rate. Respondents
answered a set of questions about employment, child outcomes, and
other issues. Community service employment (CSE) surveys. In 2000, surveys were administered to 81 CSE participants and to 79 CSE supervisors across the state of Vermont. Program expenditure data from DSW. These data were used to estimate the costs of WRP and the ANFC program. |
|
Vermont's Welfare Restructuring Project Figure 1 Samples and Subsamples Used in This Report |
| Full research sample Randomly assigned 7/94 - 12/96 in the research districts 10,637a |
 |
| Report sample Randomly assigned 7/94 - 6/95 in the research districts 7,691b |
|
|
| Fielded sample for the 42-Month
Client Survey Subset of families randomly assigned 10/94 - 6/95 in the research districts 2,326d |
|
|
| 42-Month Client Survey sample Members of the fielded sample who were interviewed 1,872 |
|
NOTES: aThis figure excludes minor parents and cases with invalid Social Security numbers. Nonrelative caretakers are also excluded. The corresponding figure for the statewide sample is 17,175.(back) bThe corresponding figure for the statewide sample is 12,183.(back) cThe figure for two-parent families includes 1,652 ANFC-UP families and 570 families with an incapacitated parent.(back) dThis figure does not include the 176 two-parent families who had an incapacitated parent.(back) |
Evaluation sample. As discussed earlier, cash assistance applicants and recipients were randomly assigned to the three research groups throughout Vermont between July 1994 and December 1996. Shown in Figure 1, the 10,637 people randomly assigned during this period in the research districts are referred to as the full research sample for the WRP evaluation (The total sample, including cases randomly assigned outside the research districts, is 17,175.)16 However, the impact analysis presented in this report focuses primarily on a subset of these cases — the 7,691 people randomly assigned in the research districts between July 1994 and June 1995. This group is referred to as the report sample. It includes the entire “on-board” caseload in the research districts (people who were already receiving cash assistance when WRP began) as well as people who applied for assistance during the first year of WRP’s operations. (The report also includes some results for all cases randomly assigned between July 1994 and June 1995 throughout the state — a total of 12,183 cases — and for the cases in the research districts randomly assigned between July 1995 and December 1996.)
A subset of the sample members who were randomly assigned between October 1994 and June 1995 was selected to be surveyed about three-and-a-half years after entering the study (the fielded sample illustrated in Figure 1). Eighty percent of these sample members responded to the survey; these 1,872 individuals make up the 42-Month Client Survey sample.
Most members of the report sample (71 percent) were single parents when they entered the study. The rest were members of one of two categories of two-parent families. The first category, in which both parents are able-bodied, received benefits through the ANFC-Unemployed Parent (UP) program. The second includes families in which a parent is incapacitated (“incap”). The report separately examines WRP’s effects for single parents and for each category of two-parent families. In general, WRP’s rules for the so-called incap two-parent families were similar to those for single parents, but the rules were quite different for ANFC-UP families. (WRP’s policies for two-parent UP families are described later.)
The impact analysis presented in this report is based on data from Vermont’s administrative records and the 42-Month Client Survey (see Box 2). Because the quarterly earnings data from the UI system cover through the second quarter of 2001, there are at least 24 quarters of post-random assignment earnings data available for each member of the report sample. In other words, 24 quarters (six years) elapsed between the date when the last member of the report sample was randomly assigned (June 30, 1995) and the last date for which earnings data are available (June 30, 2001). There are also six years of cash assistance and Food Stamp data available.17 As noted in Box 2, the client survey data cover the 42 months after each respondent’s date of random assignment. These follow-up periods are illustrated in Figure 2.
|
[D] |
Subgroups. The impact analysis also compares the results for various subgroups — subsets of the report sample defined by their baseline characteristics — in order to see whether WRP had different impacts for different groups of people. The most important subgroup analysis compares people who entered WRP with differing levels of disadvantage. The analysis also looks at WRP’s impacts for each of the six research districts and for several other subgroups.
9All applicants were assigned to a group when they came to the DSW office to apply for benefits. Parents who were already receiving cash assistance when WRP began operating were randomly assigned when they came to the office for semiannual eligibility reviews. To control the flow of people into WRP, only half of those who appeared for a review were randomly assigned; the rest remained subject to ANFC policies and were randomly assigned at their next review meeting.(back)
10It is important to note that the comparison between the WRP group and the WRP Incentives Only group does not necessarily show the independent impact of the work requirement. To obtain that result, it would be necessary to create a group whose members were subject to a work requirement but did not receive financial incentives.(back)
11The analysis could measure changes that occurred only after individuals were assigned to the research groups. Because the assignment occurred at the point people applied for welfare, the study could not determine whether WRP affected the number of people who took this step.(back)
12The maximum welfare grant had increased to $708 by January 2000.(back)
13The research districts were selected by DSW; they were not chosen randomly.(back)
14Appendix Table A.2 presents additional measures from the BIF for the single-parent report sample, and Appendix Table A.3 presents this information separately for each of the research districts. Appendix Tables A.4 and A.5 present selected measures from the BIF for sample members who were members of two-parent families when they entered the study.(back)
15Appendix Table A.6 shows all the measures from the POS for the single-parent report sample. Results from the POS were good predictors of eventual outcomes on employment and cash assistance receipt. For example, sample members who reported barriers to employment on the POS had weaker employment outcomes than those who did not report barriers. Also, sample members who said that they expected to be working a year from the time they responded to the POS had stronger employment outcomes than those who said that they did not expect to be working. Appendix Table D.16 presents WRP’s effects on employment and cash assistance receipt for various subgroups of sample members defined using POS responses.(back)
16Nonrelative caretakers, minor parents, and cases randomly assigned with invalid Social Security numbers were excluded from the research sample.(back)
17Fewer months of follow-up are available for sample members who were randomly assigned after June 1995.(back)
| Table of Contents | Previous | Next |