Assets for Independence Act Evaluation:
Design Phase Final Report
August 9, 2000

2.	Program and Participant Tracking and Monitoring
	2.1	Purpose
	2.2	Data Collection Plan
	2.3	Data Analysis Plan
	2.4	Cost Estimate

 

For program and participant tracking and monitoring, the Department of Health and Human Services specified the use of an existing or comparable Management Information System for Individual Development Accounts (MIS IDA). Therefore, MIS IDA and the Monitoring Instrument from which it was developed will be used as the basis for this section of the evaluation plan.

2.1 Purpose

MIS IDA was designed to collect data and answer basic questions regarding program process and goal attainment in IDA programs. Tracking and reporting on these questions will yield basic information about the performance of the IDA demonstration, as called for in Section 412 of AFIA items (1) through (8), and in evaluation as called for in Section 414(b) items (1) and (2). These questions are listed below:

Section 412. Annual Progress Reports

1. The number and characteristics of individuals making a deposit into an individual development account.
2. The amounts in the Reserve Fund established with respect to the project.
3. The amounts deposited in the individual development accounts.
4. The amounts withdrawn from the individual development accounts and the purposes for which such amounts were withdrawn.
5. The balances remaining in the individual development accounts.
6. The savings account characteristics (such as threshold amounts and match rates) required to stimulate participation in the demonstration project, and how such characteristics vary among different populations or communities.
7. What service configurations of the qualified entity (such as configurations relating to peer support, structured planning exercises, mentoring, and case management) increased the rate and consistency of participation in the demonstration project and how such configurations varied among different populations or communities.
8. Such other information as the Secretary may require to evaluate the demonstration project.

Section 414b. Evaluations

1. The effects of incentives and organizational or institutional support on savings behavior in the demonstration project.
2. The savings rates of individuals in the demonstration project based on demographic characteristics including gender, age, family size, race or ethnic background, and income.
   

 top of page

---

2.2 Data Collection Plan

The data collection plan includes discussions on collecting data in the field, and collecting data by the reporting contractor, as well as data cleaning procedures.

2.2.1 Data collection in the field

It is expected that AFIA grantees will purchase, install, and maintain MIS IDA software at one or more of their affiliated IDA program organizations. In order to ensure data integrity, it is highly recommended that AFIA grantees receive both training and technical support in the use of MIS IDA.

IDA program staff will collect data in the field on an ongoing basis. Programmatic information should be entered at program start-up; participant demographics should be entered when a participant enrolls in the program. Participant account information should be entered when a participant opens an account at the financial institution, and account statements should be entered on a monthly basis (quarterly if the financial institution only issues quarterly statements). Semi-annually, (or at a minimum annually), from a participant's date of enrollment, the field administrator should obtain updated information on the participant's demographics, monthly income, assets, and liabilities. Also semi-annually from program startup, the field administrator should review and enter any changes made in the overall program design. Each of these updates should be made in the Semi-Annual Update function in MIS IDA because the function will make a copy of the old information and retain a copy of the new record.

The Monitoring Instrument is the paper version of the information collected in MIS IDA and can be used for collecting data from participants (attached in Appendix B). Financial information can be collected from financial institutions either from paper copies of participants' financial statements or electronically.

MIS IDA Version 2.02 was used for reporting on the American Dream Demonstration (ADD) data through December 31, 1999. Version 3.03 is the latest version of the software. Because of differences in program guidelines between ADD and AFIA, CSD staff revised the software to better serve the AFIA programs. Version 3.03 includes 34 new (added or modified) program and participant variables, and a program design choice to calculate participant matches based on a per-year savings cap or a lifetime (of the program) savings cap. There are seven new program variables, and 27 new participant variables.

2.2.2 Data collection at the reporting organization

Annually, data from the AFIA sites will be sent to a reporting contractor for reporting. It is recommended that collection occur six months prior to reporting to give ample time for data cleaning and analysis. In year one of the demonstration, data should be collected after the first six months to check that IDA program staff are entering data correctly into MIS IDA.

Data are easily copied from MIS IDA to a Microsoft® Excel 97 file through a menu function in MIS IDA. The Excel file can then be sent to the reporting organization by email, or via conventional mail.

The reporting organization will aggregate the Excel files received from each of the AFIA grantees into a single MIS IDA database. After data cleaning has been completed (described below), all data in the aggregated Excel file will then be imported into SPSS statistical analysis software for additional cleaning.

2.2.3 Data cleaning procedures

Data cleaning is one of the most critical and time-consuming elements of data collection. Reporting and tracking will only be as accurate as the data that are provided from the field. Data sets may include data entry errors, misreported values, or missing data. In reporting for ADD, CSD has created a separate database system called MIS IDA Quality Control (MIS IDA QC). This database is comprised of cleaning procedures that help detect potential data entry errors, missing values, or accounting inconsistencies. Use of MIS IDA QC has significantly reduced the amount of time involved in identifying and correcting data inconsistencies. Examples of queries for cleaning participant data include:

• Range of values (above and below) where individual assets and liabilities fields exceed logical values.
• Value that includes home mortgage amount but zero home value amount.
• For an asset or liability, a question is checked "yes", but no values are shown.
• Total monthly income equals zero or is above $4000 per month (exceeds 200% of poverty on a monthly income basis).
• Number of adults in household equals zero (minimum is one).
• Computed total net worth is above or below logical values.
• Matched withdrawals shown with zero funding partner contribution.
• First account statement for a participant has beginning balance greater than zero.
• Missing fields or status "unknown" data.
• Participant age is less than 16 years old.
• Computed average monthly deposit (net) is less than zero.
• Economic education not specified or is specified but zero hours of participation recorded.
• Participant exit status is closed but account still open.
• Participant account status is closed but exit status still open.
• Missing account statement periods.

Queries for cleaning programmatic data include:

• Discrepancy in number of IDA staff hours and corresponding salaries amount.
• Illogical or inconsistent values in program expenditures or staffing fields.
• Missing fields or status "unknown" data.
• Unidentified funding partners.
• Missing reporting periods.

Each site will run the data cleaning reports, which correct all known errors, and submit its data to the reporting contractor, which will also run MIS IDA QC to verify that data are correct. For multi-site programs, the sponsoring organization's central site administrator should work with all of its affiliated sites in assuring the quality of each site's data set. The administrator must send the corrected aggregate (of all its sites) data to the reporting contractor who then verifies that each questionable item has been addressed and updated. The reporting contractor will determine when the data are sufficiently accurate. Once data are transferred into SPSS, the reporting contractor's analysts will identify any other data inconsistencies prior to analysis.

 top of page

---

2.3 Data Analysis Plan

Data from the AFIA sites will be collected and reported by site and in aggregate on an annual basis over the four years of the AFIA demonstration. The MIS IDA data set must be transferred into an SPSS (or other statistical analysis software) data set for analysis.

MIS IDA data analysis strategy will include descriptive characteristics for program, participants, and savings patterns for each site; and univariate, bivariate, and multivariate analyses across all sites. Savings patterns and uses of IDAs will be analyzed by program and by participant characteristics, with basic statistical analyses as appropriate. All of these analytical findings for each IDA site and for the AFIA demonstration as a whole, will be reported.

2.3.1 Definition of an IDA participant

Because field administrators sometimes add participants to the database during the recruitment phase and therefore prior to their actual enrollment, entries which can skew results of both participation and attrition, it is important to define who is considered a participant, whether actively enrolled or exited from the IDA program. Since the definition of an IDA program includes having an IDA account, an IDA participant that is included in the AFIA evaluation should be defined as a participant who has enrolled in the IDA program and has at least one account statement recorded in MIS IDA. Similarly, a "dropout," is a participant who has enrolled in the IDA program, has at least one account statement, but exited from the program based on the reason noted in the "participant exit" record.

2.3.2 Independent and dependent variables

Independent and dependent variables used in the data reporting are listed below.

Program independent variables:

• Age of organization
• Age of program
• Organization type
• Location of funds
• Program asset uses
• Account held in whose name
• Match funds held in whose account
• Statement period
• Average program match rate
• Program incentives for match dollars
• Waiting period in weeks to use match dollars
• Penalties for unapproved use
• Types of funding partners
• Types of IDA marketing activities
• Average months of marketing activity
• Total hours of economic education offered
• Average number of participants
• Average maximum monthly deposit
• Average organizational FTEs
• Average IDA FTEs (total)
• Average IDA salaried FTEs
• Average IDA unsalaried FTEs
• Average IDA expenses
• Average IDA salary expense

Participant independent variables:

• Number of participants in IDA program
• Monthly household income
• Age (5 Categories) (created from continuous variable "age")
• Gender
• Ethnicity (re-categorized into 4 groups)
• Residence (urban or rural)
• Marital status (re-categorized into 3 groups)
• Number of adults
• Number of children
• Household type (married with children; married without children; single with children; single without children) (created)
• Dependency ratio (number of kids/number of adults) (created)
• Education attainment (re-categorized into 4 groups)
• Employment status (re-categorized into 4 groups )
• Welfare status (created)
• Banked or not (created)
• Total value of income (created)
• Sources of income (created)
• Income-poverty ratio (created)
• Type of assets
• Total value of assets (created)
• Financial assets (created)
• Type of liabilities
• Total value of liabilities (created)
• Consumer debt (created)
• Net worth (created)
• Intended use of IDA
• Actual use of IDA
• Economic education hours received
• Reason for exit (re-categorized into 4 groups)
• Prior relationship with IDA sponsoring organization or affiliated partner organization
• Currently receiving food stamps
• Currently receiving SSI/SSDI
• Health insurance status
• Life insurance status

Dependent variables (all created):

• Participant savings. All deposits and interest minus unapproved withdrawals. Equivalently, it is the account balance on the date of data collection plus matched withdrawals. Thus, participant savings count financial assets held in an IDA program as well as matched withdrawals used to purchase approved assets.
• Average monthly deposit. Participant savings divided by the number of months of participation. Unlike participant savings, average monthly deposit does take account of (in its denominator) the length of time that a participant has had the opportunity to save.
• Deposit regularity. The number of months in which a deposit was made divided by the number of months in which a deposit was possible. One deposit each month would yield a ratio of 1.00. As a participant misses months, the ratio gets smaller, although it cannot get smaller than zero. Deposit regularity indicates to what extent participants save steadily through time. For the purpose of this measure, deposits of accrued interest are not counted as deposits.
• Deposit lumpiness. While deposit regularity measures the steadiness of deposits in terms of time, deposit lumpiness measures the steadiness of deposits in terms of amount. Deposit lumpiness is defined as the biggest single deposit divided by the average monthly deposit. If a participant made equal-sized deposits each month, then deposit lumpiness would be 1.00. If deposits vary in amount, or if some months have no deposits, then the ratio increases away from 1.00.
• Proportion of savings goal. The ratio of the average monthly deposit to the monthly savings goal. (The word goal here represents the IDA program goal as defined by the maximum matchable amount of savings, not the participant's goal.) The monthly savings goal is taken as one-twelfth of the annual maximum potential matched deposit, as set by the program. Thus the proportion of savings goal indicates the closeness of actual saving behavior to the behavior that would take full advantage of the incentives offered by the program. A ratio of 1.00 implies that on average, a participant saved the maximum matchable amount. (Under such a scenario, there may have been months in which the participant's deposits exceeded the program savings goal.)
• Proportion of savings goal over time. The number of dollar-months saved divided by the number of dollar-months that would have been saved had the participant made a deposit equal to the annual maximum potential matched deposit on the first day of each year. (A dollar-month is a dollar held in an account for a month. For example, a deposit of 2 dollars withdrawn after three months is six dollar-months of saving.)
  

 top of page

---

2.4 Cost Estimate

This section presents the costs associated with program and participant tracking and monitoring. The key assumptions of these cost estimates are described below.

2.4.1 Reporting versus evaluation

In preparing these cost estimates, we have distinguished between the "reporting" and "evaluation" purposes to be served by the tracking and monitoring of program and participant information.

The "reporting" purpose of tracking and monitoring is to provide the grantee-by-grantee descriptive information needed for annual progress reports to HHS, as specified in items (1) through (5) of Section 412(a) of the Act:

(1) The number and characteristics of individuals making a deposit into an individual development account;

(2) The amounts in the Reserve Fund established with respect to the project;

(3) The amounts deposited in the individual development accounts;

(4) The amounts withdrawn from the individual development accounts and the purposes for which such amounts were withdrawn; and

(5) The balances remaining in the individual development accounts.

The "evaluation" purpose of tracking and monitoring pertains to specific aspects of both Section 412 and Section 414 of AFIA. Under Section 412, the relevant evaluation-related language appears in items (6) through (8) of Section 412(a). These items specify that the annual progress reports from grantees to HHS shall include:

(6) The savings account characteristics (such as threshold amounts and match rates) required to stimulate participation in the demonstration project, and how such characteristics vary among different populations or communities;

(7) What service configurations of the qualified entity (such as configurations relating to peer support, structured planning exercises, mentoring, and case management) increased the rate and consistency of participation in the demonstration project and how such configurations varied among different populations or communities; and

(8) Such other information as the Secretary may require to evaluate the demonstration project.

Under Section 414, the tracking and monitoring information will be used to address the second "factor to evaluate," as specified in item (2) of Section 414(b): "the savings rates of individuals in the demonstration project based on demographic characteristics including gender, age, family size, race or ethnic background, and income."

Our approach to estimating the evaluation-related costs is to include the costs of data analysis and presentation of findings with respect to items (6) through (8) of Section 412(a) and item (2) of Section 414(b). The costs to be incurred in the collection of tracking and monitoring data at each AFIA-funded program are not included here as an "evaluation" cost. We provide an estimate below of the data collection costs to the reporting organization and the associated staffing requirements for grantees. The statutory language specifies that such grantee costs are to be funded, at least in part, out of the reserve funds established by each grantee.[7]

2.4.2 Reporting-related costs

The costs to the reporting organization for planning and implementing the collection of tracking and monitoring data have been estimated on an annual, per-grantee basis. This estimate of $5,604 per grantee per year, shown in Exhibit 2-1, is based on the following assumptions.

• Technical assistance to grantees to enable the consistent collection of tracking and monitoring information from all grantees will be the responsibility of a "reporting organization." This organization will also be responsible for aggregating such data nationally.
• The tracking and monitoring system used by all grantees is the Management Information System for Individual Development Accounts (MIS IDA), developed by the Center for Social Development of Washington University in St. Louis, or an equivalent system.
• Planning activities include organizing grantee-level personnel and coordinating data collection with the grantee staff. Additional technical assistance is assumed for multi-site grantees.
• Grantee staff will collect program and participant data at multiple intervals during a given year. Data are collected about the program design and entered into MIS IDA at the beginning of program implementation and updated as the program design changes. Data are collected from the IDA participant at program enrollment. On a monthly or quarterly basis, participant account statement information is also recorded. Every six months, grantees update the participant information and record changes in demographics, income, assets, and liabilities. The assumed annual staffing requirement for each grantee is 1,000 hours (amounting to approximately a 0.50 full-time-equivalent staff person), based on 4 hours annually per participant times 250 participants per grantee (reflecting the characteristics of an average grantee in the FY 1999 cohort and the experience of program staff now using MIS IDA under the American Dream Demonstration).
• Grantee-level data collection through MIS IDA will include the use of MIS IDA QC , a quality control database that generates reports of missing values and potential data errors. Each grantee will use MIS IDA QC to assist in data cleaning. After the reports indicate a clean data set, each grantee will export their data out of MIS IDA and send their files to the reporting organization for aggregation. For most grantees, telephone support will be sufficient to complete this process (as opposed to an onsite visit). Staff at the reporting organization will run similar quality control reports to verify the quality of the data. If potential errors are found, the grantee will be notified of the errors so that grantee staff can correct them and re-send the data set.
• For one-half of the grantees, it will be necessary for an associate-level member of the reporting organization to make an annual site visit, to address data problems.
• The data collected by grantees will be transmitted to the reporting organization every six months. This semi-annual data collection will assist in data quality control and will prompt grantees to perform participant updates on a timely basis.
• The cost structure for the reporting organization is assumed to be the same as for the evaluation contractor, with respect to inflation adjustments, fringe, overhead, G&A, and fee.

Note that the estimates shown in Exhibit 2-1 reflect the data collection costs to the reporting organization. No attempt has been made here to determine the extent to which such costs would be borne by non-federal sources or federal funds and, if the latter, whether this would be through an earmarked percentage of each grantee's AFIA funding or through separate federal administrative funding.

Exhibit 2-1 Program and Participant Monitoring and Training -
Reporting Organization - Summary by Task
Total Estimated Costs Per Grantee Per Year

Item			Rate	Task 1 Planning		Task 2A Data Collection		Task 2B Data Collection - Grantee Visits		Total
				Units	Cost	Units	Cost	Units	Cost	Units	Cost
Staff Labor
		Senior		1	$103	0	$0	0	$0	1	$103
		Associate		4	$291	8	$767	8	$399	20	$1,457
		Junior		4	$0	16	$0	0	$0	20	$0
	Labor Inflation Adjustment		4%		$16		$31		$16		$62
	Subtotal Staff Labor				$411		$798		$415		$1,623
	Fringe and Overhead				$452		$879		$457		$1,788
Total Staff Labor				9	$863	24	$1,676	8	$872	41	$3,411
Other Direct Costs
	Travel				$0		$0		$788		$788
	Telephone and Computer				$42		$93		$39		$174
	Duplicating and Delivery				$2		$26		$13		$40
	ODC Inflation Adjustment		3%		$1		$4		$25		$30
Total Other Direct Costs					$45		$122		$865		$1,032
G&A and Fee					$237		$470		$454		$1,161
Total Estimated Costs					$1,146		$2,268		$2,190		$5,604

-Task 2B reflects the cost of half a trip to each grantee, assuming a two-day trip to each grantee to provide additional technical assistance is conducted every two years.

The cost estimate of $5,604 per grantee per year can be interpreted as follows. With forty grantees in the FY 1999 cohort, the annual data collection cost to the reporting organization would be $224,000 for this cohort. The additional cost for other cohorts would depend on the number of grantees per cohort.

2.4.3 Evaluation-related costs

As noted above, the tracking and monitoring data will be used to support analyses under both Sections 412 and 414 of the Act. The estimated cost of such activities, shown in Exhibit 2-2, is based on the following assumptions:

• Section 412 analysis will include descriptive data on each grantee as well as data aggregated across grantees. Multivariate analysis will be performed to show patterns of program participation and account usage across grantees. The interim evaluation reports (submitted in September of 2001 through 2004) will present the findings of this analysis across grantees, as well as appendices with descriptive data on each grantee's IDA program.[8]
• Under Section 414, multivariate analysis will be performed with respect to savings patterns by demographic characteristics of participants. The findings of this analysis will be included in the interim evaluation reports.
  
  

Exhibit 2-2 Program and Participant Monitoring and Training -
Estimated Costs Per Grantee Per Year

Item			Rate	Year 1		Year 2		Year 3		Year 4		Total
				Units	Cost	Units	Cost	Units	Cost	Units	Cost	Units	Cost
Staff Labor
		Class I - Senior		145	$15,000	145	$15,000	145	$15,000	145	$15,000	580	$60,001
		Class II - Associate		433	$21,560	433	$21,560	433	$21,560	433	$21,560	1730	$86,241
		Class III - Intermediate		0	$0	0	$0	0	$0	0	$0	0	$0
		Class IV - Junior		130	$2,991	130	$2,991	130	$2,991	130	$2,991	520	$11,965
		Class V - Clerical		100	$2,112	100	$2,112	100	$2,112	100	$2,112	400	$8,448
	Labor Inflation Adjustment		4%		$1,667		$3,400		$5,202		$7,077		$17,346
	Subtotal Staff Labor				$43,330		$45,063		$46,866		$48,741		$184,000
	Fringe and Overhead				$47,733		$49,642		$51,628		$53,693		$202,695
Total Staff Labor				808	$91,063	808	$94,705	808	$98,494	808	$102,433	3230	$386,695
Other Direct Costs
	Survey Direct Costs				$0		$0		$0		$0		$0
	Travel				$0		$0		$0		$0		$0
	Telephone and Computer				$5,113		$5,113		$5,113		$5,113		$20,453
	Duplicating and Delivery				$1,140		$1,140		$1,140		$1,140		$4,560
	Payments to Respondents				$0		$0		$0		$0		$0
	ODC Inflation Adjustment		3%		$188		$381		$580		$785		$1,933
Total Other Direct Costs					$6,441		$6,634		$6,833		$7,038		$26,946
G&A and Fee					$25,487		$26,490		$27,532		$28,616		$108,126
Total Estimated Costs					$122,991		$127,829		$132,859		$138,087		$521,766

 top of page

---

Notes

[7] Section 407 of the Act indicates that one of the uses of the reserve fund established by each grantee is to "provide the research organization evaluating the demonstration project under Section 414 with such information with respect to the demonstration project as may be required for the evaluation." Section 407 also indicates that "not less than 2 percent" of the AFIA grant to each funded program is to be used for this purpose. [Return to Text]

[8] As noted in Chapter 1, we assume that these reports will meet the requirement under Section 414(d)(1) of the Act for periodic reports from the Secretary of HHS to the Congress. [Return to Text]