Detailed Information on the
Weatherization Assistance Assessment

Conduct an independent analysis of the cost effectiveness of the program.

Address recommendations identified in an audit by the Department's Inspector General.

Measure: Number of low-income family homes weatherized annually with partial support from DOE.

Explanation:Annual targets are adjusted based on appropriations and policy changes, such as the decision to fund the lead-safe weatherization protocol in 2003, which increased the cost per home.

Measure: Average DOE cost per home weatherized

Explanation:The maximum average cost per home weatherized is determined by statute formula and is shown in the Target column. Actual average costs should not exceed these values.

Measure: Program benefit-cost ratio excluding non-energy benefits. This ratio represents the discounted value (3.2 percent discount rate) of energy saved divided by program costs (excluding some management costs). Internal program assessment; not conducted by independent third party.

Explanation:The ratio depends in part on EIA-estimated long-term energy prices at the time of the assessment and average energy savings per household of 29.1 Btu. Estimates of the B/C ratio tested for the 90 percent confidence range of MBtu savings and various price scenarios range from 1.19 to greater than 2 but in no event are less than 1.

Measure: Average household natural gas savings after weatherization (90 percent confidence interval in parentheses) in thousands of British thermal units (MBTU)

Explanation:Point results for the the 1996, 1999, and 2002 Metaevaluations all fall within the 90 percent confidence range of the most recent Metaevaluation of weatherization program results.

Is the program purpose clear?

Explanation: The statutory purpose of the program is to increase energy efficiency, reduce total residential expenditures, and improve the health and safety of qualifed low-income persons.

Evidence: 42 U.S.C. 6851-6872. Program was last reauthorized in 1998 for 5 years in Public Law No. 105-388. Program is up for reauthorization in 2003. Funding has been appropriated every year since inception in 1976.

Does the program address a specific and existing problem, interest, or need?

Explanation: Low-income families do not have resources for capital investments in energy efficiency that will return savings over a long time period (20 years or more). Low-income persons spend 4x more on energy (proportion of income) than others.

Evidence: Weatherization Works: Final Report of the National Weatherization Evaluation 1994 (ORNL/CON-395).

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The program is not completely distinct from LIHEAP, which allows 15 percent of funds to be used for energy conservation measures. However, the weatherization program is the only one to solely address energy retrofits for low-income families. States are unable to support such a program alone, although many provide significant cost sharing.

Evidence: Most State offices transfer LIHEAP funds to their weatherization offices, and use DOE Weatherization regulations to implement the program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Weatherization Assistance Program infrastucture has been in place for more than 25 years and has generally served those within the network well. Program has made improvements over the years as appropriate, and the Department's Inspector General (IG) has recently suggested other modest improvements.

Evidence: DOE IG audit found the program was properly administered, but recommended improvements in reporting administrative costs and reporting performance data on the number of homes weatherized. DOE IG Audit Report Number OAS-L-03-15.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program prioritizes elderly, families with children, persons with disabilities, high residential energy users, and households with high energy burden. State and local agencies may choose among these groups.

Evidence: State and local agencies can select the most appropriate eligibility priority to target those most in need. It also permits them to target the priorities of their leveraging partners to maximize avaiilable resources.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program established long term objective of weatherizing 1.2 million low-income family homes by 2011. The program also added long-term efficiency goals, including maintaining a positive benefit-cost ratio and maintaining relatively constant energy savings per household. While 10 percent of program funds are set aside for training and technical assistance, these activities are considered a program support function, and are therefore not captured in a separate performance measure.

Evidence: FY 2004 Budget. The goal assumes outyear funding according to constant maximum cost per home weatherized.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program sets targets for and tracks the number of homes weatherized annually, which directly measures progress toward the long term goal.

Evidence: FY 2004 Budget. The goal assumes outyear funding according to constant maximum cost per home weatherized.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The program's main annual performance measure is the number of homes weatherized. The program has also adopted a new annual measure, the average cost per home weatherized, to compare to statutory limits.

Evidence: FY 2004 Budget.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: The program has baseline performance data and has set reasonable targets for the number of homes weatherized.

Evidence: FY 2004 Budget.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program has specific application and reporting requirements for State and local agencies to report on a quarterly basis on the expenditure of funds and performance in terms of number of homes weatherized. The program does not publish this information, but uses it internally for management of the program.

Evidence: Quarterly Financial Status Reports on Production and Expenditure of Weatherization Funds.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program does not conduct annual evaluations on a national basis because of the high cost of such evaluation and the limited amount of change that occurs in program activities from year to year. The program has contracted with Oak Ridge National Laboratory (ORNL) to devise evaluation methodologies and report periodically on program results based on state grantee-level performance evaluation. ORNL has also conducted selective evaluation activities designed to inform program management of performance characteristics in areas in which the program performance has been below average (hot climate zones) or in areas in which there has been growing strategic program interest but little evaluation data. The latter includes base load electric measures as well as nonenergy benefits. To assure independence, the program should consider using an alternative contractor in future assessments, or at least having future ORNL reports assessed by a third party. (Peer reviewers of ORNL weatherization reports are generally employees of ORNL or DOE.)

Evidence: State-level Evaluation of WAP 1990-1996: A Meta Evaluation of 17 States Evaluations - 1997 (ORNL/CON-435); Metaevaluation of National Weatherization Assistance program Based on State Studies, 1996-1998 (ORNL/CON-467); Metaevaluation of National Weatherization Assistance Program Based on State Studies, 1993-2002 (ORNL/CO-488); Nonenergy Benefits from the Weatherization Assistance Program: A Summary of Findings from the Recent Literature, 2002 (ORNL/CON-484)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests reflect the program's needs to achieve the performance target for the number of homes weatherized. However, budget documents do not clearly indicate the full costs of achieving the program goals. Salaries, benefits, and other admininstrative expenses to support the program are included in a separate budgetary line item ("Policy and Management"). EERE does not report the allocation of Policy and Management funding to the various programs it supports. In addition, a 2003 Inspector General report found that "certain organizations inappropriately charged expenses such as administrative staff, office rent, and administrative supplies as direct program costs and thus understated total administrative costs."

Evidence: FY 2003 Congressional Budget Justification identifies program funding allocated for training and technical assisstance (10 percent, including 1.5 percent for DOE and 8.5 percent for States), but not the amount of Policy and Management funding used to implement the program. DOE IG Audit Report Number OAS-L-03-15.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has taken steps to address deficiencies. For example, the current strategic plan (Weatherization PLUS, 1999) does not adequately address weatherization improvements in hot-climate areas. The program has undertaken a Hot-climate Initiative designed to improve program performance in those States where targeting cooling measures may provide greater energy savings. The Hot-climate Initiative will be worked into an updated strategic plan.

Evidence: Senior Headquarters and Regional Office program staff have met to discuss updates to the strategic plan to address new issues, such as targeting cooling measures. In the fall of 2003, program staff will meet with weatherization network stakeholders to update and revise, as necessary, the program strategic plan.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects data quarterly on the production and expenditures of DOE funds. Regional Offices review these reports and monitor production performance and fiscal performance as well as through site visits to State and local offices. DOE and the States use the information on these reports to implement remedies and to direct future training and technical planning activities to correct any deficiencies noted.

Evidence: Quarterly Reports on Production and Expenditure of Weatherization Funds.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: DOE regulations do not permit setting production quotas on the States. Each State's Annual Plan indicates the level of production the State will attain, which must generally be consistent with levels of production in previous years, considering funding levels and policy changes as appropriate.

Evidence: Application and Reporting Requirements for the DOE Weatherization Assistance Program.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: DOE grants to States are made "on time," meaning before State program years commence (April 1 for 33 States, July 1 for 17 States). Quarterly reports on expenditures generally indicate that State awards to local agencies are made within a reasonable time. Unobligated balances brought forward from FY 2003 were $97,000, less than one percent of the FY 2003 appropriation of approximately $224 million.

Evidence: Quarterly Reports on Production and Expenditure of Weatherization Funds.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has three efficiency measures which can be used to assess programmatic efficiency and cost effectiveness.

Evidence: See Measures Tab.

Does the program collaborate and coordinate effectively with related programs?

Explanation: DOE staff communicate regularly with staff of related programs in HHS (Low-Income Home Energy Assistance Program) and HUD (Lead Paint Removal Program). At the State and local levels, there are many instances of shared resources and joint projects.

Evidence: LIHEAP regulations. HUD's Energy Action Plan (May, 2002). Joint Research Project on Lead with HUD designed to evaluate the impact of lead in homes weatherized under the program (2003).

Does the program use strong financial management practices?

Explanation: DOE Regional Offices and the Golden Field Office have financial management systems in place and Headquarters staff use a customized IT system (WinSAGA) to provide oversight and mangement of financial expenditures by the States. However, a recent Inspector General report found that some organizations inappropriately charged administrative expenses as direct program costs, a practice which DOE's systems apparently were unable to identify.

Evidence: DOE IG Audit Report Number OAS-L-03-15.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The National Association of Public Administrators (NAPA) found dozens of management deficiencies in the program's bureau (the Office of Energy Efficiency and Renewable Energy, or EERE) in a review published in 2000. EERE provided evidence that it addressed some of management deficiencies identified by NAPA, and has prepared a Management Action Plan that will address many of the remaining findings. While a few NAPA recommendations have not been addressed, in general, EERE has taken meaningful steps to address most deficiencies. However, the Department's Inspector General (IG) found that some organizations receiving weatherization funds inappropriately charged administrative-type expenses as program operating costs, and that some States combined the results of weatherization efforts completed with HHS Low Income Home Energy Assistance Program (LIHEAP) funds with those completed with Departmental funds, which could distort reported program results. It is too early to evaluate the impact of the program's corrective actions to the IG report.

Evidence: DOE IG Audit Report Number OAS-L-03-15. A Review of Management in the Office of Energy Efficiency and Renewable Energy (NAPA, 2000). EERE Letter Report in Response to NAPA Review (July 11, 2001), EERE Management Action Plan (August 2003). DOE IG Audit Report Number OAS-L-03-15.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: DOE has specific program oversight requirements that the DOE Regional Office monitor State compliance and that the State, in turn, monitor the local agancy compliance with program regulations.

Evidence: Weatherization Program Notice 01-6: Updated Weatherization Assistance Program Monitoring Policy (January, 2001), Sample monitoring report

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program summarizes annual performance data on State-by-State basis and publishes it on the internet. While the key measure (number of homes weatherized) is reported for each State, the program should consider (1) reporting on efficiency measures on a State-by-State basis as well, and (2) drafting the summaries using more objective language.

Evidence: www.eere.energy.gov/weatherization/state_activities.html

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: The program is on target to meet the long-term goal for number of homes weatherized by 2011.

Evidence: FY 2002 Performance and Accountability Report.

Does the program (including program partners) achieve its annual performance goals?

Explanation: In recent years, the program has met its annual performance targets for number of homes weatherized. For example, in 2001, the performace estimate was 75,350 and the actual units completed by the States was 77,697. However, a 2003 Inspector General report notes that some States combined the results of weatherization efforts funded by the HHS Low Income Home Energy Assistance Program (LIHEAP) with those completed with Departmental funds, which may distort upwardly the programs reported results.

Evidence: FY 2001 Performance and Accountability Report. DOE IG Audit Report Number OAS-L-03-15.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: Benefit-cost ratio rose from 1.06 in 1989 to 1.79 in 1996, and then declined to 1.51 and 1.30 in 1999 and 2002, respectively. These estimates depend largely on EIA estimated long-term energy prices. Given the 90 percent confidence range of actual energy savings per household (which followed a similar pattern), the benefit-cost ratios are statistically similar (except for 1989, which is lower). However, there are two factors that contribute to a lower rating on this question than would otherwise be expected. (1) The benefit-cost ratio calculations do not include administrative expenses appropriated under "Policy and Management" portion of the appropriation. It is also not clear whether the Training and Technical Assistance portion of program funds (10 percent) are included the calculation. (2) The Department's 2003 Inspector General report notes that some States combined the results of weatherization efforts funded by the HHS Low Income Home Energy Assistance Program (LIHEAP) with those completed with Departmental funds, which may distort upwardly the programs reported benefit-cost ratio.

Evidence: State-level Evaluation of WAP 1990-1996: A Meta Evaluation of 17 States Evaluations - 1997 (ORNL/CON-435); Metaevaluation of National Weatherization Assistance program Based on State Studies, 1996-1998 (ORNL/CON-467); Metaevaluation of National Weatherization Assistance Program Based on State Studies, 1993-2002 (ORNL/CO-488); DOE IG Audit Report Number OAS-L-03-15. Internal program memo on the benefit-cost ratios for the Weatherization Assistance Program (Joel Eisenberg, April 8, 2003).

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: State, non-profit, and private sector funding is used to augment Federal funding provided for weatherization services. There does not appear to be a weatherization program operating independent of Federal support to which the Weatherization program can be compared.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Despite concerns noted in reponse to Question 4.3, Oak Ridge National Lab (ORNL) reports on program performance generally indicate that the program is effective. The Vermont State Auditor generally gave a positive assessment of the Weatherization program based on that State's perspective and performance. A 2002 ORNL report attempting to quantify additional non-energy benefits suggests that the societal benefit-cost ratio of the program is 2.7 (but such studies can be controversial).

Evidence: State-level Evaluation of WAP 1990-1996: A Meta Evaluation of 17 States Evaluations - 1997 (ORNL/CON-435); Metaevaluation of National Weatherization Assistance program Based on State Studies, 1996-1998 (ORNL/CON-467); Metaevaluation of National Weatherization Assistance Program Based on State Studies, 1993-2002 (ORNL/CO-488); Vermont State Auditor's Review of the Weatherization Assistance Program (19998); Nonenergy Benefits from the Weatherization Assistance Program: A Summary of Findings from the Recent Literature, 2002 (ORNL/CON-484)