Detailed Information on the
Economic Research Service Assessment

Integrate research and market analysis for commodities.

Implement National Research Council recommendations for ARMS.

Determine the impact of its research by surveying users to identify which products were found useful in decision making.

Track those measures that have only baseline or partial data to ensure performance is improving.

Conduct an independent evaluation of the Agricultural Management Survey (ARMS).

Measure: Customer satisfaction with the ERS website

Explanation:In recent years, ERS recast its information dissemination and communications channels to adopt a webcentric approach to communicating with customers. As a result, all ERS research, data, and other information disseminated by the agency is available through the ERS website. This measure is an indicator of customer satisfaction with the ERS website using a survey based on the American Customer Satisfaction Index (ACSI). The measure tracks satisfaction of website users and provides a basis for comparison with similar government and private-sector websites. The target for this measure is at or above the average rating for government websites in the Information/News category.

Measure: ACSI Customer Satisfaction Rating

Explanation:This measure is designed to assess the satisfaction of private and other external customers with the relevance, usefulness, and accessibility of ERS research, data, and analysis, as measured by the American Customer Satisfaction Index (ACSI). This measure tracks relevance and usefulness of ERS research, analysis, data products, and services, as determined through a survey of agency customers using the ACSI. The survey will be conducted in July 2005, with final results available in September 2005.

Measure: Percent of requested analysis delivered on time

Explanation:ERS provides a broad range of research, data, and analysis for public and private decisionmakers to use in their analysis of economic issues affecting the food and agricultural sector. Throughout the year, policy officials from USDA agencies or outside of the department request that ERS provide analysis on a specific question of interest to the requestor. Such questions, referred to as 'Staff Analysis', provide policymakers with assessments relevant to their particular questions, and are typically requested based on quick turnaround. This measure tracks the timeliness of responses by ERS to requests for short-term, tailored research, analysis, and data from government policymakers. A measure of agency performance is the timeliness with which responses are provided to the customer. Over the last 5 years, ERS staff analysis has met predetermined deadlines for over 90 percent of all such requests

Measure: Policy Official Satisfaction Survey

Explanation:This measure is designed to assess the satisfaction of USDA and other government decisionmakers with the relevance and usefulness of requested analysis. ERS provides a broad range of research, data, and analysis for public and private decisionmakers to use in their analysis of economic issues affecting the food and agricultural sector. Throughout the year, policy officials from USDA agencies or outside of the department request that ERS provide analysis on a specific question of interest to the requestor. Such questions, referred to as 'Staff Analysis', provide policy officials with assessments relevant to their particular questions, and are typically requested for quick turnaround. This measure assesses requestors' satisfaction with the usefulness of materials provided by ERS in response to their requests for short-term, tailored research, analysis, and data. The survey has been pilot-tested and will be launched as a web-based survey in July 2005.

Measure: Index of ERS Product Releases per Staff Year

Explanation:During the 2001-05, the number of products released increased from 238 to 365. During that time staff years (SYs) declined from 491 SYs to 430 SYs. Using these numbers and basing them to 2001=1 results in product release efficiency going 1.0 in 2001 to 1.75 in 2005. Even if the number of products remained constant over time, productivity would need to increase to sustain production levels. The ratio is calculated as [number of products released]/[total SYs]. Every year is then compared to the ratio value in 2001. The measure defines outputs as all published items, web-based briefing rooms, and data products produced by ERS staff during the fiscal year. Staff years include all actual ERS employees, whether research, support, or administrative.

Measure: Portfolio Review Score -- Qualitative assessment by external experts of the relevance, quality, and performance of ERS research portfolios to enable better informed decisions on food and agricultural policy issues.

Explanation:A series of independent expert review panels will conduct a cycle of reviews over 5 years to evaluate the effectiveness of the ERS program of economic research and analysis to enable better informed decisions on food and agricultural policy issues. The reviews will include a capstone review of the entire program at the end of the review cycle. The program components scheduled for review are: (a) food choices, diet, safety, and health (Sep. 21-23, 2005), (b) agricultural competitiveness, (c) natural resource conservation and management, and (d) farm and rural well-being. In each review, the external panel will assess the relevance, quality, and performance of program plans, activities, and accomplishments. This assessment will include an evaluation using a quantitative analysis tool to rate portfolio effectiveness on a multi-category scale (excellent, adequate, needs improvement). The panel recommendations will be used in agency strategic planning and priority setting.

Is the program purpose clear?

Explanation: The mission of the Economic Research Service (ERS) is to provide economic analysis on efficiency, efficacy, and equity issues related to agriculture, food, natural resources, and rural development to improve public and private decisionmaking. Activities to support this mission and the USDA strategic goals involve research and development of economic and statistical indicators on a broad range of topics; including, but not limited to, global marketing conditions, trade restrictions, agribusiness concentration, farm and retail food prices, food assistance, foodborne illnesses, food labeling, nutrition, agrichemical usage, livestock waste management, conservation, genetic diversity, technology transfer, and rural employment.

Evidence: ERS was established in 1961 from components of the former Bureau of Agricultural Economics, principally under the authority of the Agricultural Marketing Act of 1946 (7 U.S.C. 1621-1627). ERS's portfolio was expanded to establish extramural research activities on the economics of food assistance (1998) and invasive species (2003). Consolidated Appropriations Act, 2005 (Public Law 108-447). ERS has five strategic goals, which correspond to USDA's strategic goals.

Does the program address a specific and existing problem, interest, or need?

Explanation: ERS supports USDA's strategic goals by supplying research that enhances policymakers' understanding of the issues. ERS research and analysis enhances understanding of economic issues by policymakers, regulators, and others who shape public debate affecting global markets and U.S. agriculture; farm and rural well-being; food safety and security; nutrition and health of the U.S. population, including nutrition assistance programs; and farm, natural resource, and rural policies and programs to protect the environment while improving agricultural competitiveness and economic growth.

Evidence: Program authorization, strategic plans, and performance documents for USDA, REE, and ERS are located at www.usda.gov/ocfo/usdasp/usdasp.htm (USDA), www.csrees.usda.gov/ree/strategic_plan.htm (REE), www.ers.usda.gov/AboutERS/mission.htm (ERS), and Principles and Practices for a Federal Statistical Agency, 3rd Edition, www.nap.edu/catalog/11252.html. ERS is a key participant in USDA's Interagency Commodity Estimates Committee (ICEC) process. The process is described at: www.ers.usda.gov/Publications/Outlook/process.htm and www.usda.gov/agency/oce/waob/mission.htm.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ERS provides unbiased research and analysis that is national and international in scope. The research and analysis is presented in a broader policy context, unlike program implementation and administration agencies. The scope of ERS' intramural research program enables it to conduct long-term, innovative, high-risk, strategic research that cannot be conducted by any other agency or industry in the United States. ERS, as an intramural agency, has the unique ability, authority, and expertise to respond quickly and effectively to critical national and/or international issues and emerging problems. The program is effectively coordinated with other research efforts conducted across the Federal Government, including within USDA (AMS, APHIS, CSREES, FAS, FSA, FNCS, GIPSA, NASS, NRCS, and WAOB) and with other Federal agencies (DHS, EPA, NIH, NSF, and FDA), as well as with academia and industry.

Evidence: ERS responds to emerging issues with data and analysis for policy officials and other customers. Examples of successful partnerships with other agencies include conservation policy design (NRCS and FSA), creating a component to the National Health and Nutrition Examination Survey (HHS, FNS, and CNPP), and the economics of invasive species management (APHIS). ERS augments its research capacity with 93 cooperative agreements, 14 research grants, and 26 Memorandums of Understanding (MOUs). ERS is a key participant in USDA's Interagency Commodity Estimates Committee (ICEC) process. The process is described at www.ers.usda.gov/Publications/Outlook/process.htm and www.usda.gov/agency/oce/waob/mission.htm.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The research quality of the program's design and products is ensured through a rigorous peer review process that includes internal and external experts. ERS employs a selective hiring and promotion process to encourage and provide adequate incentives for individuals to perform at the highest levels. ERS's research methodologies are continually vetted by the broader economic research community through conferences, internal seminars, and publication in peer-reviewed professional journals. The National Agricultural Research, Extension, Education, and Economics Advisory Board (NAREEEAB) provides advice to ERS on top priorities and policies for food and agricultural research and analysis. ERS ensures that its data collection efforts meet quality standards and users' critical data needs by seeking review and recommendations from stakeholders.

Evidence: The ERS program is designed to be effective and efficient and to respond to current and emerging needs. The program is validated by mechanisms, including stakeholder meetings and independent external program and peer review. The result of ERS' program is timely, relevant, and high-quality research. ERS has a record of working collaboratively with other Federal agencies, industry, and universities. Quality and relevance is ensured through the ERS Information Quality Guidelines, www.ers.usda.gov/AboutERS/QualityGuidelines.htm and the NAREEEAB, www.nareeeab.com/. The National Academy of Sciences (NAS) has provided guidance for food security and consumption data in www.nap.edu/catalog/11227.html and www.nap.edu/catalog/11126.html. General NAS feedback is in books.nap.edu/catalog/6320.html, www.nap.edu/catalog/11252.html and books.nap.edu/catalog/10585.html. The Economist Position Classification System ensures quality of research staff.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: ERS produces different types of reports for different target audiences. For example, ERS publishes information bulletins for lay readers, market outlook reports for industry users, research reports for technical audiences, and research briefs for all users. ERS offers equal access to bulletins, report and data to all users via its website. ERS customers sign up on the website (by topic and report type) to receive e-mail notifications when a report or data product is released.

Evidence: ERS received the Blue Pencil Award from the National Association of Government Communicators in 2003 for its Amber Waves magazine, www.nagc.com/awards/index.html#bluepencil. The ERS website, www.ers.usda.gov/, provides all the reports and data released by ERS, which are listed on the ERS Calendar of Releases, www.ers.usda.gov/calendar.htm. Customers can provide feedback via the ERS Customer Service Hotline, 1-800-694-5000, or via e-mail, contact@ers.usda.gov. ERS participates in the USDA Data Users Meeting, www.usda.gov/nass/events/forum/forum102004.htm, to receive feedback from key users of our market analysis. The annual Food Assistance Priority-Setting Research Conference, www.ers.usda.gov/Briefing/FoodNutritionAssistance/ConferenceMaterials/conference2005.htm, provides a forum to discuss emerging issues and research priorities in this area.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ERS's five strategic goals align with USDA and REE mission area strategic goals and provide a guide to ensure that ERS's program and products promote results and accountability in accomplishing its mission. Of the performance measures used to track the program's progress, two are long-term measures. The first measure, Portfolio Review Score, is based on evaluation of the relevance, quality, and performance of the ERS program of economic research and analysis using external expert review. The annual panels will evaluate the agency's success in enabling better informed decisions on food and agricultural policy issues. The panel recommendations will be used in agency strategic planning and priority setting. The second measure is an ACSI Customer Satisfaction survey results used to assess the usefulness of ERS' research for its external customers.

Evidence: This measure is based on a cycle of regular expert reviews of the ERS program. A schedule of reviews has been established for the components of the ERS program. In each topic area, the expert panel will assess the relevance, quality, and performance of program plans, activities and accomplishments. This assessment will include an evaluation that uses a quantitative analysis tool to rate the program effectiveness on a multi-category scale (excellent, adequate, needs improvement). The components for review are farm and rural well-being; agricultural competitiveness; natural resource conservation and management; and diet, nutrition, food safety, and consumption. At the end of the cycle, the individual assessments will provide input for a capstone review, which will include members of the previous panels. The capstone review will address the broader issues of resource allocation, priority setting, and mission effectiveness facing the agency. ERS Explanatory Notes.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The purpose of the Portfolio Review process is to rate the relevance, quality, and performance of the agency's research portfolio. To accomplish that goal, the program components of the agency (farm and rural well-being; agricultural competitiveness; natural resource conservation and management; and diet, nutrition, food safety, and consumption) will be evaluated on a 5-year cycle for relevance, quality, and performance. After the components have been reviewed, the individual assessments will provide input for a capstone review of agency resource allocation, priority setting, and mission effectiveness. The targets related to customer satisfaction reflect levels above ACSI average benchmarks for government performance (www.theacsi.org/government.htm).

Evidence: Each component's relevance, quality, and performance will be evaluated using a rigorous quantitative analysis tool to rate effectiveness on a multi-category scale (excellent, adequate, needs improvement). An overall rating for each component will be derived from the scores for relevance, quality, and performance. The annual targets and milestones for this long-term measure is a rating of "Excellent" by the independent panel. When the four components (farm and rural well-being; agricultural competitiveness; natural resource conservation and management; and diet, nutrition, food safety and consumption) of the ERS program have been reviewed, overall relevance, quality, and performance of the entire ERS program can be assessed by a capstone review, which will address the broader issues of resource allocation, priority setting, and mission effectiveness facing the agency.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ERS has specific annual performance measures and targets that measure the agency's movement toward its long-term strategic goals. Annual measures include three outcome (Policy Official Satisfaction Survey, Customer Satisfaction with the ERS Website, and Timeliness of ERS Information and Analysis on Current and Emerging Events/Issues), one output (Percent of Requested Analysis Delivered On Time), and one efficiency measure (Index of ERS Product Releases per Staff Year).

Evidence: Measures will provide evidence of the usefulness of ERS information and analysis to customers making decisions related to the five ERS strategic goals. ERS Strategic Plan, REE Strategic Plan, USDA Strategic Plan, and ERS Explanatory Notes.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baseline and targets have been established for all measures. Annual performance targets call for high levels of performance and continuing performance improvement. Baselines are established on recent performance, and targets have been established to routinely and continuously improve performance. ERS intends to review and update its performance measures annually to provide accountability for achieving ambitious goals.

Evidence: The measures are designed to provide evidence of the usefulness of ERS information and analysis to enabling better-informed decisions by ERS customers. The targets reflect the recognition that ERS needs to continuously improve agency performance to meet evolving customer needs. The targets related to customer satisfaction reflect levels above ACSI average benchmarks for government performance (www.theacsi.org/government.htm). The targets related to timeliness of information delivery (100%) and availability of issue analysis (83%-86%) reflect the goal of continued, high responsiveness to customer interests. The productivity targets for growth in outputs assume a continued 2.5% annual decline in staff and output growth of 1.5%. The increased rate of output coupled with a decline in staff implies a growth in output per staff person of 4% per year. ERS Strategic Plan, REE Strategic Plan, USDA Strategic Plan, and ERS Explanatory Notes

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ERS enters into research partnerships only when mutual benefit is clearly demonstrated and consistent with the ERS mission, Strategic Plan, and long-term and annual goals. Submitted proposals specify the scope of work, milestones, measures, outputs, and outcomes, along with performance monitoring procedures. For ERS's extramural programs, submission of proposals and acceptance of awarded funds requires commitment to working to accomplish annual and long-term goals.

Evidence: Cooperative research agreements meet long-term goals by requiring statements of mutual benefits, cost-sharing arrangements, and quarterly reporting. ERS cooperative agreements operate under USDA policy requirements which may be found at www.afm.ars.usda.gov/ppweb/04-154.pdf. The FANRP extramural research program is linked to USDA and stakeholder needs, see: www.ers.usda.gov/Briefing/FoodNutritionAssistance/2004FinalReport.pdf, page 2. Applicants for program awards must describe the relationship of proposed research to the program's priority research areas and to potential improvement in efficiency of USDA food assistance and nutrition programs, see: www.ers.usda.gov/Briefing/FoodNutritionAssistance/2005Application.pdf, page 15.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: A schedule of independent reviews has been established for the components of the ERS program, including a capstone review of the entire program at the end of the Portfolio Review cycle. The capstone review will address the broader issues of resource allocation, priority setting, and mission effectiveness facing the agency. In the program components reviewed (farm and rural well-being; agricultural competitiveness; natural resource conservation and management; and diet, nutrition, food safety and consumption), the independent panels will assess the relevance, quality, and performance of program plans, activities, and accomplishments. This assessment will include an evaluation using a quantitative analysis tool to rate program effectiveness on a multi-category scale (excellent, adequate, and needs improvement). The first review, ?Food Choices, Diet, Safety, and Health,? will be Sep. 21-23, 2005. Independent reviews of specific programs or projects within the overall portfolio are conducted as needed.

Evidence: The Portfolio Review evaluates key ERS program components over a 5-year cycle, ending with an Agency capstone review. Panel recommendations will be used in agency strategic planning and priority setting. The National Academy of Sciences (NAS) has provided guidance for food security and consumption data in Measuring Food Insecurity and Hunger, www.nap.edu/catalog/11227.html, and Enhancing the Data Infrastructure in Support of Food and Nutrition Programs, Research, and Decision Making, www.nap.edu/catalog/11126.html. Other independent evaluations include: Sowing Seeds of Change: Informing Public Policy in the Economic Research Service of USDA, books.nap.edu/catalog/6320.html; www.nap.edu/catalog/11252.html; and Frontiers in Agricultural Research: Food, Health, Environment, and Communities, books.nap.edu/catalog/10585.html.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The ERS budget-planning process ensures that the direction of the agency research activities meet annual and long-term performance goals. The explanatory notes developed during the budget process highlight how ERS budget requests are tied to annual and long-term performance measures. In addition, performance measures from both the strategic and annual performance plans are monitored to determine the maximum benefit from appropriated funding toward meeting pre-established goals. ERS budget requests depend on the review and approval of USDA's Budget and Performance Integration Board. This provides a secondary executive review to ensure alignment of funding and performance with USDA's strategic goals.

Evidence: ERS Explanatory Notes, ERS Strategic Plan, GPRA documents, and National Agricultural Research, Extension, Education, and Economics Advisory Board reports (www.nareeeab.com/publications/).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: ERS developed a Strategic Plan that specifically incorporates both PART and the Budget Performance Integration initiatives in the President's Management Agenda. The Strategic Plan aligns ERS priorities with USDA's Strategic Plan. Beginning in FY 2005, the ERS budget request followed the goals and performance measures contained in its Strategic Plan. The Plan is designed to better align the research program to producing tangible results that are useful to public and private decisionmakers. ERS provides research and analysis of issues across all five USDA strategic plan goal areas.

Evidence: The ERS Strategic Plan was designed to address the needs of the Budget Performance Integration initiative, PART, and USDA and REE Strategic Plan goals and objectives, and to link budget requests with specific performance measures. The Strategic Plan is available on the ERS website at www.ers.usda.gov/AboutERS/mission.htm.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: ERS's research and analysis is unique is national and international in scope. ERS provides research and analysis in a broader policy context, unlike program implementation and administration agencies. ERS, as an intramural agency, has the unique ability, authority, and expertise to respond quickly and effectively to critical national and/or international issues and emerging problems. The program is effectively coordinated with other research efforts conducted across the Federal Government, including within USDA (AMS, APHIS, CSREES, FAS, FSA, FNCS, GIPSA, NASS, NRCS, and WAOB) and with other Federal agencies (DHS, EPA, NIH, NSF and FDA), as well as with academia and industry.

Evidence: ERS fills critical gaps in research that compliment the work of other research and analysis entities and that reduce duplication. Examples of successful partnerships with other agencies include conservation policy design (NRCS and FSA), creating a component to the National Health and Nutrition Examinate Survey (HHS and CNPP), and the economics of invasive species management (APHIS). ERS augments its research capacity with 93 cooperative agreements, 14 research grants, and 26 Memorandums of Understanding (MOUs). Cooperative research agreements meet long-term goals by requiring a statement of mutual benefits, cost-sharing arrangements, and quarterly reporting.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Input from customers and stakeholders is obtained in developing the research agenda for the ERS program. External reviewers conduct prospective and retrospective reviews at different points in the 5-year Portfolio Review cycle. The goal of these processes is to evaluate, prioritize, and implement a mission-oriented research program that is tied to guiding budget requests and funding decisions. The ERS Strategic plan reflects goals of the Mission Area and USDA Strategic Plans. ERS has incorporated the OMB R&D Investment Criteria into its Strategic Plan and provides several mechanisms to ensure that the funds requested in its budget request are relevant to the needs of U.S. agriculture.

Evidence: ERS research and management practices use many methods to apply the research and development investment criteria. These practices, including priority-setting conferences, customer and stakeholder feedback, and internal assessments, are designed to ensure that the direction of agency research activities reflects current and anticipated needs. Research and analysis produced by the agency adheres to disciplinary standards to ensure the highest possible quality, and that the agency's research products are delivered in a way that is accessible to customers. The agency's budget decisions are also informed by an annual review of the REE mission-area portfolio by the NAREEE Advisory Board. The Board conducts the portfolio review with regard to its relevance to priorities and adequacy of funding and considering funding allocation across research programs. The Board's 2004 recommendations provided input to the REE mission area and to the ERS FY 2006 budget process.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ERS regularly collects and monitors information for managing progress on intramural and extramural research activities for meeting stated agency performance goals. The Senior Executive Team meets regularly to review performance information, modify and/or implement new program changes, and manage allocation of limited program resources. ERS uses its Program Information Management System (PIMS) to monitor all research projects, resource allocations, product development, and report review and clearance. Project reports on status, outputs, and impacts are provided annually to USDA's Current Research Information System (CRIS) for government accountability and public dissemination. Decisions made by peer review or management committees are based on quality of proposals and links to priority research objectives.

Evidence: ERS tracks intramural and extramural performance through the following mechanisms. ERS cooperative agreements require quarterly reporting; the guidance is in REE's General Administrative Policy for Non-Assistance Cooperative Agreements, www.afm.ars.usda.gov/ppweb/04-154.pdf. CRIS reports and documentation are at cris.csrees.usda.gov/. Terms and conditions for competitive awards for the Food Assistance and Nutrition Research Program (FANRP) and Program of Research on the Economics of Invasive Species Management (PREISM) specify quarterly progress and final reporting requirements. Annual reporting for FANRP is found in www.ers.usda.gov/Briefing/FoodNutritionAssistance/2004FinalReport.pdf. Administrative guidance is in Final FANRP Project Manager's Administrative Guide Book for the ERS Competitive Grants and Cooperative Agreements Program. PREISM guidance is at www.ers.usda.gov/Briefing/InvasiveSpecies/invasivespecies%2002_05awards.pdf.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Annual performance plans for managers include goals and measures that are linked to the ERS strategic goals. Annual performance evaluations are conducted with our cooperators to ensure work quality, timeliness, and efficiency. All managers and cooperative agreement Authorized Departmental Officer's Designated Representatives (ADODRs) are held accountable for contributing to and achieving the level of performance coinciding with ERS strategic efforts with regard to cost, scheduling, and outcomes for providing timely, accurate, and usable economic information and analysis.

Evidence: Each employee's performance plan (AD-435B) links project goals to the ERS strategic plan. These plans are developed between staff and management and progress is reviewed at mid-year and yearend. ERS cooperative agreements operate under policy requirements established for the Research, Education, and Extension mission area of USDA, which may be found at http://www.afm.ars.usda.gov/ppweb/04-154.pdf. These requirements include a statement of mutuality of interest between ERS and the cooperator, regular performance reporting to ERS, and a comparison of actual accomplishments with goals and objectives. Cooperative agreements are linked to strategic plan goals through language that identifies the need for the project and ties that need to a principal research focus of ERS. Accountability is ensured by requiring adherence to USDA's General Administrative Policy for Non-Assistance Cooperative Agreements, http://www.afm.ars.usda.gov/ppweb/04-154.pdf.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: ERS obligates funds in a timely and appropriate manner using the National Finance Center's (NFC) Financial Foundation Information System (FFIS). All obligations are consistent and aligned with the budget request. Unobligated balances at the end of the fiscal years are less than 1 percent of the annual appropriated agency budget.

Evidence: Monthly and quarterly FFIS obligation reports. ERS clean audits.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In recent years, ERS has made a series of strategic investments to improve information and data delivery to customers. These investments have transformed ERS into a webcentric organization for delivery of research and analysis. ERS complies with agency and Federal requirements to maximize cost-effectiveness in purchasing and procurement. ERS has an efficiency measure that compares product output with resources available. ERS' information management system, PIMS, provides data on inputs, status, outputs, and impacts of ERS projects. The system integrates all research and reporting functions. ERS manages IT improvements through an annual IT budget. Recent IT investments have focused on improving information dissemination. ERS contracts for agencywide services, such as mail, security and selected research. ERS participates in the USDA and REE Competitive Sourcing Working Groups.

Evidence: The ERS intranet, InsidERS, contains comprehensive documentation on agency policies and procedures, including the IT budget, PIMS, performance awards, strategic planning documents, agency budget explanatory notes, and IT investment plans. ERS annual performance and strategic plans are located at http://ers.usda.gov/AboutERS/. The ERS Online Strategic Plan and plans for future development are at http://ers.usda.gov/AboutERS/oursite/ Two ERS performance measures, Percent of Requested Analysis Delivered on Time and the Index of ERS Product Releases per Staff Year, are described in the performance measures section.

Does the program collaborate and coordinate effectively with related programs?

Explanation: ERS collaborates extensively with other Federal agencies both inside and outside USDA. For example, ERS collaborates with FNS in setting research priorities for food assistance programs and defining research responsibilities through the budget development process. ERS coordinates with NASS extensively on developing and implementing the ARMS survey and census of agriculture. ERS also coordinates with AMS, ARS, FAS, and NRCS, as well as HHS and Education. ERS uses these collaborations to leverage resources and reduce duplication of efforts among Federal and with their university partners. ERS is a key participant in the Interagency Commodity Estimates Committees (ICEC), which includes all USDA agencies responsible for analyzing U.S. and global commodity markets. The annual ICEC Review Forum sets priorities for inter-agency activities related to world supply and demand commodity estimates.

Evidence: ERS collaborates with other USDA agencies, such as FNS (WIC and food security), CNPP (food market surveillance), AMS (mandatory livestock price reporting), and FAS (WTO negotiations) to refine research questions, establish priorities, and review and clear final products prior to release. Examples of our collaboration for survey development include the Agricultural Resources Management Survey (all three phases, www.ers.usda.gov/data/arms/GlobalAbout.htm; Census of Agriculture, www.nass.usda.gov/census/; U.S. Food Security Survey, ers.usda.gov/briefing/FoodSecurity/surveytools/; the Food Market Surveillance Survey; and the Flexible Consumer Behavior Survey, www.cdc.gov/nchs/nhanes.htm.

Does the program use strong financial management practices?

Explanation: ERS's strong financial management practices have resulted in a clean USDA-OIG audit opinion of its financial statements. Expenditures are processed and monitored in the National Finance Center's (NFC) Financial Foundation Information System (FFIS).

Evidence: ERS financial audit reports. FFIS is used to manage financial data to meet budget and funds control requirements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: No material weaknesses in financial management or other areas of the Presidential Management Agenda have been identified for ERS. The USDA FY 2004 Performance and Accountability Report showed that ERS has no material deficiencies, in accordance with the Federal Managers? Financial Integrity Act. ERS conducts organizational climate surveys to assess organizational and management proficiencies and deficiencies. ERS assessment committees provide recommendations to senior managers on ways to identify and improve deficiencies and also to provide continued support for areas of excellence. ERS requires regular management training for Branch Chiefs and supervisory training for project leaders and contract managers. Senior management has a variety of mechanisms to identify, resolve, and communicate management issues facing the agency, including weekly SES and division meetings, and quarterly all Management Meetings, the intranet, a daily email bulletin board and an anonymous suggestion box.

Evidence: ERS budget and execution is processed through the Foundation Financial Information System, USDA's financial system. Integrity of the data is evaluated when the Department's Financial Statements are audited. All of the USDA received a clean opinion on both the FY 2003 and FY 2004 audit of its Financial Statements. See also, U.S. Department of Agriculture's Consolidated Financial Statements for Fiscal Years 2004 and 2003(Report No: 50401-53-FM, 11/15/2004, www.usda.gov/oig/webdocs/50401-53-FM.pdf. Initiatives to improve agency management and climate include task forces to address Human Capital Management, Employee Position Classification System (EPCS) review, agency award policies and practices, and expanded flexible working hours. The review of agency awards resulted in the establishment of an agency-wide award for excellence in research, communication and program effectiveness, a system for non-monetary awards, and better defined criteria for cash awards.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: FANRP's and PREISM's Competitive Grants and Cooperative Agreements are widely publicized and awarded annually. Proposals may be submitted by any State agricultural experiment station, college, university, other research institutions, Federal, State, or county agencies, private organization, corporation, or individual. Research proposals for these competitive programs target high-priority research that has direct implications for USDA's programs. Priority research areas are identified in supporting ERS documentation and developed, in part, from priority setting conferences. The proposal evaluation process includes peer review panels consisting of experts from academia, government and the private sector. In addition to reviewers' comments, the selection process considers coverage of priority research areas, overlap among proposals and ongoing projects, program needs, potential benefits resulting from research collaborations on particular projects and the availability of funding.

Evidence: Terms and conditions for competitive awards for FANRP and PREISM specify quarterly progress and final reporting requirements. Annual reporting for FANRP is found in http://ers.usda.gov/Briefing/FoodNutritionAssistance/2004FinalReport.pdf. Administrative guidance is in Final FANRP Project Manager's Administrative Guide Book for the ERS Competitive Grants and Cooperative Agreements Program. PREISM guidance is at http://www.ers.usda.gov/Briefing/InvasiveSpecies/invasivespecies%2002_05awards.pdf. The Competitive Grants and Cooperative Agreements Programs are announced publicly each year, with program announcements being mailed out to universities, nonprofit and for-profit organizations, and State and local governments. Announcements are also posted on the Internet through the ERS website, http://www.ers.usda.gov/.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Project managers of grants and cooperative agreements are required to develop monitoring plans, which are to be submitted to FANRP and PREISM offices before projects are initiated. These plans serve as a tool by which ERS and the Principal Investigator mutually recognize and document the project's milestones and the timetable by which performance will be accessed. Project Managers work with the Principal Investigator in developing the monitoring plan. It is the responsibility of the Project Manager to monitor projects' progress and keep the FANRP and PREISM offices appraised of significant issues. All awardees are required to send in quarterly progress reports to the Project Manager describing their progress in achieving stated goals (receipt of a quarterly report is required for payment to awardees). Each year, the Project Manager is instructed to check on the status of the project and update the list of outputs that resulted from the project during the year.

Evidence: The FANRP Project Manager's Administrative Guide Book for the ERS Competitive Grants and Cooperative Agreements Program contains the program monitoring and reporting requirements for ERS Project Managers. Reporting requirements for awardees are outlined in Food Assistance and Nutrition Research Program, Fiscal 2005 Competitive Grants and Cooperative Agreements Program: Description and Application Process, www.ers.usda.gov/Briefing/FoodNutritionAssistance/funding/. PREISM guidance is at www.ers.usda.gov/Briefing/InvasiveSpecies/invasivespecies%2002_05awards.pdf. Annually reporting is required for each project, and Project Managers must submit annual Progress Reports on CRIS Form AD-421. Also, a Termination Report, CRIS Form AD-421, must be submitted to ERS for each completed or terminated project. Such reports must be submitted in conjunction with progress reports on active projects and must include a brief summary of accomplishments for the entire life of the project.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Grantees and cooperators are required to submit quarterly reports to an ERS Project Manager describing their progress in achieving stated goals. With this information, the Project Manager assesses the status of the project and updates the list of outputs that resulted from the project during the year. The summary information is used to produce several annual reports, including the FANRP Final Report and the Q&As for Congress. In addition, the information is used to make sure that the list of outputs in PIMS is current. Annual information is also used to update information in Current Research Information System (CRIS).

Evidence: The Food Assistance and Nutrition Research Program, Final Report, www.ers.usda.gov/Briefing/FoodNutritionAssistance/2004FinalReport.pdf and the PREISM Competitive Grants and Cooperative Agreements, www.ers.usda.gov/Briefing/InvasiveSpecies/FY2003PREISMAwards.htm, provide overviews of research themes, principles, and activities and describes the objectives of research projects. Descriptions of the FANRP projects and awards from previous years are available at www.ers.usda.gov/Briefing/FoodNutritionAssistance/funding/descript2004.html. Annual output is also reported in CRIS, a comprehensive repository of performance data for funded projects (CRIS Form AD-421). Projects in CRIS are identified by research problem area and include proposal summaries, annual performance reports, outcomes and impacts and termination reports. This information is searchable, available to the public, and has the capacity to produce various types of reports according to customer needs.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: ERS uses cooperative agreements to enhance and augment its resources to ensure the ability to meet the agency's mission, acquire external expertise, and address short-term issues in a more expedited manner than hiring can achieve. Cooperators are required to submit budgets linked to research objectives. Quarterly reporting is required before payments are made and project managers withhold payment when cooperators fail to file reports or fall behind in delivering promised outputs. An internal competitive process is used to allocate a large share of funds awarded for cooperative agreements. THe process relies on expert panel to assess quality of proposed research, as well as considerations of such factors as compatibility with program needs, overlap with ongoing projects, benefits from collaboration, and the availability of funding.

Evidence: ERS has delegated authority to enter into cooperative agreements with researchers under 7CFR 3015, 3016, 3017, 3018, 3019. The program provides a regular accounting of funding going to cooperative agreements. In FY2004, ERS awarded $5.8 million for FANRP and PREISM and $2.7 million for other cooperative agreements. ERS has 93 cooperative agreements. Examples of unique expertise provided to ERS include updates by World Resources Institute of the US Math Programming Model parameters; studies by University of Wyoming in the emerging field of experimental economics to test hypotheses about farmers? responses to the interaction of two 2002 Farm Act policies (countercyclical payments and base updating); and updates and revision by U of MN, Food Industry Center and industry contacts to conversion factors (raw agricultural commodities into consumer products) used in USDA estimates of commodity supply and use and per capita food consumption.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: A schedule of independent reviews has been established for ERS, including a capstone review of the entire program at the end of the Portfolio Review cycle. The first review, centered on 'Food Choices, Diet, Safety, and Health,' will be conducted Sep. 21-23, 2005. The capstone review will address, both retrospectively and prospectively, the broader issues of resource allocation, priority setting, and mission effectiveness facing the agency. Independent panels will assess the relevance, quality, and performance of program plans, activities, and accomplishments. These assessments will include an evaluation using a quantitative tool to rate the program effectiveness on a multi-category scale. The reason for the "small extent" is that this year only a fifth of the program's total scope will be evaluated. ERS will also use the American Customer Satisfaction Index (ACSI) to assess the satisfaction of external customers with the relevance, usefulness, and accessibility of ERS research, data, and analysis. Since this is the first year the ACSI Customer Satisfaction Rating is measured, the result constitutes the baseline.

Evidence: Portfolio Review, the first long-term measure, evaluates key ERS program components over a 5-year cycle. Panel recommendations will be used in agency strategic planning and priority setting. The Portfolio Review process is summarized in FY 2006 ERS Explanatory Notes. Previous independent reviews to assess the ERS mission effectiveness performed by the National Academy of Sciences (NAS) provided guidance for food security and consumption data in www.nap.edu/catalog/11227.html and www.nap.edu/catalog/11126.html. General NAS feedback is in books.nap.edu/catalog/6320.html, www.nap.edu/catalog/11252.html and books.nap.edu/catalog/10585.html. The ACSI survey (www.theacsi.org/), used for the second long-term measures, is the standard for measuring customer satisfaction in public and private sector organizations.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Specific indicators measure (a) the timeliness of responses to requests for analysis from policy officials (percentage of requested analysis delivered on time), (b) a survey of ERS website customers (customer satisfaction with the ERS website), (c) timeliness of ERS information and analysis, and (d) policy official satisfaction survey (customer satisfaction with requested analysis).

Evidence: ERS has not met its targets for percent of requested analysis delivered on time but has met its target for customer satisfaction with the ERS website. The other two measures have only baseline data available.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ERS produces information and analysis in printed and electronic form through its website. The products include published reports, briefing rooms, and data products. Briefing rooms are web pages that contain synthesized information on a variety of economic and policy topics for which ERS performs analysis. Data products include summarized survey data, periodic reports on commodity production and prices, and online searchable databases. Published reports include Economic Research Reports and Economic Information Reports. The growth in production of these products has also been substantial over the past 4 years in an attempt to meet the growing demand.

Evidence: During 2001-05, ERS increased the number of products released from 238 to 365. During that time, staff years have declined from 491 to 430 in FY 2005. These numbers indicate ERS product release efficiency has increased from a base of 1.00 in FY 2001 to 1.75 in FY 2005.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ERS is the intramural social science research arm of USDA. The research and analysis is presented in a broad policy context, unlike program administration agencies. While ERS is unique in terms of the scope of policy and research issues addressed, the approach and research methods are not unique. To assess the quality of the ERS research program, twice ERS was examined by National Research Council panels. One panel explicitly examined ERS program of research and quality of output. Another commended ERS for it quality and efforts in addressing key policy issues.

Evidence: ERS research program was reviewed in (1) Sowing Seeds of Change: Informing Public Policy in the Economic Research Service of USDA, http://books.nap.edu/catalog/6320.html, and (2) Frontiers in Agricultural Research: Food, Health, Environment, and Communities, http://books.nap.edu/catalog/10585.html. These reviews commended ERS for the areas of research addressed, its extensive use of peer reviews to improve the quality of this research and its contribution to important policy decisions. In (1) reviewers indicted, ERS demand modeling made an important contribution in support of GATT negotiations, p 27. In (2) the agency's successful contributions to professional conferences and journals test the appropriateness and rigor of the research methods in its analysis with respect to disciplinary standards. p. 122.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The program's scope has been developed and validated by various mechanisms, including stakeholder meetings; indepth, external program review, and rigorous external peer review. The National Academy of Sciences (NAS) has provided rigorous reviews for ERS: a comprehensive review of the entire program and reviews of specific components. The 2003 report issued by NAS commended ERS for using peer review process and documenting research that supports the action agencies. A 1997 NAS review noted that ERS provided important value for policy deliberations related to agriculture, food, natural resources, and rural development. The National Agricultural Research, Extension, Education and Economics Advisory Board (NAREEEAB) provided advice to ERS on top priorities and policies for food and agricultural research and analysis. ERS ensures its data collection efforts meet quality standards and users' critical data needs by seeking review and recommendations from stakeholders.

Evidence: Comprehensive National Academy of Sciences reviews of ERS: Sowing Seeds of Change: Informing Public Policy in the Economic Research Service of USDA, books.nap.edu/catalog/6320.html and Frontiers in Agricultural Research: Food, Health, Environment, and Communities, books.nap.edu/catalog/10585.html. The NAS has provided guidance for food security and consumption data in Measuring Food Insecurity and Hunger, www.nap.edu/catalog/11227.html, and Enhancing the Data Infrastructure in Support of Food and Nutrition Programs, Research, and Decision Making, www.nap.edu/catalog/11126.html. Quality and relevance is ensured through ERS Information Quality Guidelines, www.ers.usda.gov/AboutERS/QualityGuidelines.htm and NAREEEAB, www.nareeeab.com/.