Detailed Information on the
NASA Aeronautics Technology Assessment

Under the leadership of PA&E, benchmark R&T practices in performance & budget integration and performance measurement (i.e. efficiencies & evaluations) with other government agencies.

Conduct an annual review by experts from outside the program, FFRDC, &/or from other government agencies to assess the restructured Aeronautics Research Mission Directorate program's quality of research and alignment with national priorities. The review will determine how well the program is aligned with the stated objectives of the NASA Strategic Plan and the National Aeronautics Research & Development Policy, identify any gaps, and assess the quality of the research. ARMD will setup the charter & validation of the annual review to meet the standards of Independent Evaluation. (to be completed in January 2008)

Complete the independent assessment of NASA's fundamental aeronautics research, contracted to the National Research Council of the National Academies - study, titled "Evaluation of NASA's Fundamental Aeronautics Research Program." (To be released publicly June 2008; preliminary report due in March 2008.)

Ensure that NASA's aeronautics research is in alignment with the research needs of the Next Generation Air Transportation System (NextGen) as defined in the NextGen Research and Development Plan and Integrated Work Plan.

Measure: Progress, as evaluated by external expert review, in developing future concepts, capabilities, and technologies that will enable major increases in airspace mobility (which includes capacity, flexibility and affordability), while maintaining safety and protecting the environment, to meet the needs of the Next Generation Air Transportation System (NGATS).

Explanation:This outcome measure is consistent with the National Aeronautics R&D Policy; it flows down to the long-term output measures for each research program; and it emphasizes the fact that each program works together to achieve the overarching NGATS goals and objectives, consistent with the statement in the R&D Policy that "treating the entire system as a whole is complex but necessary". Subject matter experts from other Government agencies, FFRDCs, as well as experts internal to NASA who are not participants in the ARMD projects, will be invited to participate in an Agency review of the aeronautics portfolio, and the findings and recommendations will be documented.

Measure: ATP ground test facilities are available and operational in order to support the research, development, test and engineering milestones of NASA and DOD programs from both schedule and cost perspectives. This metric is known as "on-time availability."

Explanation:Unscheduled facility down-time, often caused by unplanned maintenance and repairs, results in lost revenue and not meeting contractual commitments to various facility customers (NASA, other government agencies, industry). The cost to operate/run the ATP facilities is roughly $500K each day. These costs can be reimbursed during active testing. Therefore, it is in the best interest of ARMD and the agency to do strategic facility maintenance in order to reduce the Deferred Maintenance liability for ATP facilities and maximize facility on-time availability. The cumulative effect of doing annual maintenance is what translates into an efficiency for the ATP facilities. The target is % on-time availability based on a fixed annual maintenance investment.

Measure: Develop an integrated tool set to accurately predict noise, emissions, and performance of both conventional and unconventional subsonic aircraft.

Explanation:In 2006, complete design of geared turbofan components that have the potential to reduce noise and emissions relative to current engines; in 2007, evaluate and rank candidate materials for strength, durability, maximum use temperature and functionality potential; in 2008, develop and test component technology concepts used in conventional aircraft configurations that establish the feasibility of achieving Stage 3-42 EPNdb (cumulative) noise reduction; in 2009, develop a database for alternative hydrocarbons using accepted testing standards, then characterize the fuels (freezing point, break point, etc) in comparison to current Jet-A; in 2010, complete new suite of integrated multidisciplinary analysis tools to predict noise, NOx, takeoff/landing performance, cruise performance, and Take-Off Gross Weight (TOGW) for conventional ("tube and wing") aircraft and unconventional aircraft (e.g. hybrid wing-body); in 2011 achieve validated accuracy, for conventional and unconventional aircraft, respectively, of (2.5%, 5%) for NOx, takeoff/landing performance, cruise performance, and TOGW, and of (1.25 dB, 2.5dB) for noise.

Measure: Demonstrate a variable-speed rotor concept that incorporates the ability to change rotor rotational speed without performance or handling qualities penalties

Explanation:In 2006, conduct a test in the 14x22 Subsonic Wind Tunnel to demonstrate the ability to measure unsteady surface pressures for a rotor in forward flight using pressure sensitive paint; in 2007, complete design for large field-of-view particle image velocimetry system in the 14x22 wind tunnel; in 2008, assess predictive capability of tools to model one engine stall control concept (minimum success: within 10% of measured stall point) using component test data in order to extend engine operability range; in 2009, develop and validate transmission tools and technologies to support variable speed drive systems using data from several transmission test cells at GRC; in 2010, demonstrate a flight control optimization tool through simulation that enables the control of a variable speed engine and transmission with no negative handling qualities effects; in 2011, validate the ability to predict the effects of active flow control systems for level flight by comparison with wind tunnel data (accuracy goals are rotor performance:15%, first-principles vibration and hub loads: 100%, and noise: 5 dB of peak levels); and by 2012, integrate and demonstrate these combined technologies to meet the goal to change rotor rotational speed by 50% without performance or handling qualities penalties through large-scale wind tunnel demonstration of integrated proof-of-concept test articles

Measure: Develop validated multidisciplinary framework for the analysis and design of supersonic aircraft that are cruise efficient and have both low noise, and low emissions

Explanation:In 2006, complete initial study of the impact of atmospheric turbulence on very low noise sonic boom waveforms; in 2007, complete analysis for off-design conditions at 100% fan speed and near stall with the help of Large Eddy Simulation; in 2008, demonstrate a composite supersonic engine fan blade containment system that is 20 percent lighter than the High Speed Research Program metallic containment system and validate through laboratory tests; in 2009, demonstrate an adjoint-based design method for configuration shaping; also establish the capability to design and analyze supersonic vehicles that achieve efficiency improvements within 10% of the defined targets including engine plume effects and verify the results using wind tunnel and flight experiments; in 2010, develop computational models to predict integrated inlet and fan performance and operability and compare models to experimental data; use a computational design optimization study verified by wind tunnel experiments to demonstrate a two-week multidisciplinary design, analysis and optimization cycle time for cruise efficiency under imposed low-speed constraints with a two-times reduction in design uncertainty in 2011 (based on the 2007 baseline); and by 2013, develop validated multidisciplinary framework for the analysis and design of supersonic aircraft that are cruise efficient and have both low noise, and low emissions. The framework will predict cruise efficiency of candidate configurations to within 1.5% of actual values. In addition, predictions of the following quantities will achieve the specified accuracy: take-off and landing noise, 3.5 dB, outdoor sonic boom intensity, 4 dB, indoor sonic boom intensity, 5 dB, and NOx and particulate emissions, 10%.

Measure: Develop a computer-based full flight simulation capability using physics-based high-fidelity multidisciplinary design tools with sufficient accuracy to enable highly reliable reusable launch systems for space access

Explanation:In 2006, complete Mach 5.0 wind tunnel test of Pratt and Whitney Rocketdyne's Ground Demonstrator Engine No. 2; in 2007, deliver two fundamental aerophysics flight experimental payloads, Hy-BoLT (Hypersonic Boundary Layer Transition) and SOAREX (Sub-Orbital Aerodynamic Reentry Experiments), to Wallops Flight Facility for integration with the ATK ALV-X1 suborbital vehicle and launch; in 2008, complete a thermal-structural analysis of a control surface component under simulated re-entry conditions and compare to multi-cycle thermal and structural test results from the DFRC hot structures facility; in 2009, complete the CFD pretest predictions of performance and operability of a high Mach fan for a TBCC propulsion system and compare to fan test data from the GRC W8 facility; in 2010, complete CFD predictions of ramjet-to-scramjet mode-transition and compare to wind tunnel and/or X-51 flight test data; in 2011, evaluate the accuracy of phenomenological turbulence and sub-grid scale models by comparing CFD predictions with test data from wind tunnel vehicle models at the ARC ballistic range and at LaRC wind tunnel facilities; also in 2011, validate combustor wall thermal-structural performance and critical failure modes to within 10% of predictions; and by 2012, develop a computer-based full flight simulation capability using physics-based high-fidelity multidisciplinary design tools with sufficient accuracy to enable highly reliable reusable launch systems for space access. Sufficient accuracy is defined by ensuring that predicted payload mass fractions are correct to within 10% of the actual values

Measure: Deliver and demonstrate sensors, software, validated tools and guidelines that will enable a designer or operator to implement on board IVHM systems

Explanation:In 2006, in partnership with the FAA, the Commercial Aviation Safety Team (CAST), and the aviation community, provide an initial demonstration of a voluntary aviation safety information sharing process, and use this shared information and process to complete one or more safety relevant studies identified by the CAST; in 2008, using a landing gear testbed as a representative current generation electro-mechanical system, develop and validate sensor integration technologies with improved diagnostic robustness and the ability to distinguish sensor failures from true component failures (no more than 20% false negative and 20% false positive rates); in 2009, demonstrate 10% improvement in estimation accuracy of integrated gas path sensing and diagnostics for aircraft engine health; in 2010, using 2008 as a baseline demonstrate improved IVHM for landing gear components and sub-systems in the aircraft landing dynamics testbed with diagnostic coverage of seeded faults (no more than 10% false negative and 10% false positive rates); in 2011, demonstrate self-healing material system concepts to mitigate damage in structural elements that are subjected to representative loading; and by 2016, deliver and demonstrate sensors, software, validated tools and guidelines that will enable a designer or operator to implement on board IVHM systems that can automatically self-monitor, detect, identify, and correct airframe or system anomalies in enough time to achieve corrective action without having to abort a mission while in an in-flight status

Measure: Deliver validated tools and methods that will enable a designer or operator the ability to implement aircraft aging mitigation strategies

Explanation:In 2006, develop multi-scale analysis methodologies to bridge length scales using a combined Finite Element and Molecular Dynamics approach to model damage processes which demonstrate numerical performance sufficient for experimental validation; in 2007, identify critical metallic material degradation processes and establish consensus within NRA-partner community on necessary analytic methods and baseline criteria for future evaluation of fatigue and fracture models; in 2008, develop analysis methods for predicting crack growth versus load for selected metallic structures with improved ultimate failure load prediction by 20% over state-of-the-art methods assessed in FY07; in 2009, conduct a spin test to verify enhanced disk rim attachment strength at component level and show 10% life improvement over criteria established in 2007; in 2010, develop an atomistic model capable of predicting within 25% of established baseline, the degradation caused by environmental effects on interfaces in selected polymer matrix composite materials; in 2011, develop a mitigation technique that demonstrates 50% performance improvement over the 2007 state-of-the-art baseline, and demonstrate that modeling tools can accurately predict the performance improvement; and by 2016, deliver validated tools and methods that will enable a designer or operator the ability to implement aircraft aging mitigation strategies with at least 50% performance improvements over current techniques, to include new physics-based models of sufficient accuracy to support certification criteria for predicting aging related hazards and durability performance of advanced materials and new structural designs intended for use in future civilian and military aircraft

Measure: Deliver guidelines, validated tools, and flight deck technologies that will enable a vehicle designer or operator to implement advanced automation and system performance improvements

Explanation:In 2006, produce a performance assessment of the turbulence detection and hazard assessment capabilities of a modern airborne radar; 2007, baseline IIFD state-of-the-art capabilities and identify future flight deck research needs for NGATS as envisioned for 2025 and by JPDO; in 2008, complete the feasibility study for active operator assistance in approach and landing tasks that will identify technologies with predicted potential to support selected NGATS applications and enable 90% reduction in automation interface time, elimination of mode confusion, and no degradation in operator awareness of critical decision points as compared to baseline conditions established in 2007; in 2009, assess and deliver findings on initial multi-modal presentation formats and interaction methods for uncertainty display concepts and virtual visual environments with statistically significant reductions in communication errors, mental workload, and flight technical error, as well as increases in usability and situation awareness compared with baseline capability; in 2010, deliver and validate through analysis flight deck guidelines, information, and display requirements that meet NGATS operational needs as established in 2007 baseline assessment, and without a measurable increase to safety risk; in 2011, complete the validation of selected part-task simulation results in flight environment, which supports improved crew awareness and decision making with 50% reduction in response time to pop-up hazards compared with baseline condition established in 2007; and by 2016, deliver guidelines, validated tools, and flight deck technologies that will enable a vehicle designer or operator to implement advanced automation and system performance improvements being developed to meet NGATS 2025 operational concepts, with no degradation of safety risk when compared to 2007 state-of-the-art flight deck related operational systems

Measure: Deliver validated, multidisciplinary integrated adaptive control design tools and technologies intended to prevent loss-of-control in-flight and enable safe vehicle recovery

Explanation:In 2006, conduct flight experiments with a neural network based flight control system versus a conventional control system (Neural Network turned off) to demonstrate 25% improved stability in the pitch axis, as measured by frequency response along with no degradation in pilot handling qualities of the aircraft in the presence of a simulated failure; in 2008, assess adaptive control system with improved stability (60% within a gain margin of 5dB and phase margin greater than 35degrees, 30% within a gain margin of 3dB to 5dB and phase margin in the range of 25 to 35degrees) for aircraft in damage/failure conditions by conducting piloted evaluations in a high-performance research aircraft and compare results to experimental results from 2006; in 2009, design and evaluate preliminary concepts in on-line integrity monitoring (99% failure detection with less than 1% false positives) for adaptive control systems through simulation tests; in 2010, develop a tool suite that provides an order of magnitude reduction in analysis time over current Monte-Carlo simulation methods that would be used to locate failure points in the flight envelope for a chosen adaptive control system and a set of adverse events; in 2011, assess improved control strategies to recover an unimpaired aircraft from an upset condition and achieve stable recovery from elevated angle-of-attack and stall conditions in minimal time, with minimal loss of altitude (minimal time/altitude as determined by optimum execution of established manual control recovery procedures for test aircraft under selected flight test conditions), and within 150% of nominal load; and by 2016, deliver validated, multidisciplinary integrated adaptive control design tools and technologies intended to prevent loss-of-control in-flight and enable safe vehicle recovery from selected upset and impaired situations by achieving stabilized conditions from unusual angle-of-attack and stall conditions in minimal time, with minimal loss of altitude, and within 150% of nominal load

Measure: Develop and demonstrate future airspace concepts, capabilities, and technologies that will enable air traffic service providers to provide major increases in air traffic management effectiveness, flexibility, and efficiency, while maintaining safety to meet capacity and mobility targets (at least two times aggregate system capacity) consistent with the Next Generation Air Transportation System (NGATS) vision

Explanation:In 2006, complete development of a candidate system-wide concept for the future National Airspace System; in 2007, develop trajectory analysis technology for automated separation assurance where 95% of traffic conflicts are detected and resolved prior to the 3-5 minutes to loss of separation point with overall resolution delays and near-miss separation characteristics that are comparable or better than that of today's operations while operating under a significant increase in traffic density (e.g. 2-3x); in 2008, conduct service-provider-based automated separation assurance simulations; in 2009, complete trajectory analysis for service provider-based automated separation assurance with time-based metering with 2-3 times increase in capacity without reduction of baseline metering accuracy or separation violations; in 2010, conduct laboratory analysis of technology for tactical weather and traffic complexity avoidance with time-based metering constraints; in 2010, conduct simulations and analysis of time-based metering with service-provider-based automated separation assurance; in 2011, complete simulation analysis of service-provider-based automated separation assurance with complex traffic, metering, hazardous weather, and failure recovery with 2-3 times increase in capacity without reduction of baseline metering accuracy or separation violations; and by 2016, develop and demonstrate future airspace concepts, capabilities, and technologies that will enable air traffic service providers to provide major increases in air traffic management effectiveness, flexibility, and efficiency, while maintaining safety to meet capacity and mobility targets (at least two times aggregate system capacity) consistent with the Next Generation Air Transportation System (NGATS) vision. The project will provide an increase in data necessary to establish the safety, delay reduction, and user preference metrics of technology and operations as function of automation level, traffic complexity, and aircraft performance. Comparison of objective metrics will be done with those of today's operations. In addition, the project will provide an increase in essential understanding of human/machine allocation of functionality and service provider/aircraft operator allocation of operations and responsibility for heavy arrivals flows using time-based metering as a function of traffic density and complexity, in the presence of uncertainty, and with limited failure modes in arrival metering environment. This requires objective experimental data to quantify workload, safety, and trajectory efficiency during arrival flow management. Subject matter expert feedback (FAA, airlines, controllers, and pilots) will be collected on operating concepts. Strengths and weakness of concepts and technology as a function of traffic complexity will be documented.

Measure: Develop and demonstrate future airportal concepts, capabilities, and technologies that will enable air traffic service providers to provide major increases in air traffic management effectiveness, flexibility, and efficiency, while maintaining safety to meet capacity and mobility targets (at least two times aggregate system capacity) consistent with the Next Generation Air Transportation System (NGATS) vision

Explanation:In 2009, test taxi route optimization algorithms in fast-time simulations. Evaluations will include benefits in both nominal and off-nominal conditions under increased Airportal traffic density and consider environmental constraints and aircraft operator schedule preferences. Evaluation of taxi delay reduction and Airportal throughput increase will be based on comparisons with taxi route decisions made by subject matter experts when presented with the same scenarios. Results will determine the degree to which taxi optimization can increase Airportal throughput and thereby guide future project research investments. Traffic scenarios and off-nominal conditions will be defined by ongoing concept of operation definition and integration with Airspace Project baseline scenario development. In 2011, validate initial super-density concepts, to include a set of culminating experiments to understand and validate key Airportal contributions to superdensity operations. Validation requirements will be established as part of early project system studies to determine super-density airportal concept of operations definitions. Anticipated metrics may include increase in runway, airport, and metroplex throughput per hour, as well as average taxi delays during peak hours, fuel savings, noise/emissions impacts, runway crossing times, detected runway/taxiway conflicts, and runway productivity. Experimental conditions will represent mixed equipage and will include uncertainty in intent and weather information. Selected non-normal situations, including system failures and emergency events, will be studied. By 2016, develop and demonstrate future airportal concepts, capabilities, and technologies that will enable air traffic service providers to provide major increases in air traffic management effectiveness, flexibility, and efficiency, while maintaining safety to meet capacity and mobility targets (at least two times aggregate system capacity) consistent with the Next Generation Air Transportation System (NGATS) vision. The project will provide an increase in essential data to improve runway throughput per hour, average taxi delays during peak hours, fuel savings, noise/emissions impacts, runway crossing time, and detected runway/taxiway conflicts. Key aspects include optimization of arrival, departure, and taxi scheduling, balanced allocation of airportal resources to maximize airportal productivity in response to arrival, departure and surface traffic demands. Additional metrics will involve robustness of operational concept to non-normal situations, including system failures and emergency events. Subject matter expert feedback will be collected on operating concepts. Strengths and weaknesses of concepts and technologies as a function of traffic complexity will be documented.

Is the program purpose clear?

Explanation: The purpose of the Aeronautics Technology Theme is to advance U.S. technological leadership in aeronautics in partnership with industry, academia, and other government agencies that conduct aeronautics-related research. NASA's Aeronautics Research Mission Directorate (ARMD) is responsible for the agency's Aeronautics Technology Theme. Specifically, ARMD conducts cutting-edge research that includes foundational research across a breadth of core aeronautics competencies that supports aeronautics and space exploration activities; research in key areas related to the development of advanced aircraft technologies and systems, including those related to aircraft safety, environmental compatibility, and fuel efficiency; and research that supports the Next Generation Air Transportation System (NGATS) in partnership with the Joint Planning and Development Office (JPDO). In addition, ARMD is pursuing a coordinated approach to managing the Nation's research, development, test, and evaluation infrastructure with other agencies, particularly the Department of Defense. The program's purpose relates directly to the NASA Strategic Plan, the President's Vision for Space Exploration, and the President's National Aeronautics Research and Development (R&D) Policy. ARMD supports the Agency's goal of developing a balanced overall program of science, exploration, and aeronautics, consistent with the redirection of the human spaceflight program to focus on exploration. The ARMD research plans also directly support the National Aeronautics R&D Policy and accompanying Executive Order 13419.

Evidence: NASA's Strategic Plan can be found at http://www.nasa.gov/about/budget/index.html. The President's Vision for Space Exploration can be found athttp://www.whitehouse.gov/space/renewed_spirit.html. The President's National Aeronautics R&D Policy and its accompanying Executive Order can be found at http://www.ostp.gov/nstc/aeroplans/nard_policy.htm. A report to Congress detailing how ARMD's programs and projects are aligned with the new National Aeronautics R&D Policy can be found at http://www.aeronautics.nasa.gov/nasa_nardp.htm.

Does the program address a specific and existing problem, interest, or need?

Explanation: ARMD conducts research that addresses several important national challenges that are aligned with the principles established in the National Aeronautics R&D Policy. The National Aeronautics R&D Policy, and accompanying Executive Order 13419 signed by the President on December 20, 2006, cites seven principles that are vital in order for the U.S. to maintain its technological leadership in aeronautics. These principles will guide the research investments of all agencies that conduct aeronautics-related research through 2020. NASA's aeronautics research contributes directly and substantially to six of the seven principles. Specifically, the Fundamental Aeronautics Program directly addresses the first principle of "mobility through the air" by conducting research that can enable the development of advanced aircraft systems that fly with higher performance, lower fuel consumption, and minimum environmental impact at a range of speeds and from a wide variety of airports. The Airspace Systems Program directly addresses this principle by conducting air traffic management research that will develop concepts, capabilities, and technologies required to meet the Nation's anticipated growth in airspace operations, both in the air and on the ground. Both the Fundamental Aeronautics Program and the Aviation Safety Program address the second principle of "national security and homeland defense" by conducting "dual-use" research in a number of areas including advanced aircraft design, integrated propulsion concepts, multi-functional materials development, and advanced aviation safety technologies. The core mission of the Aviation Safety Program directly addresses the third principle that states that aviation safety is paramount. All of ARMD's programs directly address the fifth principle of developing a "world-class aeronautics workforce" by focusing on cutting-edge research, and by fostering intellectual partnerships with industry and academia by means of cooperative Space Act Agreements (SAAs) and fully and openly competed research awards that emphasize true collaborations among partners. In addition, ARMD currently funds graduate student scholarships and intends to expand its scholarship efforts to include scholarships for undergraduates. Both the undergraduate and graduate scholarships will include summer internship opportunities at NASA research centers. Finally, the Fundamental Aeronautics Program simultaneously addresses the sixth and seventh principles by conducting research to improve aircraft performance, increase fuel efficiency, lower emissions and reduce noise. In addition, the Airspace Systems Program also addresses these two principles by conducting research to improve efficiency and reduce environmental impact through better utilization of the airspace.

Evidence: The President's National Aeronautics R&D Policy and its accompanying Executive Order can be found at http://www.ostp.gov/nstc/aeroplans/nard_policy.htm. A report to Congress detailing how ARMD's programs and projects are aligned with the new National Aeronautics R&D Policy can be found at http://www.aeronautics.nasa.gov/nasa_nardp.htm.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ARMD is the only entity dedicated to long-term, cutting-edge applied research that expands the boundaries of aeronautical knowledge for the benefit of the broad aeronautics community. More limited or focused aeronautics research is conducted by other Federal entities: FAA, DOD, DOC, NSF, and DHS. NASA has put in place several mechanisms to ensure that its research is not duplicative of research being funded by those agencies. For example, NASA is an active participant in several interagency organizations at both the senior executive level and the research level. At the senior level, NASA is an active participant in the JPDO, the NASA/Air Force Executive Research Committee (ERC), the National Partnership for Aeronautical Testing (NPAT) Council, the Commercial Aviation Safety Team (CAST), the International Helicopter Safety Team (IHST) and the Hypersonics Joint Technology Office. At the research working level, NASA participates in the Versatile Affordable Advanced Turbine Engine (VAATE) Technology Program, the Airspace and Safety Integrated Planning Teams (IPTs) of the JPDO, the Vertical Take-off and Landing (VTOL) Science and Technology Partnership Council, the Joint Council on Aging Aircraft (JCAA), the Fixed-Wing Executive Council, and the Aircraft Icing Research Alliance (AIRA), among others. Finally, as a member of the NSTC Aeronautics Science and Technology Subcommittee, NASA will be working alongside representatives from DOD, DOE, DHS, DOS, DOT, FAA, JPDO, NSF, USITC, CEA, DPC, NSC, OMB, OSTP, OVP, and USTR to develop a National Aeronautics R&D Plan and a National Aeronautics RDT&E Infrastructure Plan as part of the implementation of the National Aeronautics R&D Policy. As part of the formulation of the plans, the subcommittee will be examining whether there are areas of duplication among any of the agencies and will make recommendations as appropriate. Both the R&D plan and the RDT&E Infrastructure plan will be completed and delivered to the Director of OSTP by the end of December 2007. Regarding the private sector, NASA has made it explicitly clear in several public documents, including testimony given by both the NASA Administrator and Associate Administrator of Aeronautics, that it will not conduct research that is more appropriately conducted by the private sector. NASA will engage with the private sector by means of technical working groups, technical interchange meetings, and frequent visits to companies across the country, to gather input and perspectives regarding pre-competitive research that no one company will engage in because it requires long-term investment, is high-risk, and is not appropriable to a single company. ARMD will enter into Space Act Agreements with industry only when such collaborations will yield significant benefit to NASA and its constituencies (the aerospace community, aerospace industry, academia, and ultimately, the taxpayer), and only when the results of the collaboration can be appropriately disseminated and validated through a peer-review process. This approach will ensure that NASA does not conduct research that benefits only a small number of companies, and that NASA does not conduct research that is beyond the pre-competitive stage. This approach is consistent with the guidance provided in the National Aeronautics R&D Policy. NASA is unaware of any significant aeronautics research efforts being funded by state and local governments.

Evidence: The President's National Aeronautics R&D Policy can be found at http://www.ostp.gov/nstc/aeroplans/nard_policy.htm. Details about the various interagency working groups are provided in Appendix A of a Report to Congress found at http://www.aeronautics.nasa.gov/nasa_nardp.htm. Details about the JPDO can be found at http://www.jpdo.aero, and details about CAST can be found at http://www.aia-aerospace.org/departments/civil/CAST/participants.html. Copies of testimony by NASA witnesses can be found at http://legislative.nasa.gov/hearings/index.html.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: ARMD put together a four-step process during FY2006 that was designed to ensure that the Aeronautics Technology Theme was free of major flaws that would limit its effectiveness or efficiency. During the fall of 2005, ARMD completed the first step, during which researchers at the four research Centers, with input from other Government agency partners, developed preliminary ten-year roadmaps that included technical milestones for each project in each program. In January 2006, ARMD began its second step by releasing a Request for Information (RFI), soliciting interest from industry for non-reimbursable cooperative partnerships in pre-competitive research that would allow NASA to leverage industry's systems-level expertise while facilitating the rapid transfer of knowledge and technology from NASA to industry. ARMD received over 230 responses from more than 100 different organizations, many of which have resulted in working collaborations. During the third step, NASA researchers used the preliminary roadmaps as a starting point and incorporated feedback from respondents to the RFI, as well as from colleagues in other Government agencies, to develop refined technical proposals for each project. Panels of Government subject-matter experts from NASA, DOD, FAA, NOAA, and the JPDO then reviewed and evaluated the proposals based on their technical, management, resource, and partnership plans. The proposal review process entailed a review by each panel of each written proposal, an oral presentation by the NASA researchers to the panel, and a second review of the revised written proposal (revised based on feedback from the first written and oral reviews). This proposal review process ensured that NASA had technically credible and relevant research objectives and a sound approach for pursuing these objectives. It also allowed NASA to identify research areas where it needed to supplement in-house capabilities with external expertise. During the fourth and final step, ARMD released an NRA to solicit proposals from the external community in foundational research areas where NASA needs to enhance its core capabilities. ARMD will continue to rely upon the NRA process to ensure full and open competition for all out-of-house research. During the restructuring of the aeronautics research portfolio, all 10 projects were formulated by inter-center proposal teams to ensure full coordination and integration of research and competencies at the four NASA research centers. In addition to the rigorous reviews conducted at the ARMD level, each of the ten proposals was independently reviewed by an inter-center management team for technical integrity. (Members of these inter-center review teams are part of what the Agency calls its "Technical Authority" chain which is independent of the program chain.) The resulting technical plans improved efficiency and effectiveness by maximizing the utilization of critical skills and facilities at all four centers while eliminating center-to-center competition that leads to duplication of effort and suboptimal research, as well as any center-specific research that does not directly contribute to the program goals and objectives. Finally, the Aeronautics Technology Theme is supported by an adequate capital asset management infrastructure that is consistent with the principles and techniques of effective capital programming. The Aeronautics Test Program (ATP) is responsible for the strategic availability of a minimum, critical suite of aeronautical test facilities that are necessary to meet the long-term needs and requirements of the agency and the nation. Strategic utilization, operations, maintenance, and investment decisions are made for facilities at the Ames Research Center, Glenn Research Center, Langley Research Center, and Dryden Flight Research Center. There is no strong evidence that another design would be more effective.

Evidence: Technical plans and milestones for each ARMD project, the critical facilities covered by the Aeronautics Test Program can be found at http://www.aeronautics.nasa.gov/programs.htm, and details on the NRA process can be found at the ARMD website at http://www.aeronautics.nasa.gov/nra.htm. Further information on ARMD's planning process can be found at http://www.aeronautics.nasa.gov/present.htm.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The planning process described in the response to Question 1.4 ensures that ARMD is designed and structured to meet its purpose and that its results reach the intended beneficiaries. Each of the project's goals, and the accompanying research plans, milestones, and metrics that support those goals, were established after extensive engagement with other Government agencies as well as with the private sector. Specifically, before the plans were put in place, the ARMD Associate Administrator met with the FAA Administrator and her staff to discuss the research needs of the JPDO and the NGATS; the FAA's Associate Administrator for Safety to discuss the long-term research needs in aviation safety; the Director of the JPDO to further discuss NGATS research needs; and the Chief Scientist of the Air Force, the Army Deputy Assistant Secretary for Research and Technology, the Deputy for the Army Research, Development and Engineering Command, the Director of DARPA, and the Director of DARPA's Tactical Technology Office to discuss collaborative opportunities in a range of aeronautical areas. Each of these leaders provided several people from their organizations to help in the formulation of ARMD's project plans as well as in the proposal review process. ARMD purposefully limited the proposal reviewers to Government personnel to ensure that no conflicts of interest arose. To engage the broader external community, an RFI process was used early in the formulation of the plans. This process ensured that everyone had an equal opportunity to provide inputs to ARMD while making sure that no one company or individual was given preferential access. ARMD received over 230 responses from more than 100 different organizations, and these inputs were used during the proposal formulation process. ARMD has received support from the leadership of the FAA, JPDO, and DOD that the program is conducting research that is relevant to their needs. To ensure that ARMD continues to conduct research that targets its beneficiaries, ARMD will involve the FAA, JPDO, and DOD in its 6-month and 12-month program reviews, and will solicit their written evaluations during its 12-month reviews. In addition, ARMD projects will hold regular technical working groups that are open to the broad community as well as technical interchange meetings during which NRA research is presented to the broad community and feedback is solicited. ARMD-funded research (both in-house and out-of-house) will be published by means of archival journals and conference proceedings, ensuring peer review by the broad community. In addition, as described in the response to Question 1.3, NASA participates in a wide variety of interagency organizations, such as the CAST, the REDAC (the ARMD Associate Administrator is a member), the JPDO, and the NASA/AF ERC, to ensure that the program's research remains relevant and that its beneficiaries remain fully aware of its research. ARMD's involvement with the JPDO is particularly extensive. ARMD researchers have participated and continue to participate in all elements of the JPDO, from the IPTs to the architecture and engineering and analysis divisions, to the JPDO Board. The Administrator of NASA is a member of the JPDO SPC. Finally, NASA is ensuring that it does not conduct research more appropriate for the private sector, as described in the response to Question 1.3; ARMD is therefore not providing inappropriate subsidies.

Evidence: The technical plans and milestones for each ARMD project can be found at the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. The NGATS ATM:Airspace Project in the Airspace Systems Program held a kick-off technical interchange meeting in March of this year that was attended by about 160 people from industry, academia, other government agencies, FFRDCs, and NASA researchers. The Fundamental Aeronautics Program and Aviation Safety Program will have their interchange meetings in the fall of 2007. NASA has been heavily involved and integrated into the JPDO NGATS planning process, which has ensured that our programs are closely aligned with NGATS needs. NASA personnel hold several key JPDO positions, including the JPDO deputy director, along with the directors of the Systems Engineering and Analysis Division (SEAD), Enterprise Architecture and Engineering Division (EAED), and the Agile Air Traffic System Integrated Product Team (AATSIPT). The Airspace Systems Program (ASP) NGATS Technical Integration Manager is on the AATS IPT executive council, and the ASP ATM Airspace Project Principal Investigator is on the AATS IPT steering group. The Fundamental Aeronautics (FA) Program Deputy and the Subsonic Fixed Wing project Principal Investigator are members of the EIPT Steering Group. In addition, the three ARMD programs of Airspace Systems, Aviation Safety, and Fundamental Aeronautics have researchers representing them in the general membership of the Agile, Weather, Safety, Airports, and Environment IPTs along with SEAD and EAED. As the JPDO transitions to its new organizational structure this year, NASA will continue to contribute key personnel.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Each project within the program has a long term output measure that assesses production of systems-level aeronautics technologies. The program also has one long-term outcome measure to evaluate its contribution to the Next Generation Air Transportation System (NGATS). Enabling NGATS is a key goal of the program that cuts across many projects; many of the projects' long-term output goals support NGATS. Output measures are used because of the broad set of customers for aeronautics technologies and the nature of foundational research.

Evidence: The project measures can be found in the Program Performance Measures section of the 2007 PART. Further details on the project outputs can be found in the project plans at http:///www.aeronautics.nasa.gov/programs.htm.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: ARMD's planning process produced technical plans that include long-term performance measures with verifiable targets and specific timeframes. The projects have done assessments of the current state-of-the-art technology as well as previous related research in order to establish baselines. In addition, based on the feedback received during each step of the restructuring process, ARMD is confident that its long-term measures are ambitious. As previously noted, ARMD put together a four-step process to restructure its programs in FY06. The process began with NASA researchers, using input from other Government agency partners and stakeholders, developing preliminary ten-year roadmaps that included technical milestones for each project. Next, ARMD released an RFI soliciting interest from industry for non-reimbursable cooperative partnerships, and received numerous responses. NASA researchers then took the original roadmaps, incorporated the feedback from respondents to the RFI, as well as from colleagues in other Government agencies, and developed refined technical proposal for each project. These proposals were then reviewed by panels of Government subject-matter experts on their technical, management, resource, and partnership plans.

Evidence: The detailed research plans for each project, which contain milestone and metric descriptions, can be found on the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. The project measures can be found in the Program Performance Measures section of the 2007 PART. Specifically, one example of the ambitious long-term measures for aeronautics research is that of the Subsonics Fixed Wing (SFW) Project in the Fundamental Aeronautics Program. The long-term measure is indeed ambitious considering the SFW project has set goals for noise, emissions and performance that are two generations beyond the current state-of-technology, and one generation beyond what industry is currently striving for. Technologies to achieve each of these goals on their own are at their infancy today. The development of an integrated tool set will allow accurate, multidisciplinary predictions of all three goals in parallel with the maturation of these technologies, and will enable optimization at the system and vehicle-level. Today, we are unable to predict emissions, specifically NOx, with better than 50% accuracy. For unconventional configurations necessary to meet long-term goals, current errors in our predictions of noise are at least 10 db. Further improvements in fuel burn can only be predicted with approximately 50% accuracy. The necessary improvements in predictive capability (as illustrated by our long-term measures) are clearly very ambitious from a technology and engineering standpoint.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Aeronautics Technology Theme has specific annual targets to demonstrate progress toward its long-term output goals. These outputs support the long-term outcome goal of the program. Because of the long-term nature of the program's research, annual outcome goals cannot be devised.

Evidence: The detailed research plans for each project, which contain milestone and metric descriptions, can be found on the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. The project measures can be found in the Program Performance Measures section of the 2007 PART.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The projects have done assessments of the current state-of-the-art technology as well as previous related research in order to establish baselines. In addition, based on the feedback received during each step of the restructuring process, ARMD is confident that its annual measures are ambitious. As previously noted, ARMD put together a four-step process to restructure its programs in FY06. The process began with NASA researchers, using input from other Government agency partners and stakeholders, developing preliminary ten-year roadmaps that included technical milestones for each project. Next, ARMD released an RFI soliciting interest from industry for non-reimbursable cooperative partnerships, and received numerous responses. NASA researchers then took the original roadmaps, incorporated the feedback from respondents to the RFI, as well as from colleagues in other Government agencies, and developed refined technical proposals for each project. These proposals were then reviewed by panels of Government subject-matter experts on their technical, management, resource, and partnership plans.

Evidence: The detailed research plans for each project, which contain milestone and metric descriptions, can be found on the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. The project measures can be found in the Program Performance Measures section of the 2007 PART. Specifically, one example of the ambitious annual measures for aeronautics research is that of the Integrated Resilient Aircraft Controls Project in the Aviation Safety Program. The 2008 annual measure is ambitious considering the fact that no adaptive control systems are currently flying in any air or space vehicles. Three sources are cited as part of the state-of-the-art assessment done in setting the annual targets for the project.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ARMD clearly communicates the annual and long-term goals of the programs in its collaboration and coordination with academia, industry and other Government agencies. The mechanisms utilized for these collaborations are Memoranda of Agreement (MOA), Memoranda of Understanding (MOU), SAAs, NRAs, and Technical Working Groups (TWGs). The research plans for the 10 projects within the four ARMD programs contain long term project goals as well as specific research milestones and metrics that have been established to enable the achievement of those goals and that permit assessment of the work that will be conducted. These project goals and milestones are included in all solicitations and contracts. Proposals are required to include a set of milestones that can be used to help ensure that the funded effort will be aligned with project needs. The outcomes from all solicited research will be evaluated to ensure that the partner is making sufficient progress toward achieving the stated research objectives. In particular, every subtopic in the Research Opportunities in Aeronautics (ROA) NRAs contains a list of the specific project milestones that the proposal is intended to support. One requirement in the ROA NRA is that proposals should include a set of milestones that can also be used to help ensure that the funded effort will be aligned with project needs. The use of cooperative agreements is favored over grants to help promote a close working relationship and alignment of research between NASA and its partners. The outcomes from all solicited research will be evaluated to ensure that the partner is making sufficient progress toward achieving the stated research objectives. The performance information collected from ARMD's partners includes task/milestone status as well as budget status.

Evidence: Milestones for each project, and details on ARMD's ROA NRA process can be found at the ARMD website at http://www.areonautics.nasa.gov/nra.htm.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: ARMD activities are tracked and monitored by the NASA Program Management Council (PMC) through the State of the Agency review. The Agency implements a quarterly performance review, entitled the "State of the Agency" process to independently evaluate programs and projects. On a monthly basis, an independent assessment team (consisting of PA&E, OCE, OSMA) provides the Agency with a performance assessment and rating for all key programs and projects. The results of the independent assessment (technical, schedule, cost, & programmatic) are reported at the Agency Program Management Council (PMC) and informs decisions at the Agency and the Mission Directorate (program) level on the programs and project directions. The PMC reviews performance information, including task/milestone status, resource usage, budget status (commits, obligations and costs), status of partnerships and agreements, NRA status (new solicitations and awardee performance), and any identified issues. Furthermore, ARMD will invite subject matter experts from other Government agencies, Federally Funded Research and Development Centers (FFRDCs), as well as experts internal to NASA who are not participants in it the ARMD projects, to participate in its six-month and annual reviews, and the findings and recommendations of the annual review will be documented. In addition to these regular reviews, the Aeronautics Committee of the NASA Advisory Council provides input each year reflecting community opinions and recommendations. In 2006 the National Research Council (NRC) issued a Decadal Survey for Civil Aeronautics to provide a strategy for the federal government's involvement in civil aeronautics, with particular emphasis on NASA's research portfolio. The survey identified fifty-one research and technology challenges with five common themes. Forty-seven of the challenges and all five themes are represented in the ARMD research portfolio. The NRC is currently assessing NASA's fundamental aeronautics research against the needs of the Nation, with a report due in early 2008. ARMD's ground test facilities were assessed by RAND in 2004, and a follow-on study for the Office of Science and Technology Policy is due for release in 2007.

Evidence: The charter, membership, presentations and minutes from the NASA Advisory Council can be found at http://www.hq.nasa.gov/office/oer/nac. The NRC Decadal Survey for Civil Aeronautics can be found at http://www7.nationalacademies.org/aseb/AeroDecadal_Main_Page.html. The RAND report - "Wind Tunnels and Propulsion Test Facilities: An Assessment of NASA's Capabilities to Serve National Needs," can be found at the RAND website at http://www.rand.org/pubs/monographs/2004/RAND_MG178.pdf.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ARMD's budget requests are explicitly tied to accomplishments of its annual and long-term performance goals. The proposed plans in each ARMD project were reviewed by subject matter experts, who assessed the resource plans, technical goals, technical approach, management approach, and partnership plans. Hence, each project has a budget profile that is closely tied to its project goals. Each project will be reviewed on an annual basis by a panel of subject matter experts who will assess the progress of each project against its technical milestones and who will make recommendations regarding any technical, managerial, or budget deficiencies. Budget requests for each project for the following year will be informed by the outcome of these reviews as well as the reviews conducted by the agency's PMC. NASA presents its budget with an Integrated Budget and Performance Document (IBPD) that explicitly ties together its resource request, program activities, and annual goals. The IBPD presents the full cost of programs, including agency overhead.

Evidence: The technical research plans for each project can be found at the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. ARMD budget request and annual and long-term performance measures are detailed in the NASA FY08 IBPD, and performance against each annual measure is reported to Congress each year in the Performance and Accountability Report. Both documents can be found at the NASA Budget Information website at http://www.nasa.gov/about/budget/index.html.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The restructured aeronautics portfolio ensures that all research in each project is now integrated, with all milestones linked together to support the project's long-term strategic goals. This is in stark contrast to the isolated '1000 flowers blooming' approach of the recent past, an approach that had been criticized by many, including the NRC. The restructured programs have also set out to prevent "stove-piping" by focusing on means to ensure that research can be leveraged across projects, programs and centers. Program Directors have established interdisciplinary teams to work across projects and programs, and projects are collaborating on joint-NRA topics that are cross-cutting. Examples include Multidisciplinary Design Analysis and Optimization (MDAO) in Fundamental Aeronautics, metroplex operations in the Airspace Systems Program, Verification and Validation of Complex Systems in Aviation Safety, and runway incursion prevention across the Aviation Safety and Airspace Systems. In addition, all of the programs are looking at ways to coordinate system studies of the future air transportation system that include the vehicle, safety, and air traffic management challenges. Finally, the approach that ARMD used to restructure the aeronautics research portfolio was designed to enable ARMD to foster close collaboration with and to facilitate the exchange of ideas and information among researchers at NASA, industry, academia, and other Government agencies, in a manner that broadly benefits the long-term, strategic needs of the community. This prevents ARMD's research from being driven by the needs of a select few individual companies. One tool that ARMD has to ensure this type of collaboration is the NRA, which ensures full and open competition for the best and most innovative research ideas from the external community (industry, academia, and non-profits). Finally, ARMD developed technical metrics and demonstrated quantitative progress against those metrics by meeting all of its 2006 targets included in the 2007 PART measures. In addition, by defining new Aeronautics performance measures in the 2007 PART, ARMD has successfully completed the six performance improvement areas that were identified to increase its last PART rating of "Moderately Effective."

Evidence: The research plans for each project can be found at http://www.aeronautics.nasa.gov/programs.htm. Details about the NRAs can be found at http://www.aeronautics.nasa.gov/nra.htm. The performance improvement actions assigned to ARMD from the 2004 PART, as well as comments on their status, can be found at http://www.expectmore.gov.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: ARMD's Aeronautics Test Program (ATP) has undergone an analysis of alternatives to ensure the strategic availability of a minimum, critical suite of aeronautical test facilities that are necessary to meet the long-term needs and requirements of the agency and the nation. To determine that suite of facilities, ATP collected facility usage projections from each of the projects in the restructured aeronautics portfolio. In addition, ATP collected facility usage projections from other parts of the agency, and facility usage projections from other Government agencies, most notably, the DOD. ATP also relied heavily on a recent facility study completed by RAND. Currently, OSTP is directing and ARMD is funding a follow-on study by RAND on aeronautics test facilities that will build upon the 2004 study, and the results of that report will inform ATP's decisions going forward. Regarding the facility usage projections of the ARMD projects, each project looked at the capabilities, cost, and schedule availability of various facilities at NASA, DOD, and industry in order to determine where to conduct its tests. Each project included this information in its technical plans which were reviewed by subject matter experts from other Government agencies as well as NASA researchers who were not participants in the ARMD projects (as described previously).

Evidence: The suite of facilities covered by the ARMD Aeronautics Test Program, can be found at http://www.aeronautics.nasa.gov/atp/index.html. In addition, the RAND report - "Wind Tunnels and Propulsion Test Facilities: An Assessment of NASA's Capabilities to Serve National Needs," can be found at the RAND website at http://www.rand.org/pubs/monographs/2004/RAND_MG178.pdf

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The program does not pursue multiple strategies to achieve its benefits, and there are no other programs designed to achieve the same benefits.

Evidence: ARMD's role is unique from that of any other Government or private entity, as is clearly articulated in the National Aeronautics R&D Policy. As stated in that policy, NASA should maintain a broad foundational research effort aimed at preserving the intellectual stewardship and mastery of aeronautics core competencies, including those that support NASA's robotic and space activities.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: All program decisions in ARMD are guided by three core principles: 1) We will dedicate ourselves to the mastery and intellectual stewardship of the core competencies of aeronautics for the Nation in all flight regimes; 2) We will focus our research in areas that are appropriate to NASA's unique capabilities; and 3) We will directly address the fundamental research needs of NGATS while working closely with our agency partners in the JPDO. These principles led to the termination of several projects more appropriate for other agencies. These principles guided the planning process described in the response to Question 1.4, and led to the project performance measures discussed in Section II. Each project will be reviewed on an annual basis by a panel of subject matter experts from other government agencies, who will assess the progress of each project against its technical milestones and who will make recommendations regarding any technical, managerial, or budget deficiencies. Budget requests for each project for the following year will be informed by the outcome of these reviews as well as the reviews conducted by the Agency's PMC. In addition, subject matter experts from other government agencies, especially the JPDO, will be consulted in the event of a budget reduction or augmentation to ensure the ARMD priorities are in-line with the high priority needs of these agencies.

Evidence: Technical plans and milestones for each ARMD project, the critical facilities covered by the Aeronautics Test Program can be found at http://www.aeronautics.nasa.gov/programs.htm. Further information on ARMD's planning process can be found at http://www.aeronautics.nasa.gov/present.htm.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Aeronautics Technology Theme collects relevant technical and programmatic performance data quarterly. The performance information collected includes task/milestone status, resource usage, budget status (commits, obligations and costs), status of partnerships and agreements, NRA status (new solicitations and awardee performance), and any identified issues. This information is used to assess progress toward meeting long-range outcomes, and can be used to develop risk mitigation strategies, adjust priorities, make additional resource allocation or take other appropriate management actions.

Evidence: ARMD programs and projects are subjected to regular reviews and evaluations to ensure their effectiveness and relevance. At the program level, status reviews are conducted weekly and data (budget, FTEs and procurement status) is collected monthly. The programs are internally tracked and monitored at the mission directorate level through quarterly reviews, and are subsequently tracked and monitored by the NASA Program Management Council through the State of the Agency review. ARMD will invite subject matter experts from other Government agencies, FFRDCs, as well as experts internal to NASA who are not participants in the ARMD projects, to participate in the annual review, and the findings and recommendations will be documented. All grants, cooperative agreements, and contracts will be individually assessed on at least an annual basis to ensure sufficient progress is being made. The ROA NRAs require proposals to include a set of well-defined research milestones with metrics, which will facilitate an effective progress review. Details on the ROA NRAs can be found at the ARMD website at http://www.aeronautics.nasa.gov.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and program partners are held accountable for cost, schedule and performance results through a series of formal procedures and requirements. Federal managers who fail to demonstrate the required level of performance are subject to a variety of disciplinary actions. Partners who likewise fail to demonstrate the required level of performance may find their level of participation in the program either diminished or terminated. In addition, obligations for all selections and awards from the ROA NRA are closely tracked.

Evidence: All ARMD Program Directors report directly to the ARMD Associate Administrator and are accountable for their program cost, schedule and performance results. While their performance is formally evaluated annually through the OPM approved Agency performance evaluation process, their performance is routinely monitored and assessed through frequent reviews that ARMD conducts. Bonuses and promotions are dependent upon the Program Director's performance in making positive progress toward meeting the goals of the program. The performance of the project leadership team (Principal Investigator (PI), Project Manager (PM), and Project Scientist (PS)) is monitored and assessed by the ARMD Program Director. The Program Director provides input to the center supervisors of the project leadership team regarding their performance during the annual performance evaluation. The Program Director has full authority with concurrence of the AA to remove any project level managers who fail to meet the required level of performance. Regarding program partners, ARMD launched an extensive, competitive process last fiscal year utilizing the NRA procedure to solicit proposals from educational institutions, non-profit organizations, and industry engaged in foundational research. The NRAs are typically awarded for multiple years; however the decision for continued funding is contingent on effective technical, schedule and cost management performance at the end of each year. In addition, a final report, called a "Summary of Research," is due within 90 days of the expiration date of the award. Details on the NRA process can be found at the ARMD website at http://www.aeronautics.nasa.gov. A list of awardees is also posted on the ARMD website at http://www.aeronautics.nasa.gov/nra2_research_awards.htm. ARMD awards its supplies and services contracts as performance-based. Such contracts set forth requirements with "measurable outcomes as opposed to either the manner by which the work is to be performed or broad and imprecise statements of work" (FAR 2.101b). Finally, it should be noted that "other government partners" are held accountable to their respective agencies for cost, schedule and performance results. However, when ARMD partners with other government agencies, reviews occur routinely between and among all parties to ensure commitments are delivered on schedule and within allotted budget.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: ARMD's new program/project work breakdown structure was implemented in FY06 and allows more visibility into the expenditure of funds, allowing Aeronautics' program managers to verify that funds are being spent for the intended purpose. Operating plans for the program year are submitted to Congress and are revised as needed if changes in program expenditures need to be made. Internally, full cost obligation and cost plans are developed for both current and prior year funding, compared to actual spending, and are reviewed monthly by the Program Directors at all levels of the program and reviewed by the ARMD AA at the program and project level. ARMD's goal is to obligate 100% of its funds in the year appropriated. If less funds are required in a project during the year in a particular full cost element (e.g., salaries), then funds are available to the PI for reprogramming into other budget elements (e.g., procurement) based upon project priorities and concurrence from the Program Director. Contingency planning by ARMD's programs and projects facilitates the ability to efficiently and effectively realign funding during the year to maximize the benefit to project deliverables and milestones. Relative to the reporting of out-of-house program awards, ARMD is providing periodic reporting of all of its competitive peer-review NRA awards to OMB and Congress throughout the year. The balance of major ARMD procurement awards is associated with performance-based contracts to provide facility testing and on-site research support at NASA's Research Centers.

Evidence: Over the past decade, actual direct program end-of-year obligations have ranged between 94 to 98 percent by the end of the fiscal year. Routine plan versus actual reporting to ARMD management and the Office of the Chief Financial Officer (OCFO) on aeronautics spending by full cost element is the process used to monitor how effectively the programs utilize funds and workforce guide-lined to them. In addition, monthly reports depicting NRA obligations by contractors and universities are reviewed along with plan vs. actual data. Programmatic milestone and deliverable impact assessments, end-of-year forecast, and recovery plans are also included in ARMD's monthly budget execution status reports.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The ARMD programs have adopted effective management procedures to ensure that programs are executed in a cost-effective and efficient manner. ARMD programs have policies to incentivize competitive outsourcing, best-value procurement practices, and employee performance and productivity improvements. All in-house research was formulated through an inter-center proposal process that ensures full coordination and integration of the skills and facilities at the research centers to achieve high levels of efficiencies and effectiveness. Competitive sourcing for purchasing any on-site services or hardware is fully in place and is being practiced to support the program implementation at the research centers. In addition, full and open competition for all out-of-house research through the NRA process is fully in place for all ARMD programs. The decision for continued funding of the proposed research on all ARMD NRA contracts, grants, and cooperative agreements is contingent on effective technical, schedule and cost management performance during the year. Finally, ARMD has adopted an efficiency measure for its test facility. The goal is to minimize unscheduled downtime while keeping maintenance costs fixed. Unscheduled facility down-time, often caused by unplanned maintenance and repairs, results in lost revenue and not meeting contractual commitments to various facility customers (NASA, other government agencies, industry). The cost to operate/run the ATP facilities is roughly $500K each day. These costs can be reimbursed during active testing. Therefore, it is in the best interest of ARMD and the agency to do strategic facility maintenance in order to reduce the Deferred Maintenance liability for ATP facilities and maximize facility on-time availability. The cumulative effect of doing annual maintenance is what translates into an efficiency for the ATP facilities. The target is % on-time availability based on a fixed annual maintenance investment, and through data collection, it was determined at the onset of the Aeronautics Test Program that the actual percentage or baseline in 2005 was 83%.

Evidence: The project measures can be found in the Program Performance Measures section of the 2007 PART.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The approach that ARMD used to restructure the aeronautics research portfolio was designed to foster close collaboration with and to facilitate the exchange of ideas and information among researchers at NASA, industry, academia, and other Government agencies, in a manner that benefits the community broadly. ARMD works closely with other government agencies that conduct aeronautics research to assess and compare the benefits of efforts within its own portfolio. ARMD has recently signed MOUs with the FAA, USAF, DOD (for aeronautical test facilities), and JPDO, and is in the process of finalizing an MOU with the Army (for collaborative research in rotorcraft aeronautics). These MOUs establish strategic goals for collaboration in particular research areas, and typically establish the mechanism by which these collaborations will be conducted (e.g., Executive Councils and/or Boards). NASA and the Air Force held their first ERC meeting in January 2006, and the Committee members have subsequently identified a number of research areas that should be examined for closer collaboration, particularly in our aviation safety projects (e.g., Integrated Vehicle Health Management and Integrated Resilient Aircraft Control), and these will be pursued. In addition, there are a number of activities in which the Air Force and NASA are already closely collaborating, including hypersonics partnerships (X-51 and HiFire) and subsonic fixed wing partnerships (X-48B and CESTOL). NASA's research is closely coordinated with the JPDO by means of participation in the Airspace and Safety IPTs, Architecture and Portfolio activities, and the JPDO Board. Furthermore, NASA coordinates with the FAA in safety-related issues such as data mining by establishing joint efforts to focus on transitioning NASA research into the hands of the FAA. ARMD also interfaces with its Government partners by means of interagency working groups and the NSTC AS&T subcommittee, as described in more detail in question 1.3. ARMD also cooperates with the other Mission Directorates within NASA on a variety of topics of broad strategic relevance, particularly to the Vision for Space Exploration. For example, ARMD has participated with the Science Mission Directorate (SMD) and the Exploration Systems Mission Directorate (ESMD) in the Mars Science Laboratory Entry Descent and Landing Instrumentation (MEDLI) effort, in which the Mars Science Laboratory will be instrumented to collect thermal protection system and aerothermodynamic data relevant to all mission directorate future needs. ARMD will analyze the data and use the data to improve its modeling tools. ARMD is also a participant in the Mars Architecture System Studies Group along with ESMD, SOMD, and SMD, and ARMD recently coordinated a Boundary Layer Transition Workshop with participation from ESMD, SOMD, the Air Force, and DOE to begin laying the groundwork for a national research plan for long-term foundational research in boundary layer transition.

Evidence: The MOUs with the AF, DOD, and the FAA are found at http://www.aeronautics.nasa.gov/partners.htm. JPDO planning documents that have informed ARMD research goals, including the latest ConOps, can be found at the JPDO website at http://www.jpdo.aero. Details about partnerships with the USAF in hypersonics and subsonic fixed wing can be found at http://www.aeronautics.nasa.gov/programs_fap.htm, as well as in the recent presentation at the AIAA Reno conference, found at http://www.aeronautics.nasa.gov/present.htm.

Does the program use strong financial management practices?

Explanation: The most recent Independent Auditor report for NASA identified two (2) material weaknesses, all of which are repeats, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: NASA's FY 2006 Performance and Accountability Report (www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the Independent Auditor. In addition, the GAO has published numerous reports identifying shortcoming in NASA's new financial management system as well as its financial management processes (example is GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: ARMD is responsible for the Aeronautics Technology Theme and for managing four programs, and has a well-structured process in place to conduct quarterly and annual performance reviews. Each program is required to present management, technical, and financial status information at these reviews held at NASA Headquarters. Further, each program is evaluated and management deficiencies are noted through the budget formulation process. It should be noted that beginning in FY07, each of the ARMD projects will be evaluated annually by a panel of internal and external technical experts to evaluate the projects' performance and progress. The Program Director has frequent contact and interaction with the project leadership team (PI, PM, and PS) at the implementing research centers to discuss and mitigate any management deficiencies. One management challenge with multi-center research projects is that a person responsible for a given set of tasks (e.g., acoustics) will need to monitor what is being done at multiple centers to achieve the milestones. Therefore, one of the evaluation criteria during the 6-month and 12-month reviews is how the PI's project leadership team is addressing this challenge. The 6-month and 12-month review panels will use the technical progress of each project against its milestones as well as its financial/budget information (e.g., planned vs actual obligation data) to assess how well this management challenge is being addressed. Program Directors will assess this issue much more frequently as part of their project and program tag-ups.

Evidence: ARMD programs and projects are subjected to regular reviews and evaluations to ensure their effectiveness and relevance. At the program level, status reviews are conducted weekly and data (performance and financial) is collected monthly. The programs are internally tracked and monitored at the mission directorate level through quarterly reviews, and are subsequently tracked and monitored by the NASA Program Management Council through the State of the Agency review. ARMD will invite subject matter experts from other Government agencies, FFRDCs, as well as experts internal to NASA who are not participants in the ARMD projects, to participate in the annual review, and the findings and recommendations will be documented.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: The Aeronautics Test Program (ATP) is a long-term, funded commitment by NASA to retain and invest in test capabilities that are considered to be important to the Nation and the Agency. Through the ATP, the Agency will adopt consistent management processes and procedures across the NASA Research Centers as it relates to the operations and maintenance of the ATP ground and flight test facilities. The ATP will review the status of its test facilities annually. In doing so, NASA is ensured that near-term decisions, such as that to close a facility, have financial gains relative to long-term capability risks. The ATP will cooperate with the DOD to coordinate investments. There are four focus areas in the ATP - facility operations support, facility maintenance, facility upgrades and test technology, and facility related research - and each has clearly defined deliverables, budget and schedule goals.

Evidence: The suite of facilities covered by the ARMD Aeronautics Test Program, can be found at http://www.aeronautics.nasa.gov/atp/index.html. The annual and long-term program commitments for ATP can be found in the FY08 IBPD which is available at NASA's Budget Information website at http://www.nasa.gov/about/budget/index.html. ATP exceeded its performance measure target in 2006, and details of the measure and targets can be found in the Performance Measures section of the 2007 PART.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: ARMD has developed the ROA NRA, which covers all of the mission directorate programs and projects. The ROA NRA is the primary process for awarding external research opportunities. Awards that result from this NRA may be in the form of grants, cooperative agreements, or contracts. NASA chooses the award instrument based on a determination of the appropriate funding mechanism from a procurement standpoint for the particular selected proposal. Competition for 100% of the ROA NRA is full and open. Panels of both NASA and external reviewers are used to assess each proposal's intrinsic scientific and technical merit, its relevance to the Program's stated objectives, and its cost realism and reasonableness. The evaluation criteria for every solicited topic are defined within the ROA NRA and the NASA Guidebook for Proposers. Prior to conducting an evaluation, each reviewer undergoes a rigorous conflict of interest check to ensure he or she can make a fair assessment. The Project leadership team utilizes the results of these panels to formulate a recommendation for selection or non-selection of each proposal. These recommendations are briefed to the NASA Selecting Official, who then makes the selection. Afterward, each proposer is notified by letter or electronic mail of the disposition of his/her proposal. ARMD follows standard NASA procedures for continuation of multi-year awards as described in the NASA Guidebook for Proposers. Two components for renewal are availability of funds and sufficient technical progress. The NRA specifies that proposers must include a work plan that describes milestones with metrics. The work plan is part of the initial evaluation criteria for selection and also facilitates an annual assessment of progress by ensuring that there is a mutual understanding of what progress is expected. In order to ensure the widest outreach possible, several venues are used to publicize the NRA. ARMD follows standard NASA procedures for synopsizing the solicitation before release on the NASA Acquisition Internet Service (NAIS). There is also a special section of the ARMD website devoted to information about the NRA. Potential awardees can search for and access the ROA NRA on both the Grants.gov and NSPIRES electronic systems. Press releases have also been issued for the initial release of the NRA as well as for each amendment when new research topics have been added. Finally, information on the NRA process is also provided at a variety of public events such as the AIAA Aeronautics Conference in Reno.

Evidence: The ROA NRA is available on the NSPIRES system at http://nspires.nasaprs.com. A formal evaluation process is defined by the NRA itself and the NASA NRA Proposers Guidebook available at http://www.hq.nasa.gov/office/procurement/nraguidebook/. All evaluations are conducted by highly qualified peers internal and external to NASA. In the first round of proposals from the ROA NRA, which was released in May 2006, over 600 subject matter experts from both NASA and other Government agencies participated in the evaluation, which resulted in 138 selections. The process for the renewal of awards is described in the NASA Guidebook for Proposers. This process is supported by the requirement in the NRA for work plans that include milestones with measurable metrics. The pre-solicitation synopsis is available at: http://prod.nais.nasa.gov/cgi-bin/eps/bizops.cgi?gr=D&pin=04#119999. Information for the NRA and examples of presentations with NRA references are available on the NRA website:http://www.aeronautics.nasa.gov/nra.htm. Press releases related to the NRA can be found at: http://www.nasa.gov/audience/formedia/archives/index.html. As evidence of a wide outreach, in the first evaluation round, ARMD received more than 700 proposals from more than 110 universities and over 120 other organizations (companies and non-profits).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: An important aspect of the ROA NRA is that it requires proposals to include a set of research milestones with metrics to help NASA personnel determine if the grantee is making sufficient progress. NASA technical monitors are also assigned to track the performance of the grantee's activities. All grants, cooperative agreements, and contracts will be individually assessed on at least an annual basis to ensure sufficient progress is being made. Finally, all ARMD Projects are planning to host forums where grantees will present the progress of their work to a wider audience.

Evidence: Details on the ROA NRAs can be found at the ARMD website at http://www.aeronautics.nasa.gov/nra.htm.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NASA collects grantee performance data and makes it available to the public in a manner that is both useful and meaningful. All research results, whether generated by NASA internally, or by its partners through the NRA, will be openly disseminated through archival publications and conference proceedings as well as NASA publications (consistent with national security and foreign policy guidelines). Such conditions are stipulated in the NRA itself; specifically, the following clause is included in the evaluation criteria: "Annual oral presentations made as part of an open Technical Exchange Meeting for purposes of technology transfer and knowledge dissemination will be expected. Documentation of approach and results in the form of final written technical reports is required. A clear statement of what intellectual property is expected to be publicly available at the conclusion of the work is required. It is our intent to share all knowledge developed under this solicitation, thus, any restrictions to that objective will cause a lower score in this area. Collaboration with NASA researchers (including joint use of facilities, sharing of materials, development of computer code modules compatible with NASA software, and synergistic research goals) is desirable, with the objective of enhancing knowledge transfer and the long-term value of the proposed work." Formal progress reports, which are a required output of each research and analysis activity, are submitted on an annual basis. NASA researchers review the progress reports before recommending continuation of the research activity. All projects are planning on hosting annual interchange meetings to discuss the progress and results of funded research. In addition to these meetings, NASA researchers are encouraged to publish high-quality technical publications, which will also incorporate the relevant results of funded research when possible.

Evidence: Details of the ROA NRA can be found at http://www.aeronautics.nasa.gov/nra.htm.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Peer-review is an important element of ARMD's research, both in-house and out-of-house, to ensure program quality. In addition, ARMD participated in the agency's SBIR initiative and has taken positive steps to ensure that SBIR topics are aligned with the Mission Directorate's research needs. A formal competitive process is used to ensure that highly meritorious proposals are selected and that they are aligned with program and project objectives. As described in the response to Question 1.4, ARMD went through a four-step restructuring process that began in the fall of 2005. The entire proposed approach of each project, from the technical goals to the technical approach proposed to achieve those goals, the management approach of the project, the resources to be applied against that technical approach, and the proposed partnerships with other agencies as well as the private sector, were all thoroughly reviewed by panels of subject matter experts from other Government agencies. Such experts were chosen to ensure that the technical assessments were conducted by people who really understood the state-of-the-art and the realm of the possible for each area, and they were also chosen to ensure that ARMD avoided any conflict of interest. In addition, the technical scope of ARMD's portfolio aligns very well with the recommendations from the independently conducted 2006 NRC Decadal Survey. Thus, the program and project goals, as well as the detailed approach to achieving those goals, are well-vetted by experts external to NASA. NASA's "division of labor" between in-house and out-of-house research to implement those plans took into account its first core principle, to dedicate itself to the mastery and intellectual stewardship of the core competencies of aeronautics for the Nation in all flight regimes. The National Aeronautics R&D Policy recently identified this as a critical role for NASA in the advancement of U.S. technological leadership in aeronautics. Intellectual stewardship necessitates a robust in-house expertise that is unbiased by conflicts of interest (and therefore ensures that the research conducted remains pre-competitive and broadly applicable), that is capable of conducting world-class research, that is capable of assessing the quality and relevance of external research, and that can integrate multiple research efforts in a manner that advances the field of aeronautics as a whole. ARMD therefore ensured that for each core competency that was identified in the project plans (e.g., acoustics, combustion, aerodynamics), sufficient in-house expertise existed and could be applied to the effort to achieve these objectives. In many cases, research areas were identified that required an enhancement of NASA's in-house expertise to properly address the research goals in those areas. Those "gaps" were what informed the NRA solicitation topics, and it should be noted that those "gaps" were also reviewed and agreed upon by the external reviewers. The proposal process helped ARMD to ensure that no civil servants were placed on the projects that did not have the appropriate expertise, and has resulted in a downsizing of our civil servant workforce (42% from FY05 through FY12), as well as a better-tuned workforce (e.g., a significant reduction in systems engineers and project managers).

Evidence: The description of the four step restructuring process can be found at http://www.aeronautics.nasa.gov/present.htm. The technical plans for each project can be found at the ARMD website at http://www.aeronautics.nasa.gov/programs.htm.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: As part of the restructuring of the aeronautics research portfolio in 2005, ARMD created new long-term performance measures that reflect the goals of the new organization. The long-term performance measures include annual targets to monitor progress. All of the 2006 targets were met.

Evidence: The detailed research plans for each project, which contain milestone and metric descriptions, can be found on the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. The project measures can be found in the Program Performance Measures section of the 2007 PART.

Does the program (including program partners) achieve its annual performance goals?

Explanation: As part of the restructuring of the aeronautics research portfolio in 2005, ARMD created new long-term performance measures that reflect the goals of the new organization. The long-term performance measures include annual targets to monitor progress. All of the 2006 targets were met.

Evidence: The detailed research plans for each project, which contain milestone and metric descriptions, can be found on the ARMD website at http://www.aeronautics.nasa.gov/programs.htm. The project measures can be found in the Program Performance Measures section of the 2007 PART

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Aeronautics Technology Theme has an efficiency measure tied to the strategic management of test facilities in its Aeronautics Test Program, and successfully met its target in 2006. In addition, ARMD has reorganized and has put in place a management approach that ensures that it will effectively and efficiently achieve its annual goals. As part of this reorganization, program management responsibility for multi-center, multi-discipline programs was brought back to NASA Headquarters which minimized center-specific and stove-piped approaches to conducting aeronautics research. The Program Offices are comprised of a Program Director, Deputy Program Director, Technical Integration Manager and a Program Integration Manager. In addition, the project leadership teams (PI, PM, and PS) for each project have been selected. The establishment of the program offices and the leadership teams at the centers has enabled the programs to operate effectively and efficiently during the initial implementation efforts. In addition, as noted previously, ARMD has developed the ROA NRA, which covers all of the mission directorate programs and projects, and is the primary process for awarding external research opportunities. In FY06, ARMD utilized NSPIRES/Grants.gov/E-gov to run its large, complex, national proposal review and selection process, thereby enabling the efficient and effective selection of 138 proposals for negotiation for award. Finally, ARMD continues its use of NX and PBMA as an online collaboration tool to decrease information sharing overhead and enable distributed program and project management.

Evidence: The ATP efficiency measure is detailed in the 2006 PAR which can be found at the NASA Budget Information website at http://www.nasa.gov/about/budget/index.html, as well as in the Performance Measures section of the 2007 PART.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ARMD has been restructured recently, and no studies have been performed since comparing the performance of the program to other applied research programs. Prior to its restructuring, ARMD's PART ratings compared favorably to that of other programs. In 2004, ARMD received a PART rating of "Moderately Effective". In previous PART assessments, other government agencies with programs that fund aeronautics-related R&D were rated as follows: In 2006, DOD's T&E Programs received a "Results Not Demonstrated" rating; in 2002, DOD's T&E Programs received an "Effective" rating; in 2004, the FAA's Aviation Safety Program received a rating of "Moderately Effective"; in 2003, FAA's Research, Engineering and Development Program received a rating of "Effective"; and finally, in 2005, NSF's Fundamental Science and Engineering Program received a rating of "Effective".

Evidence: Details of PART assessments can be found at http://www.expectmore.gov.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: ARMD's most significant assessment since its restructuring is the on-going NRC study of ARMD's research portfolio against the research needs of the Nation. This assessment has not yet been completed. The results are due in 2008. As mentioned previously, ARMD activities are tracked and monitored by the NASA Program Management Council (PMC) through the State of the Agency (SOA) review. The most recent SOA review for ARMD was held in November of 2006. The main issue raised at that review was the fact that two projects, one in the Airspace Systems Program and one in the Aviation Safety Program, still needed to address some concerns raised by reviewers during the project formulation process and could not proceed into implementation until those had been addressed. The concerns have since been fully addressed and both projects are now underway. ARMD held its six-month reviews in May of 2007 and had participation from subject matter experts from other Government agencies, as well as experts internal to NASA who are not participants in the ARMD projects. A few minor management issues, and approaches to mitigate them, were discussed. In addition, many of the results presented fostered good technical discussions among the reviewers. ARMD programs were recently reviewed by the NAC Aeronautics Committee, and they concluded that "the agency is on the right track to build the kind of relationship to work with industry in a different paradigm than in the past." To date, ARMD has sufficiently addressed all four recommendations to date from the NAC Aeronautics Committee. In addition, NASA's new aeronautics research portfolio aligns very well with the recommendations of the 2006 NRC Decadal Survey for Civil Aviation, which was conducted independently of ARMD's restructuring. All five of the Common Themes identified in the Decadal Survey are present across ARMD's research programs, and 47 of the 51 technical challenges are also well represented in the portfolio. Another study was completed by RAND in 2004 on NASA's aeronautics ground test facilities, resulting in the published report on "Wind Tunnels and Propulsion Test Facilities: An Assessment of NASA's Capabilities to Serve National Needs." The RAND study concluded that of the 31 existing major NASA test facilities, 29 constitute the "minimum set of facilities important to retain and manage to serve national needs." ARMD considered the results of the RAND study, along with the facilities that NASA felt were important for future use, in the formulation of its Aeronautics Test Program. Many of the facilities that NASA considered important were also deemed critical in the RAND study.

Evidence: The charter, membership, presentations and minutes from the NAC can be found at the NAC website at http://www.hq.nasa.gov/office/oer/nac. The NRC Decadal Survey for Civil Aviation can be found at the ARMD website at http://www.aeronautics.nasa.gov. The RAND report - "Wind Tunnels and Propulsion Test Facilities: An Assessment of NASA's Capabilities to Serve National Needs," can be found at the RAND website at http://www.rand.org/pubs/monographs/2004/RAND_MG178.pdf. Information on the NRC's assessment of ARMD can be found at http://www8.nationalacademies.org/cp/projectview.aspx?key=48781.

Were program goals achieved within budgeted costs and established schedules?

Explanation: ARMD's Aeronautics Test Program (ATP) met all of its cost and schedule goals for the facility related projects in FY06. In particular, ATP's accomplishments include exceeding the projected utilization of ATP ground test facilities, reducing the backlog of overdue maintenance activities in the ATP facilities, initiating test technology investments including standardizing wind tunnel measurement systems across all NASA Research Centers and developing test facility control system simulators. In addition, both the Constellation Program Test and Verification Office and the ATP have critical responsibilities in accomplishing the Agency's exploration mission. To ensure the successful fulfillment of each organization's responsibilities, they have entered into a Strategic Alliance Agreement. Furthermore, ATP recently entered into an agreement with DOD, the National Partnership for Aeronautical Testing, to better integrate the operations and planning of the Nation's aeronautics testing assets. In addition, ATP is working with DOD to develop an interagency charging and accessibility policy for aeronautics test facilities as part of a plan to strengthen the co-reliance of the two agencies on each other's aeronautics ground test facilities. It is also worth noting that ATP exceeded its efficiency measure target in 2006.

Evidence: The ATP efficiency measure is detailed in the 2006 PAR which can be found at the NASA Budget Information website at http://www.nasa.gov/about/budget/index.html, as well as in the Performance Measures section of the 2007 PART. Details of goals and structure of ATP as well as a copy of the newly signed NPAT agreement can be found at the ARMD website at http://aeronautics.nasa.gov. In addition, ATP has signed an agreement with the Constellation Test and Verification Office.