This is the accessible text file for GAO report number GAO-05-659R entitled 'Defense Transportation: Opportunities Exist to Enhance the Credibility of the Current and Future Mobility Capabilities Studies' which was released on September 14, 2005. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. September 14, 2005: The Honorable Donald H. Rumsfeld: Secretary of Defense: Subject: Defense Transportation: Opportunities Exist to Enhance the Credibility of the Current and Future Mobility Capabilities Studies: Dear Mr. Secretary: We are reviewing the processes the Department of Defense (DOD) is using to conduct its Mobility Capabilities Study (MCS). The MCS is to address changes in DOD's transportation force structure and mobility requirements due to changes in threats and certain national security and military strategies. The study results may underpin decisions on future strategic airlift, aerial refueling aircraft, and sealift procurements. The study relies on the use of various models and data inputs to develop and evaluate transportation alternatives, including variations in alternative transportation modes (air, land, sea) and sources (military, civilian, foreign), as well as factors that affect transportation mode and source decisions. The Senate Armed Services Committee directed us to monitor the conduct of the MCS and report on the adequacy and completeness of the report no later than 30 days after DOD completes the study.[Footnote 1] As you may be aware, DOD plans to issue the MCS report during 2005. This letter is intended to bring to your attention preliminary observations on certain aspects of the MCS methodology to permit you to ensure the credibility of this and future studies. In our letter, we address the adequacy of the department's verification, validation, and accreditation (VV&A) of the models and simulations being used to conduct the MCS--that is, the process the MCS team[Footnote 2] is using to identify the models' capabilities, limitations, and performance relative to the real world events they simulate. We will continue to monitor the MCS and will report on the adequacy and completeness of the methodology after DOD issues its report. To do our work, we reviewed applicable DOD guidance, directives, instructions, and memos that describe how DOD would conduct its mobility capabilities assessments to include the National Security and Military Strategies; DOD Strategic Planning Guidance; DOD data collection, development, and management in support of strategic analysis directives; DOD modeling and simulation instruction; Defense Modeling and Simulation Office guidance; MCS Study Plan and Terms of Reference; descriptions of models used to conduct the study; and the databases used in the models. Additionally, we reviewed and analyzed previously published DOD mobility studies and past GAO reports related to the studies. We interviewed study officials from the Office of the Secretary of Defense, Program Analysis and Evaluation and the office of the Joint Chiefs of Staff, as well as study participants and subject matter experts from the U.S. Transportation Command, Air Mobility Command, Surface Deployment and Distribution Command, the combatant commands, and military services concerning the extent of their input to the study. We interviewed a modeling and simulation subject matter expert at the Defense Modeling and Simulation Office. We interviewed DOD officials to try and identify the process used to ensure the validation and verification of the models and the reliability of the data used in the study models, and conducted a comparative analysis of this process with applicable DOD guidance. We did not evaluate the relevancy of the DOD guidance because it was outside the scope of our work. We conducted this portion of our review from July 2004 through July 2005 in accordance with generally accepted government auditing standards. Results in Brief: We are unable to assess the adequacy of the process DOD used to verify, validate, and accredit the models used to conduct the MCS. Although officials in the Office of Program Analysis and Evaluation stated that they have performed an equivalent VV&A[Footnote 3] process for the models used in the MCS, there is little documentation available to describe the equivalent process that was used. An adequate evaluation of this self-described equivalent process cannot be conducted due to this absence of documentation, which is compounded because DOD currently does not plan to disclose how it conducted its equivalent VV&A process in its MCS report. This could negatively impact the credibility of the MCS report. DOD guidance requires that models and data go through a VV&A process, but officials in the Office of Program Analysis and Evaluation believe that this guidance is not relevant for models that have been used for many years, called legacy models, because, in their view, the models and data have already undergone an equivalent VV&A process consisting of actual use, although the guidance does not identify actual use as an appropriate equivalent process. Moreover, DOD was conducting VV&A on one legacy model being used in the MCS, raising questions about the need for such actions given the department's statements that it is unnecessary. We are making recommendations to improve DOD's documentation of any equivalent VV&A process the department may have and to establish the relevancy of VV&A guidance for use with legacy models. In commenting on a draft of this report, DOD concurred or partially concurred with all of our recommendations. DOD's comments are reprinted in their entirety in enclosure I. Background: The Office of the Secretary of Defense directed its Office of Program Analysis and Evaluation to conduct the MCS. DOD was using an array of models and baseline data to develop transportation alternatives and evaluate their impact on the department's capability to support military strategy. As with past mobility requirements studies,[Footnote 4] the MCS uses a variety of models and data analyses to achieve the overall study objectives[Footnote 5] and determine the effect of the study variables on the defense transportation system and its resultant effect on the capabilities required to meet the mobility needs for all aspects of the National Military Strategy. The baseline data used in the mobility models are the foundations for the strategic analyses and contain such data as a specific warfight scenario, concept of operations for the scenario, needed forces and equipment, battlefield terrain and weather, and time frames. According to DOD officials, the models have become increasingly complex over the past 15 years and are used to analyze large volumes of data to define mobility requirements, assess risk based on the forces' ability to achieve war-fighting objectives, identify mobility gaps, and determine alternative methods to achieve desired capabilities. For example, the analysis would identify tons of equipment or number of passengers to be moved, the number of aircraft and ships needed to move equipment and forces, and the number of aircraft to be aerially refueled. Modeling and simulation are assuming a larger role in military assessments, driven in part by an appreciation for the cost, logistics, and acquisition implications associated with DOD programs. DOD models and simulations are to be developed in accordance with DOD policies, plans, and guidance.[Footnote 6] Generally, overall VV&A policy is established in DOD modeling and simulation master plans. DOD and service instructions clarify the policy and guidance indicates how to implement the policy. VV&A constitutes processes that gather and evaluate evidence to determine, based on the simulation's intended use, the simulation's capabilities, limitations, and performance relative to the real objects or events it simulates. The VV&A process entails the review, analysis, evaluation, and testing of models and simulations, incrementally over time as the models are being developed, by an independent agent or authority to improve the credibility of the process. Furthermore, VV&A provides enhanced user confidence, improved performance and reliability for the subject model and simulation results, more predictable and accurate modeling/simulation behavior, and reduced risk of inaccurate model outputs. Verification is the process of determining that a model implementation and its associated data accurately represent the developer's conceptual design. Validation is generally understood as an independently administered process where multiple parties that have no vested interest in the outcomes participate in developing (1) an appropriate set of standard protocols for a simulation and (2) protocol reviews across several occasions and settings. Generally, before formal validation is applied, the goals of the simulation's performance are thoroughly developed and specified. The validation process establishes the credibility of a simulation by evaluating its capability and accuracy relative to its intended use. Successfully completing validation enhances the credibility of the simulation by offering assurances that it can be relied on for reproducible results appropriate for its objectives. Additionally, data validation is to be put in the context of its suitability for use in models. For this reason, the data cannot be validated independently of the models for which they are intended. The results of the verification and validation phase are used to support the accreditation decision, which is the user's official certification that a model, simulation, or federation of models and simulations and the associated data are acceptable for use for a specific purpose. Documentation of VV&A Process Is Lacking: Officials in the Office of Program Analysis and Evaluation stated that they have performed an equivalent VV&A process for the legacy models used in the MCS, but there is little documentation available to describe the equivalent process that was used. As a result, we are not able to assess the adequacy of DOD's self-described equivalent process. Office of Program Analysis and Evaluation and Joint Data Support officials told us that most of the documentation does not yet exist and will not be completed until after the MCS is completed. Moreover, at the time of our review, DOD had not planned to describe the equivalent VV&A process in its MCS report. The absence of VV&A documentation for the models and data used to conduct the MCS and the lack of disclosure in the published MCS report could limit the study's credibility. DOD guidance, issued by the Under Secretary of Defense for Acquisition, Technology and Logistics, requires that DOD models and data go through a VV&A process. The Office of Program Analysis and Evaluation acknowledged that it did not comply with the guidance because it believes such an approach is not warranted for legacy models that have been used for many years. Moreover, these officials believe that such long-term use constitutes a VV&A process equivalent to that required in the DOD guidance. However, the DOD guidance does not identify the actual use of a model as constituting an equivalent VV&A process. DOD is using the following nine mobility models to conduct the MCS: * Aerial Port of Debarkation: * Air Mobility Operations Simulation: * Combined Mating and Ranging Planning System: * CONUS (Continental U.S.) Enhanced Logistics Intra-theater Support Tool: * Model for Inter-theater Deployment by Air and Sea: * Integrated Computerized Development System: * Joint Throughput Modeling Tool: * TRANSPORT: - Airlift/Sealift Throughput Tool: - Airlift/Sealift Rapid Analysis Tool: - Airlift Simulation Tool & Seaport Simulation Tool, and: * Analysis of Mobility Platform Federation. According to DOD officials, eight of the nine models did not go through the VV&A process specified in the DOD guidance.[Footnote 7] Office of Program Analysis and Evaluation, U.S. Transportation Command, Air Mobility Command, and Surface Deployment and Distribution Command officials told us that (1) most of the models have been used within DOD for many years and have proved reliable, and (2) many subject matter experts work with the models and the output daily to assure ongoing error detection and swift corrections when needed. These officials maintain that actual use of the models for a long period of time constitutes an equivalent VV&A process. Nonetheless, DOD officials were conducting VV&A on one of the legacy models being used in the MCS while MCS officials were simultaneously questioning the relevancy of DOD's guidance for the legacy models. For example, Surface Deployment and Distribution Command officials told us that the CONUS Enhanced Logistics Intra-theater Support Tool model was undergoing VV&A at the time of our review. It is unclear why DOD is conducting VV&A on this model given Office of Program Analysis and Evaluation and other officials' belief that it is unnecessary. Furthermore, the extent to which DOD's guidance may be irrelevant as asserted by these officials is unknown, because the department has not evaluated the current VV&A guidance to determine its relevancy for use with legacy models. Conclusions: Models and simulations approximate the real world. The approximations must be justified to assure modeling and simulation users that their predictions are credible within the bounds of specific situations, environments, and circumstances. When modeling and simulation are credible, decision makers have greater assurance that they are well informed and thus can make well-founded decisions. VV&A reduces the risk inherent in the use of models and simulations by improving the credibility of modeling and simulation results. VV&A also enhances credibility by applying a process of incremental review, analysis, evaluation, and testing by an independent agent. In light of the fact that DOD did not follow its guidance, the absence of model and baseline data VV&A documentation for the models and data used to conduct the MCS, and the planned lack of disclosure in the soon to be published MCS report, could limit the study's credibility. Moreover, MCS officials maintain that DOD guidance regarding VV&A is not relevant to legacy models and data. However, until the department evaluates the guidance to determine its relevancy for use with legacy models or incorporates guidance showing how actual model usage is to be applied as an equivalent VV&A process, the validity of DOD's assertion is uncertain. When conducted as intended, VV&A provides greater assurance that the MCS outputs are accurate. Ultimately, if the MCS inaccurately identifies mobility requirements, DOD officials may be less well informed and may therefore inadvertently obtain insufficient mobility assets or more than needed and thus waste resources. Recommendations for Executive Action: We recommend that you direct the Director, Office of Program Analysis and Evaluation, to take the following three actions: * develop documentation that describes the equivalent VV&A process used to verify and validate the mobility models and baseline data used to conduct the MCS prior to publishing any portion of the study results, * disclose in the published MCS report the equivalent VV&A process used on the models and baseline data, and: * work with the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics to evaluate the current DOD VV&A guidance to determine its relevance for use with legacy models and to change the guidance if appropriate. Agency Comments and Our Evaluation: In commenting on a draft of this report, the DOD concurred with two of our recommendations and partially concurred with the third. In its comments, DOD concurred with our first and second recommendations that the Office of Program Analysis and Evaluation develop documentation that describes the equivalent VV&A process used to verify and validate the mobility models and baseline data used to conduct the MCS prior to publishing any portion of the study results, and disclose in the published MCS report the equivalent VV&A process used on the models and baseline data. In its comments, DOD stated that the mobility modeling community has amassed substantial expertise during the past 25 years and that a "vigorous, collaborative VV&A process that is fully consistent with and in many respects exceeds the intent of DOD VV&A guidance has been put into place." DOD noted the MCS report will provide information on the VV&A process, and will exceed the level of documentation provided in past reports on DOD mobility studies. DOD partially concurred with our recommendation to evaluate the current DOD VV&A guidance to determine its relevance for use with legacy models and change the guidance if deemed appropriate. In its response, DOD essentially agreed with our recommendation, but pointed out that the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics is responsible for VV&A guidance. Therefore, we refined our recommendation to recommend that the Director of the Office of Program Analysis and Evaluation work with the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics to evaluate the current DOD VV&A guidance to determine its relevance for use with legacy models and to change the guidance if appropriate. Additionally, DOD expressed concern that it was premature to reach a conclusion as to the adequacy of the department's VV&A process because some of the documentation requested cannot be provided until the final report is written and that our draft report should be amended. We agree that we cannot assess the adequacy until the documentation is available and have so stated in our report. We also stated in our report that at the time of our review, DOD had not planned to describe the VV&A process in its MCS report nor had it planned to perform VV&A because the legacy models being used were reliable. As we noted in our report, at least one of the models was already undergoing a VV&A. Moreover, DOD questioned our conclusion that if an adequate VV&A process could not be documented, then the credibility of the MCS could be limited. Specifically, DOD stated that convincing empirical evidence indicates that the MCS report's credibility will not be limited by the VV&A documentation associated with the legacy models, because DOD has used the models for two decades with no credibility limitations noted. As we noted in our report, DOD guidance indicates that a well- documented VV&A process for the models used to conduct the study will add to the MCS report's credibility. Also as we noted in our report, DOD guidance states that the VV&A process provides enhanced user confidence, improved performance and reliability for the subject model and simulation results, reduced risk of inaccurate model outputs, and offers assurances that a particular model or simulation can be relied on for reproducible results appropriate for its objectives. While we support the use of empirical evidence, such evidence is normally verifiable. Finally, DOD expressed concern that we linked the VV&A of mobility models used to conduct the MCS with the prospect of inaccurate identification of mobility requirements. We disagree. DOD guidance states that VV&A reduces the risk inherent in the use of models and simulations by improving the credibility of their results and provides greater assurance that the study outputs are accurate. If the models supporting the MCS do not effectively simulate the real world and DOD uses the results to complete the MCS and subsequently base acquisition decisions, then we maintain our caution--DOD could inadvertently obtain insufficient mobility assets or could acquire more than needed and thus waste resources. Enclosure I contains the full text of DOD's comments. We are sending copies of this report to the Chairman and Ranking Members of the Senate Armed Services Committee as well as to the Chairmen and Ranking Members of the Senate Appropriations Committee, Subcommittee on Defense, the House Armed Services Committee, and the House Appropriations Committee, Subcommittee on Defense. This letter is also available at no charge on the GAO's Web site at http:www.gao.gov. If you or your staff have any questions on the matters discussed in this letter, please contact me at (202) 512-8365 or solisw@gao.gov. Key contributors to this report are listed in enclosure II. Sincerely yours, Signed by: William M. Solis, Director: Defense Capabilities and Management: Enclosures: Enclosure I: Comments from the Department of Defense: OFFICE OF THE SECRETARY OF DEFENSE: PROGRAM ANALYSIS AND EVALUATION: 1800 DEFENSE PENTAGON: WASHINGTON, DC 20301-1800: August 23, 2005: Mr. William M. Solis: Director, Defense Capabilities and Management: U.S. Government Accountability Office: 441 G Street, N.W.: Washington, DC 20548: Dear Mr. Solis: This is the Department of Defense's (DoD) response to the GAO Draft Report, GAO-05-659R, "DEFENSE TRANSPORTATION: Opportunities Exist to Enhance the Credibility of the Current and Future Mobility Capabilities Studies," dated August 10, 2005. The DoD concurs with two of the GAO's recommendations and partially concurs with the remaining recommendation. Specific comments on each recommendation are attached. Additional concerns with the report are also forwarded for your consideration. Kathleen Conley, Director of PA&E's Projection Forces Division, is my point of contact for this issue. Please contact her at (703) 697-0802, or at Kathleen.Conley@osd.mil, if you have any questions. We appreciate the opportunity to comment on the draft report. Signed by: Stanley R. Szemborski: VADM, USN: Principal Deputy Director: Attachment As stated: GAO DRAFT REPORT - DATED AUGUST 10, 2005 GAO CODE 350709/GAO-05-659R: "DEFENSE TRANSPORTATION: Opportunities Exist to Enhance the Credibility of the Current and Future Mobility Capabilities Studies" DEPARTMENT OF DEFENSE COMMENTS ON THE RECOMMENDATIONS: RECOMMENDATION 1: The GAO recommended that the Director, Office of Program Analysis and Evaluation (PA&E): * Develop documentation that describes the equivalent verification, validation, and accreditation (VV&A) process used to verify and validate the mobility models and baseline data used to conduct the mobility capabilities study (MCS) prior to publishing any portion of the study results. (p. 6): DoD RESPONSE: Concur. As DoD representatives have stated in discussions with GAO officials, the MCS report will provide information on the VV&A process, and will exceed the level of documentation provided in past reports on DoD mobility studies. The transparent, collaborative approach taken by the Department to develop realistic scenarios and data and to accurately model complex mobility processes will be documented for the benefit of readers who may not be familiar with the Department's analytic best practices. Specifically, the scenarios, baseline data, and mobility modeling associated with the MCS reflect the Department's considerable experience in conducting mobility studies over the past twenty-five years. The mobility modeling community has amassed substantial expertise during that period, and a vigorous, collaborative V V&A process has been put into place. That process is fully consistent with-and in many respects exceeds-the intent of DoD Instruction 5000.61. As a result of their continuous involvement in the study process, senior DoD leaders have developed a high degree of confidence in the MCS methodology. Because the process has provided an effective forum for resolving questions about the underlying data, models, and assumptions, the insights gleaned from the MCS can be used to frame discussions about capabilities needed to support the defense strategy. RECOMMENDATION 2: The GAO recommended that the Director, Office of Program Analysis and Evaluation: * Disclose in the published MCS report the equivalent VV&A process used on the models and baseline data. (p. 6): DoD RESPONSE: Concur. As stated above and in discussions with GAO officials, the MCS report will provide information on the V V&A process, and will exceed the level of documentation provided in past reports on DoD mobility studies. The transparent, collaborative approach taken by the Department to develop realistic scenarios and data and to accurately model complex mobility processes will be documented for the benefit of readers who may not be familiar with the Department's analytic best practices. RECOMMENDATION 3: The GAO recommended that the Director, Office of Program Analysis and Evaluation: * Evaluate the current DoD V V&A guidance to determine its relevance for use with legacy models and change the guidance if deemed appropriate. (p. 6): DoD RESPONSE: Partially concur. The Department will continuously assess the applicability of its VV&A guidance to the legacy models used in the MCS and other studies. The office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (AT&L) is responsible for managing VV&A guidance. The Department is considering revisions to this guidance. ADDITIONAL ISSUES: The GAO draft report notes that GAO was unable to assess the adequacy of DoD's V V &A process. Further, the report states that DoD "does not plan to disclose how it conducted its equivalent W&A process in its MCS report." Consequently, GAO expressed concern that a failure to fully document the study's V V&A process could limit the credibility of the MCS. (pp. 2, 4, 6): DoD COMMENT: The VV&A process will be fully described in the final report-a point that was emphasized by DoD officials in discussions with GAO. Furthermore, DoD officials have notified GAO that some of the documentation requested cannot be provided until the final report is written. Therefore, it is premature to reach a conclusion as to the adequacy of the V V&A process. For this reason, either the assessment of adequacy should be omitted from the section of the GAO draft report providing preliminary observations, or the discussion should be amended to inform the reader that a full assessment is not currently possible, and that one will be completed upon receipt of the final report. Pages 2, 4, and 6 of the GAO draft report raise the possibility that failure to fully document the VV&A process used in the MCS could limit the study's credibility. The documentation of W&A processes in the MCS report-well beyond that presented in previous studies-should benefit those readers who may not be familiar with the study's methodology. Moreover, convincing empirical evidence indicates that the report's credibility will not be limited by the VV&A documentation associated with legacy models: two decades of mobility studies have been conducted using many of the legacy models employed by MCS participants with no such effect noted. On page 6, the GAO draft report attempts to link the current VV&A processes for legacy mobility models with the prospect of inaccurate identification of mobility requirements. Again, the evidence does not support this contention. The mobility community's long-standing VV&A processes have not resulted in inaccurate assessments of mobility requirements, or of the investments needed to achieve capability objectives. [End of section] GAO Contact and Staff Acknowledgments: GAO Contact: William M. Solis, (202) 512-8365: Acknowledgments: Key contributors to this report include Brian J. Lepore, Claudia Dickey, Ron La Due Lake, Oscar Mardis, Deborah Owolabi, Kenneth Patton, and R.K. Wild. (350708): FOOTNOTES [1] S. Rep. No. 108-260, at 126 (2004). [2] The MCS study team includes officials from the Office of the Secretary of Defense, Program Analysis and Evaluation and the office of the Director of Logistics, Joint Chiefs of Staff, identified as co- leads and study management, as well as study participants to include representatives from the military services, combatant commands, and contractors employed by any of the aforementioned DOD organizations to provide input to or services in support of the MCS. [3] DOD, DOD Modeling and Simulation (M&S) Verification, Validation, Accreditation (VVA), Instruction 5000.61 (Washington, D.C.: May 2003). [4] Past mobility requirements studies conducted since the early 1990s include: Mobility Requirements Study (1992); Mobility Requirements Study-Bottom Up Review Update (1995); and Mobility Requirements Study for Fiscal Year 2005 (2001). [5] The overall study objectives are to identify and/or quantify (1) how variations in mobility capabilities support the defense strategy from point of origin to point of use and return in the 2012 time frame; (2) mobility capability gaps, overlaps, or excesses and associated risk assessments with regard to conducting operations; (3) mobility capability alternatives that mitigate operational logistic impacts caused by challenges; (4) combinations of mobility, engineering, and infrastructure capabilities required to support deployments and distributions required by the defense strategy; (5) new metrics for assessing mobility capabilities; (6) potential impact of evolving service force transformation and research and development efforts that integrate mobility concepts for the 2024 time frame; and (7) transformational accelerants to enable the defense transportation system to operate in a net-centric environment. [6] DOD, DOD modeling and Simulation Verification, Validation and Accreditation, DOD Instruction 5000.61 (Washington, D.C.: May 2003); and DOD Modeling and Simulation Office, DOD Verification, Validation and Accreditation Recommended Practices Guide (Washington, D.C.: August 2004). [7] DOD conducted VV&A on the Combined Mating and Ranging Planning System model about 15 years ago in compliance with departmental guidance but could not locate documentation to demonstrate how VV&A was done. As a result, we could not evaluate the adequacy of the VV&A process.