This is the accessible text file for GAO report number GAO-08-52 
entitled 'Business Systems Modernization: Air Force Needs to Fully 
Define Policies and Procedures for Institutionally Managing 
Investments' which was released on October 31, 2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 

GAO: 

October 2007: 

Business Systems Modernization: 

Air Force Needs to Fully Define Policies and Procedures for 
Institutionally Managing Investments: 

Business Systems Modernization: 

GAO-08-52: 

GAO Highlights: 

Highlights of GAO-08-52, a report to congressional committees. 

Why GAO Did This Study: 

In 1995, GAO first designated the Department of Defense’s (DOD) 
business systems modernization program as “high-risk” and continues to 
do so today. In 2004, Congress passed legislation reflecting prior GAO 
recommendations that DOD adopt a corporate approach to information 
technology (IT) business systems investment management including tiered 
accountability for business systems at the department and component 
levels. To support GAO’s legislative mandate to review DOD’s efforts, 
GAO assessed whether the investment management approach of one of DOD’s 
components—the Department of the Air Force (Air Force)—is consistent 
with leading investment management best practices. In doing so, GAO 
applied its IT Investment Management (ITIM) framework and associated 
methodology, focusing on the stages related to the investment 
management provisions of the Clinger-Cohen Act of 1996. 

What GAO Found: 

The Air Force has established the basic management structures needed to 
effectively manage its IT projects as investments, but has not fully 
implemented many of the related policies and procedures outlined in 
GAO’s ITIM framework (see table). Air Force has fully implemented three 
of the nine key practices that call for project-level management 
structures, policies, and procedures, and has not implemented any of 
the five practices that call for portfolio-level policies and 
procedures. Regarding project-level practices, it has established an IT 
investment board that is responsible for defining and implementing the 
department’s business systems investment governance process, has 
developed procedures for identifying and collecting information about 
its business systems to support investment selection and control, and 
has assigned responsibility for ensuring that the information collected 
during project identification meets the needs of the investment 
management process. However, Air Force has not fully documented 
business systems investment policies and procedures for directing 
investment board operations, selecting new investments, reselecting 
ongoing investments, or integrating the investment funding and 
investment selection processes. In addition, it has not implemented any 
of the policies and procedures for developing and maintaining a 
complete business system investment portfolio. 

Air Force officials stated that they are aware of the absence of 
documented policies and procedures in certain areas of project-level 
and portfolio-level management and that they are currently working on 
guidance to address these areas. For example, officials stated that 
they had begun drafting portfolio-level policies and procedures. 
According to Air Force officials, the policies and procedures are 
expected to be completed and approved by December 2007. Until Air Force 
fully defines policies and procedures for both individual projects and 
portfolios of projects, it risks not being able to select and control 
these business system investments in a way that is consistent and 
complete, which in turn increases the chances that these investments 
will not meet mission needs in the most effective manner. 

Table: Status of Air Force's Project-and Portfolio-Level Management 
Capabilities: 

Stage 2: Building the investment foundation: Instituting the investment 
board; 
Key practices executed: 1/2; 
Stage 3: Developing a complete investment portfolio: Defining the 
portfolio criteria; 
Key practices executed: 0/2. 

Stage 2: Building the investment foundation: Meeting business needs; 
Key practices executed: 0/1; 
Stage 3: Developing a complete investment portfolio: Creating the 
portfolio; 
Key practices executed: 0/1. 

Stage 2: Building the investment foundation: Selecting an investment; 
Key practices executed: 0/3; 
Stage 3: Developing a complete investment portfolio: Evaluating the 
portfolio; 
Key practices executed: 0/1. 

Stage 2: Building the investment foundation: Providing investment 
oversight; 
Key practices executed: 0/1; 
Stage 3: Developing a complete investment portfolio: Conducting post 
implementation reviews; 
Key practices executed: 0/1. 

Stage 2: Building the investment foundation: Capturing investment 
information; 
Key practices executed: 2/2; 
Stage 3: Developing a complete investment portfolio: [Empty]; 
Key practices executed: [Empty]. 

Overall; 
Key practices executed: 3/9; 
Stage 3: Developing a complete investment portfolio: [Empty]; 
Key practices executed: 0/5. 

Source: GAO. 

[End of table] 

What GAO Recommends: 

GAO recommends that the Department of the Air Force fully define the 
project and portfolio management policies and procedures discussed in 
GAO’s ITIM framework. In comments on a draft of this report, DOD stated 
that the Air Force has begun to establish a project-level management 
process that will be instituted in formal policies and is applying 
DOD’s portfolio management process in its decision making. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.GAO-08-52]. For more information, contact 
Valerie Melvin at (202) 512-6304 or melvinv@gao.gov. 

[End of section] 

Contents: 

Letter1: 

Results in Brief: 

Background: 

Air Force Has Established the Structures Needed to Effectively Manage 
Business System Investments but Has Not Fully Defined Many of the 
Related Policies and Procedures: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objective, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: DOD and Air Force Business System Investment Tiers: 

Table 2: Air Force Investment Management Governance Entities and 
Responsibilities: 

Table 3: Stage 2 Critical Processes--Building the Investment 
Foundation: 

Table 4: Summary of Policies and Procedures for Stage 2 Critical 
Processes--Building the Investment Foundation: 

Table 5: Stage 3 Critical Processes--Developing a Complete Investment 
Portfolio: 

Table 6: Summary of Policies and Procedures for Stage 3 Critical 
Processes--Developing a Complete Investment Portfolio: 

Figures: 

Figure 1: Simplified DOD Organizational Structure: 

Figure 2: Air Force Chief Information Officer Organizational Structure: 

Figure 3: The Five ITIM Stages of Maturity with Critical Processes: 

Figure 4: Working Relationships among DOD Business Investment 
Management System Governance Entities: 

Figure 5: Air Force Precertification Review and Approval Process: 

Abbreviations: 

CPM: certification process manager: 

CIO: chief information officer: 

DAS: Defense Acquisition System: 

DOD: Department of Defense: 

IT: information technology: 

ITIM: Information Technology Investment Management: 

JCIDS: Joint Capabilities Integration and Development System: 

MAJCOM: Major Command: 

OMB: Office of Management and Budget: 

PPBE: Planning, Programming, Budgeting, and Execution: 

United States Government Accountability Office: 

Washington, DC 20548: 

October 31, 2007: 

Congressional Committees: 

For decades, the Department of Defense (DOD) has been challenged in 
modernizing its timeworn business systems.[Footnote 1] In 1995, we 
designated DOD's business systems modernization program as high risk, 
and we continue to designate it as such today.[Footnote 2] Our research 
on public and private sector organizations shows that an essential 
ingredient to a successful systems modernization program is having an 
effective institutional approach to managing information technology 
(IT) investments. 

In May 2001, we recommended that the department establish a corporate 
approach to investment control and decision making.[Footnote 3] Between 
2001 and 2005, we reported that the department's business systems 
modernization program was still not being effectively managed,[Footnote 
4] and we made additional investment-related recommendations. Congress 
subsequently included provisions in the Ronald W. Reagan National 
Defense Authorization Act for Fiscal Year 2005[Footnote 5] that 
reflected our recommendations, including those for establishing and 
implementing effective business system investment management structures 
and processes. 

Between 2005 and 2007,[Footnote 6] we reported that DOD had made 
important progress in establishing and implementing these structures 
and processes; however, much remained to be accomplished. Most 
recently,[Footnote 7] we reported that, according to DOD officials, 
investment management practices are performed at the component level, 
and that departmental policies and procedures established for 
overseeing components' execution of these practices are sufficient. 
However, DOD had not fully defined many of the related policies and 
procedures outlined in GAO's IT Investment Management framework. 

The Fiscal Year 2005 National Defense Authorization Act directs DOD to, 
among other things, establish and implement effective IT business 
system investment management structures and processes. As agreed with 
your offices and to support the legislative mandate that GAO assess 
DOD's actions to comply with this requirement, the objective of our 
review was to determine whether the investment management approach of 
the Department of the Air Force (Air Force) is consistent with leading 
investment management best practices. To accomplish our objective, we 
analyzed documents and interviewed department officials to determine 
whether Air Force has developed the structures, policies, and 
procedures associated with executing those key practices in our IT 
Investment Management (ITIM) framework[Footnote 8] that assist 
departments and agencies in complying with the investment management 
provisions of the Clinger-Cohen Act of 1996.[Footnote 9] 

We performed our work at Air Force offices in Arlington, Virginia, from 
February 2007 through September 2007 in accordance with generally 
accepted government auditing standards. Details on our objective, 
scope, and methodology are contained in appendix I. 

Results in Brief: 

Air Force has established the basic management structures needed to 
effectively manage its IT projects as investments, but it has not fully 
implemented many of the related policies and procedures that our ITIM 
framework outlines. Air Force has fully implemented three of the nine 
key practices that call for project-level management structures and 
project-level policies and procedures. Specifically, it has (1) 
established an investment review board that is responsible for business 
system investment governance, (2) developed procedures for identifying 
and collecting information about its business systems to support 
investment selection and control, and (3) assigned responsibility for 
ensuring that the information collected during project identification 
meets the needs of the investment management process. However, Air 
Force has not fully developed business system investment policies and 
procedures related to the remaining six key project-level management 
practices. For example, policies and procedures do not (1) address how 
investments for systems in operations and maintenance are to be 
governed; (2) define how systems in operations and maintenance will 
support ongoing and future business needs; (3) specify how the full 
range of cost, schedule, and performance data accessible to Air Force 
is to be used in making selection (i.e., precertification) decisions; 
(4) specify how reselection decisions (i.e., annual reviews) consider 
investments that are in operations and maintenance; (5) describe how 
funding decisions are integrated with the process of selecting an 
investment; and (6) provide sufficient oversight and visibility into 
investment management activities, including specifying how corrective 
actions should be taken. 

Further, Air Force has not implemented any of the five practices that 
call for policies and procedures to develop and maintain a complete 
business system portfolio. Specifically, it does not have documented 
policies and procedures for (1) defining the portfolio criteria, (2) 
creating the portfolio, (3) evaluating the portfolio, and (4) 
conducting post-implementation reviews of business systems. In 
addition, the Air Force has not assigned responsibility for managing 
the portfolio criteria. According to ITIM, adequately documenting both 
the policies and associated procedures that govern how an organization 
manages its IT investment portfolios is important because doing so 
provides the basis for having rigor, discipline, and repeatability in 
how investments are selected and controlled across the entire 
organization. 

Air Force officials stated that they are aware of the absence of 
documented policies and procedures in certain areas of project-level 
and portfolio-level management; officials also stated that they are 
currently working on new policies and procedures to address these areas 
that they expect to issue by December 2007. Until Air Force defines 
policies and procedures for both individual projects and portfolios of 
projects--and assigns responsibility for managing its business system 
portfolios--it risks selecting and controlling these business system 
investments in a way that is not consistent and complete, which in turn 
reduces the chances that these investments will meet mission needs in 
the most effective manner. 

To strengthen Air Force's business system management capability, we are 
recommending that the agency fully define the policies and procedures 
associated with project-level and portfolio-level investment management 
and assign responsibility for managing its business system portfolios, 
as discussed in our guidance for IT investment management.[Footnote 10] 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Business Transformation) and reprinted in 
appendix II, DOD partially concurred with the report's recommendations. 
Relative to our recommendations concerning project-level management 
policies and procedures, DOD stated that Air Force has begun to 
establish a project-level management process that, while not codified 
into formal policy, is documented in its investment review guide. 
Further, it stated that Air Force is committed to formalizing its 
policies as the processes expand and mature. Our report recognizes that 
Air Force has drafted a business system investment management process. 
However, we found that it lacks critical elements needed to effectively 
carry out essential investment management activities. Until it fully 
addresses all critical elements needed for investment management, Air 
Force will be at risk of not being able to carry out investment 
management activities in a consistent and disciplined manner. Relative 
to our recommendations concerning portfolio-level management policies 
and procedures, DOD stated that Air Force is applying DOD's IT 
portfolio management process in its decision making. However, as our 
report notes, Air Force has not implemented a process for managing its 
business system portfolios. Until it implements such a process, Air 
Force increases the risk of not selecting the mix of investments that 
best supports its mission needs. 

Background: 

DOD is a massive and complex organization. To illustrate, the 
department reported that its fiscal year 2006 operations involved 
approximately $1.4 trillion in assets and $2.0 trillion in liabilities, 
more than 2.9 million military and civilian personnel, and $581 billion 
in net cost of operations. Organizationally, the department includes 
the Office of the Secretary of Defense, the Chairman of the Joint 
Chiefs of Staff, the military departments, numerous defense agencies 
and field activities, and various unified combatant commands that are 
responsible for either specific geographic regions or specific 
functions. Figure 1 provides a simplified depiction of DOD's 
organizational structure. 

Figure 1: Simplified DOD Organizational Structure: 

This figure is a chart showing the simplified DOD organizational 
structure. 

[See PDF for image] 

Source: GAO, based on DOD documentation. 

[A] The Chairman of the Joint Chiefs of Staff serves as the spokesman 
for the commanders of the combatant commands, particularly for the 
administrative requirements of their commands. 

[End of figure] 

In support of its military operations, DOD performs an assortment of 
interrelated and interdependent business functions, including logistics 
management, procurement, health care management, and financial 
management. As we have previously reported,[Footnote 11] the systems 
environment that supports these business functions is overly complex 
and error prone, and is characterized by (1) little standardization 
across the department, (2) multiple systems performing the same tasks, 
(3) the same data stored in multiple systems, and (4) the need for data 
to be entered manually into multiple systems. 

Air Force's Mission, Organizational Structure, and Use of IT: 

Air Force is a major component of DOD. Its mission is to deliver 
options for the defense of the United States and its global interests 
in air, space, and cyberspace. Air Force relies extensively on IT to 
fulfill these competencies effectively and to meet its organizational 
mission. It has 909 business systems; of these systems, 832 (91 
percent) are in operations and maintenance. In fiscal year 2006, Air 
Force was allocated approximately $651 million[Footnote 12] for its 
business systems, of which about $406 million (62 percent) was 
allocated to systems in operations and maintenance and $245 million (38 
percent) was allocated to systems in development and/or modernization. 

Air Force has created the Office of Warfighting Integration and Chief 
Information Office to provide the IT and supporting infrastructure to 
fulfill its mission. Among the goals of this organization are to: 

* deliver the ability to direct forces while anticipating situations, 
capabilities, and limitations; 

* develop adaptive, trained airmen; 

* shape enterprise investments; 

* provide policy, standards, oversight, and training to enable airmen 
to share and exploit accurate information any place and, anytime; and: 

* transform the communications and information career field to lead Air 
Force in leveraging information for its competitive advantage. 

The Office of Warfighting Integration and Chief Information Office 
consists of several organizations, as depicted in figure 2. 

Figure 2: Air Force Chief Information Officer Organizational Structure: 

This figure is a chart showing the organizational structure of an Air 
Force Chief Information Officer. 

[See PDF for image] 

Source: Air Force. 

[End of figure] 

IT Investment Management Is Critical to Achieving Successful Systems 
Modernization: 

Successful public and private organizations use a corporate approach to 
IT investment management. Recognizing this, Congress enacted the 
Clinger-Cohen Act of 1996,[Footnote 13] which requires the Office of 
Management and Budget (OMB) to establish processes to analyze, track, 
and evaluate the risks and results of major capital investments in IT 
systems made by executive agencies.[Footnote 14] In response to the 
Clinger-Cohen Act and other statutes, OMB has developed policy and 
issued guidance for the planning, budgeting, acquisition, and 
management of federal capital assets.[Footnote 15] We have also issued 
guidance in this area[Footnote 16] that defines institutional 
structures, such as investment review boards; processes for developing 
information on investments (such as costs and benefits); and practices 
to inform management decisions (such as whether a given investment is 
aligned with an enterprise architecture). 

IT Investment Management: A Brief Description: 

IT investment management is a process for linking IT investment 
decisions to an organization's strategic objectives and business plans. 
Consistent with this, the federal approach to IT investment management 
focuses on selecting, controlling, and evaluating investments in a 
manner that minimize risks while maximizing the return on 
investment.[Footnote 17] 

* During the selection phase, the organization (1) identifies and 
analyzes each project's risks and returns before committing significant 
funds to any project and (2) selects those IT projects that will best 
support its mission needs. 

* During the control phase, the organization ensures that projects, as 
they develop and investment expenditures continue, meet mission needs 
at the expected levels of cost and risk. If the project is not meeting 
expectations or if problems arise, steps are quickly taken to address 
the deficiencies. 

* During the evaluation phase, expected results are compared with 
actual results after a project has been fully implemented. This 
comparison is done to (1) assess the project's impact on mission 
performance, (2) identify any changes or modifications to the project 
that may be needed, and (3) revise the investment management process 
based on lessons learned. 

Overview of GAO's ITIM Maturity Framework: 

Our ITIM framework consists of five progressive stages of maturity for 
any given agency relative to selecting, controlling, and evaluating its 
investment management capabilities.[Footnote 18] (See fig. 3 for the 
five ITIM stages of maturity.) This framework is grounded in our 
research of IT investment management practices of leading private and 
public sector organizations. The framework can be used to assess the 
maturity of an agency's investment management processes and as a tool 
for organizational improvement. The overriding purpose of the framework 
is to encourage investment selection and control and to evaluate 
processes that promote business value and mission performance, reduce 
risk, and increase accountability and transparency. We have used the 
framework in several of our evaluations,[Footnote 19] and a number of 
agencies have adopted it. 

ITIM's five maturity stages represent the steps toward achieving stable 
and mature processes for managing IT investments. Each stage builds on 
the lower stages; the successful attainment of each stage leads to 
improvement in the organization's ability to manage its investments. 
With the exception of the first stage, each maturity stage is composed 
of "critical processes" that must be implemented and institutionalized 
in order for the organization to achieve that stage. These critical 
processes are further broken down into key practices that describe the 
types of activities that an organization should be performing to 
successfully implement each critical process. It is not unusual for an 
organization to be performing key practices from more than one maturity 
stage at the same time. However, our research has shown that agency 
efforts to improve investment management capabilities should focus on 
implementing all lower stage practices before addressing the higher 
stage practices. 

Figure 3: The Five ITIM Stages of Maturity with Critical Processes: 

This figure is a chart showing the five ITIM stages of maturity with 
critical processes. 

[See PDF for image] 

Source: GAO. 

[End of figure] 

In the ITIM framework, Stage 2 critical processes lay the foundation 
for sound IT investment management by helping the agency to attain 
successful, predictable, and repeatable investment management processes 
at the project level. Specifically, Stage 2 encompasses building a 
sound investment management foundation by establishing basic 
capabilities for selecting new IT projects. This stage also involves 
developing the capability to control projects so that they finish 
predictably within established cost and schedule expectations and 
developing the capability to identify potential exposures to risk and 
put in place strategies to mitigate that risk. Further, it involves 
evaluating completed projects to ensure they meet business needs and 
collecting lessons learned to improve the IT investment management 
process. The basic management processes established in Stage 2 lay the 
foundation for more mature management capabilities in Stage 3, which 
represents a major step forward in maturity, in which the agency moves 
from project-centric processes to a portfolio approach, evaluating 
potential investments by how well they support the agency's missions, 
strategies, and goals. 

Stage 3 requires that an organization continually assess both proposed 
and ongoing projects as parts of a complete investment portfolio--an 
integrated and competing set of investment options. It focuses on 
establishing a consistent, well-defined perspective on the IT 
investment portfolio and maintaining mature, integrated selection (and 
reselection), control, and post-implementation evaluation processes. 
This portfolio perspective allows decision makers to consider the 
interaction among investments and the contributions to organizational 
mission goals and strategies that could be made by alternative 
portfolio selections, rather than focusing exclusively on the balance 
between the costs and benefits of individual investments. Organizations 
that have implemented Stages 2 and 3 practices have capabilities in 
place that assist in establishing selection; control; and evaluation 
structures, policies, procedures, and practices that are required by 
the investment management provisions of the Clinger-Cohen Act.[Footnote 
20] 

Stages 4 and 5 require the use of evaluation techniques to continuously 
improve both the investment portfolio and the investment processes in 
order to better achieve strategic outcomes. At Stage 4, an organization 
has the capacity to conduct IT succession activities and, therefore, 
can plan and implement the deselection of obsolete, high-risk, or low- 
value IT investments. An organization with Stage 5 maturity conducts 
proactive monitoring for breakthrough information technologies that 
will enable it to change and improve its business performance. 

DOD and Air Force Approach for Identifying, Funding, and Acquiring 
System Investments: 

DOD's major system investments (i.e., weapons and business systems) are 
governed by three management systems that focus on defining needs, 
budgeting for, and acquiring investments to support the mission--the 
Joint Capabilities Integration and Development System (JCIDS); the 
Planning, Programming, Budgeting, and Execution (PPBE) system; and the 
Defense Acquisition System (DAS). In addition, DOD's business systems 
are subject to a fourth management system, which, for purposes of this 
report, we refer to as the Business Investment Management System. For 
each of these systems, DOD relies on its component agencies to execute 
the underlying policies and procedures. According to DOD, the four 
management systems, collectively, are the means by which the 
department--and its components--selects, controls, and evaluates its 
business systems investments. 

Joint Capabilities Integration and Development System: 

JCIDS is a needs-driven, capabilities-based approach to identify 
mission needs and meet future joint forces challenges. It is intended 
to identify future capabilities for DOD; address capability gaps and 
mission needs recognized by the Joint Chiefs of Staff or derived from 
strategic guidance, such as the National Security Strategy 
Report[Footnote 21] or Quadrennial Defense Review;[Footnote 22] and 
identify alternative solutions by considering a range of doctrine, 
organization, training, materiel, leadership and education, personnel, 
and facilities solutions. According to DOD, the Joint Chiefs of Staff-
-through the Joint Requirements Oversight Council--has primary 
responsibility for defining and implementing JCIDS. All JCIDS documents 
are submitted to the Joint Chiefs of Staff, which determines whether 
the proposed system has joint implications or is component-unique. If 
it is designated as joint interest, then the Joint Requirements 
Oversight Council is responsible for approving and validating the 
documents. If it is not designated as having joint interests, the 
sponsoring component is responsible for validation and approval. 

Planning, Programming, Budgeting, and Execution: 

PPBE is a calendar-driven approach that is composed of four phases that 
occur over a moving 2-year cycle. The four phases--planning, 
programming, budgeting, and executing--define how budgets for each DOD 
component and the department as a whole are created, vetted, and 
executed. As recently reported,[Footnote 23] the components start 
programming and budgeting for addressing a JCIDS-identified capability 
gap or mission need several years before actual product development 
begins and before the Office of the Secretary of Defense formally 
reviews the components' programming and budgeting proposals (i.e., 
Program Objective Memorandums). Once reviewed and approved, the 
financial details in the Program Objective Memorandums become part of 
the President's budget request to Congress. During budget execution, 
components may submit program change proposals or budget change 
proposals, or both (e.g., program cost increases or schedule delays). 
According to DOD, the Under Secretary of Defense (Policy), the Director 
for Program Analysis and Evaluation, and the Under Secretary of Defense 
(Comptroller) have primary responsibility for defining and implementing 
the PPBE system. 

Defense Acquisition System: 

DAS[Footnote 24] is a framework-based approach that is intended to 
translate mission needs and requirements into stable, affordable, and 
well-managed acquisition programs. It consists of five key program life-
cycle phases. These five phases are as follows: 

Concept Refinement: Intended to refine the initial JCIDS-validated 
system solution (concept) and create a strategy for acquiring the 
investment solution. A decision is made at the end of this phase 
(Milestone A decision) regarding whether to move to the next phase 
(Technology Development). 

Technology Development: Intended to determine the appropriate set of 
technologies to be integrated into the investment solution by 
iteratively assessing the viability of various technologies while 
simultaneously refining user requirements. Once the technology has been 
demonstrated in a relevant environment, a decision is made (Milestone B 
decision) regarding whether to move to the next phase (System 
Development and Demonstration). 

System Development and Demonstration: Intended to develop a system or a 
system increment and demonstrate through developer testing that the 
system or system increment can function in its target environment. A 
decision is made at the end of this phase (Milestone C decision) 
regarding whether to move to the next phase (Production and 
Deployment). 

Production and Deployment: Intended to achieve an operational 
capability that satisfies the mission needs, as verified through 
independent operational test and evaluation, and ensures that the 
system is implemented at all applicable locations. 

Operations and Support: Intended to operationally sustain the system in 
the most effective manner over its life cycle. 

A key principle of DAS is that investments are assigned a category, 
where programs of increasing dollar value and management interest are 
subject to more stringent oversight. For example, Major Defense 
Acquisition Programs[Footnote 25] and Major Automated Information 
Systems[Footnote 26] are large, expensive programs subject to the most 
extensive statutory and regulatory reporting requirements and unless 
delegated, are reviewed by acquisition boards at the DOD corporate 
level. Smaller and less risky acquisitions are generally reviewed at 
the component executive or lower levels. Another key principle is that 
DAS requires acquisition management under the direction of a milestone 
decision authority.[Footnote 27] The milestone decision authority-- 
with support from the program manager and advisory boards, such as the 
Defense Acquisition Board[Footnote 28] and the IT Acquisition 
Board[Footnote 29]--determines the project's baseline cost, schedule, 
and performance commitments. The Under Secretary of Defense for 
Acquisition, Technology, and Logistics has primary responsibility for 
defining and implementing DAS. 

DOD relies on its components to execute these investment management 
policies and procedures. To implement DOD's JCIDS process, the Air 
Force has designated the Joint Staff Functional Capabilities Board to 
review and approve operational capabilities. The Joint Staff Functional 
Capabilities Board seeks to establish a common understanding of how a 
capability will be used, who will use it, when it is needed, and why it 
is needed to achieve a desired effect. Each capability is assessed 
based on the effects it seeks to generate and the associated 
operational risk of not having it. In addition, the Capabilities Review 
and Risk Assessment process is being used to analyze concepts of 
operation and assess their associated capabilities. This process uses a 
phased approach to produce a prioritized list of capabilities, 
capability gaps or shortfalls, and possible capability solutions. To 
implement the PPBE process, Air Force officials stated that they use 
their Annual Planning and Programming Guidance manual. Finally, to 
implement DAS, Air Force has developed guidance that outlines a 
systematic acquisition framework that mirrors the framework defined by 
DOD and includes the same three event-based milestones and associated 
five program life-cycle phases. 

Business Investment Management System: 

The Business Investment Management System is a calendar-driven approach 
that is described in terms of governance entities, tiered 
accountability, and certification reviews and approvals. This system 
was initiated in 2005, when DOD reassigned responsibility for providing 
executive leadership for the direction, oversight, and execution of its 
business systems modernization efforts to several entities. These 
entities and their responsibilities include the following: 

* The Defense Business Systems Management Committee serves as the 
highest-ranking governance body for business systems modernization 
activities. 

* The Principal Staff Assistants serve as the certification authorities 
for business system modernizations in their respective core business 
missions. 

* The Investment Review Boards are chartered by the principal staff 
assistants and are the review and decision-making bodies for business 
system investments in their respective areas of 
responsibility.[Footnote 30] The boards are also responsible for 
recommending certification for all business system investments costing 
more than $1 million. 

* The component precertification authority is accountable for the 
component's business system investments and acts as the component's 
principal point of contact for communication with the Investment Review 
Boards. The Air Force has designated its CIO to be the precertification 
authority. 

* The Business Transformation Agency is responsible for leading and 
coordinating business transformation efforts across the department. The 
agency is organized into seven directorates, one of which is the 
Defense Business Systems Acquisition Executive--the component 
acquisition executive for DOD enterprise-level (DOD-wide) business 
systems and initiatives. This directorate is responsible for 
developing, coordinating, and integrating enterprise-level projects, 
programs, systems, and initiatives--including managing resources such 
as fiscal, personnel, and contracts for assigned systems and programs. 

Figure 4 provides a simplified illustration of the relationships among 
these entities. 

Figure 4: Working Relationships among DOD Business Investment 
Management System Governance Entities: 

This figure is a chart showing working relationships among DOD business 
investment management system governance entities. 

[See PDF for image] 

Source: GAO, based on DOD documentation. 

[End of figure] 

According to DOD, in 2005 it also adopted a tiered accountability 
approach to business transformation. Under this approach, 
responsibility and accountability for business system investment 
management is allocated among DOD (i.e., Office of the Secretary of 
Defense) and the component agencies, based on the amount of 
development/modernization funding involved and the investment's "tier." 
DOD is responsible for ensuring that all business systems with a 
development/modernization investment in excess of $1 million are 
reviewed by the Investment Review Boards for compliance with the 
business enterprise architecture, certified by the principal staff 
assistants, and approved by the Defense Business Systems Management 
Committee. Components are responsible for certifying development/ 
modernization investments with total costs of $1 million or less. All 
DOD development and modernization efforts are assigned a tier on the 
basis of the acquisition category or the size of the financial 
investment, or both. According to DOD, a system is given a tier 
designation when it passes through the certification process. Table 1 
describes the five investment tiers and identifies the associated 
reviewing and approving entities for DOD and the Air Force. 

Table 1: DOD and Air Force Business System Investment Tiers: 

Tier: Tier 1; 
Description: Major Automated Information Systems and Major Defense 
Acquisition Programs; 
Reviewing/Approving entities: Certified by Investment Review Boards and 
Defense Business Systems Management Committee; precertified by Air 
Force CIO. 

Tier: Tier 2; 
Description: Systems exceeding $10 million in total 
development/modernization costs, but not designated Major Automated 
Information Systems or Major Defense Acquisition Programs; 
Reviewing/ Approving entities: Certified by Investment Review Boards 
and Defense Business Systems Management Committee; precertified by Air 
Force CIO. 

Tier: Tier 3; 
Description: Systems exceeding $1 million and up to $10 million in 
total development/modernization costs; 
Reviewing/Approving entities: Certified by Investment Review Boards and 
Defense Business Systems Management Committee; 
precertified by Air Force CIO. 

Tier: Tier 4; 
Description: All other business systems (i.e., those systems with 
development/modernization costs of $1 million or less); 
Reviewing/Approving entities: Certified by Air Force CIO. 

Tier: Tier 5; 
Description: Systems in operations and maintenance or sustainment; 
Reviewing/Approving entities: Approved by Air Force CIO. 

Sources: DOD and Air Force. 

[End of table] 

DOD's business investment management system includes two types of 
reviews for business systems: certification and annual reviews. 
Certification reviews apply to new modernization projects with total 
costs over $1 million. These reviews focus on program alignment with 
the business enterprise architecture and must be completed before 
components obligate funds for programs. The annual reviews apply to all 
business programs and are undertaken to determine whether the system 
development effort is meeting its milestones and addressing its 
Investment Review Board certification conditions. 

* Certification reviews and approvals: Tier 1 through 3 business system 
investments in development and modernization are certified at two 
levels--component-level precertification and DOD-level certification 
and approval. At the component level, program managers prepare, enter, 
maintain, and update information about their investments in the Air 
Force data repository. The component precertification authority 
validates that the system information is complete and accessible on the 
Air Force data repository, reviews system compliance with the business 
enterprise architecture and enterprise transition plan, and verifies 
the economic viability analysis. This information is then transferred 
to DOD's IT Portfolio Repository.[Footnote 31] The precertification 
authority asserts the status and validity of the investment information 
by submitting a component precertification letter to the appropriate 
Investment Review Board for its review. 

* Annual reviews: Tier 1 through 4 business system investments are 
reviewed annually at the component and DOD-levels. At the component 
level, program managers annually review and update information on all 
tiers of system investments that are identified in their data 
repository. For Tier 1 through 3 systems that are in development or 
being modernized, information is updated on cost, milestone, and risk 
variances and actions or issues related to certification conditions. 
The precertification authority then verifies and submits the 
information for these business system investments for DOD's Investment 
Review Board review in an annual review assertion letter. The letter 
addresses system compliance with the DOD business enterprise 
architecture and the enterprise transition plan and includes investment 
cost, schedule, and performance information.[Footnote 32] 

At the DOD level, the Investment Review Boards annually review 
investments for certified Tier 1 through 3 business systems that are in 
development or are being modernized. These reviews focus on program 
compliance with the business enterprise architecture, program cost and 
performance milestones, and progress in meeting certification 
conditions. The Investment Review Boards can revoke an investment's 
certification when the system has significantly failed to achieve 
performance commitments (i.e., capabilities and costs). When this 
occurs, the component must address the Investment Review Board's 
concerns and resubmit the investment for certification. 

Air Force's Precertification Process: 

As stated earlier, DOD relies on its components to execute investment 
management policies and procedures. Air Force has developed a 
precertification process, which is intended to ensure that new or 
existing systems undergo proper scrutiny prior to being precertified by 
the Air Force CIO. First, the certification package is to be prepared 
by the Program Manager and reviewed by the Major Command (MAJCOM) and 
functional portfolio managers. 

The package is then to be provided to the Air Force Certification 
Process Manager, who is to review the package for completeness based on 
certification requirements and transmit the package to the 
Certification Review Team--which is composed of subject matter experts-
-to assess compliance with their relevant areas of expertise, such as 
the business enterprise architecture and information assurance. Once 
the certification review is complete, the package is to be sent to the 
Senior Working Group. This group is responsible for reviewing the 
package and approving or disapproving the package. Finally, the 
Precertification Authority is to precertify the system for submission 
to the relevant Investment Review Board. 

Table 2 lists decision-making personnel involved in Air Force's 
investment management process and provides a description of their key 
responsibilities. 

Table 2: Air Force Investment Management Governance Entities and 
Responsibilities: 

Entity: Precertification Authority; 
Roles and responsibilities: 
* Air Force Precertification Authority has the responsibility and 
authority to precertify any modernization investment in excess of $1 
million and certify any modernization investment less than or equal to 
$1million (Tier 4); 
Composition: The Secretary of the Air Force has the CIO as the Air 
Force Business Investment Approval Authority and Precertification 
Authority. 

Entity: Senior Working Group; 
Roles and responsibilities: 
* The Senior Working Group functions as the Investment Review Board at 
the component level. It provides cross-functional review and 
recommendations to the Air Force Precertification Authority regarding 
certification and annual review of business system investments; 
* The Senior Working Group provides guidance for Air Force 
modernization efforts. It also supports the CIO in ensuring compliance 
with public law and federal, DOD, and Air Force directives regulating 
the management and operation of IT investments; 
Composition: The Senior Working Group members are composed of general 
officers and senior executive service members from headquarter 
functional organizations. It is chaired by a representative from the 
Office of the CIO and has representation from various Air Force 
departments such as financial management and acquisition. 

Entity: Certification Review Team; 
Roles and responsibilities: 
* The Certification Review Team is comprised of subject matter experts 
tasked with reviewing each certification package for compliance with 
enterprise architecture, funding, information assurance, joint 
requirements, and the enterprise transition plan; 
Composition: The review team consists of Subject Matter Experts on 
Enterprise Architecture, Funding, Information Assurance, Joint 
Requirements, and the Enterprise Transition Plan. 

Entity: Certification Process Manager; 
Roles and responsibilities: 
* The Air Force Certification Process Manager is responsible for 
executing the certification and annual review processes. In support of 
this effort, the Certification Process Manager tracks certification 
conditions, and communicates and documents certification decisions; 
Composition: The Air Force Certification Process Manager has a unit 
within the Office of the CIO tasked with supporting investment 
activities. 

Entity: Functional Portfolio Manager; 
Roles and responsibilities: 
* Functional portfolio managers (functional CIOs) are responsible for 
managing the Air Force functional portfolios; 
* Managers are responsible for establishing a cross-functional 
investment review board to conduct portfolio management activities; 
Composition: These managers consist of Air Force's various functional 
portfolios, which include Acquisition, Financial Management, Human 
Resource Management, and Logistics. 

Entity: Major Command Portfolio Manager; 
Roles and responsibilities: 
* MAJCOM Portfolio Managers are responsible for managing a MAJCOM 
portfolio that includes the subject business system; 
* Managers are responsible for reviewing investment management 
documentation (e.g., Balanced Scorecard, Functions Gap/Redundancy 
Assessment, Architecture Alignment) on submitted initiatives, reviewing 
certification packages, and approving or disapproving the certification 
package; 
* Managers are responsible for establishing a cross-mission investment 
review board, which conducts portfolio management activities across the 
mission areas; 
Composition: These managers consist of Air Force's various major 
commands. These include Air Combat Command, Air Education and Training 
Command, Air Force Materiel Command, Air Force Reserve Command, Air 
Force Space Command, Air Force Special Operations Command, Air Mobility 
Command, Pacific Air Forces, and U.S. Air Forces in Europe. 

Entity: System Program Manager; 
Roles and responsibilities: 
* The Program Manager is responsible for managing a business system. 
Program managers are responsible for complete, current, and accurate 
information contained in the Air Force data repository relevant to 
their systems. They are also responsible for completing initial 
certification packages and modifying certification packages to address 
issues from all levels of review; 
Composition: These managers are responsible for the development, 
implementation, and maintenance of their individual systems. 

Source: GAO analysis of Air Force data. 

[End of table] 

Figure 5 shows the relationship among the key players in Air Force's 
precertification review and approval process. 

Figure 5: Air Force Precertification Review and Approval Process: 

This figure is a chart showing the Air Force precertification review 
and approval process. 

[See PDF for image] 

Source: GAO, based on Air Force documentation. 

[End of figure] 

Air Force Has Established the Structures Needed to Effectively Manage 
Business System Investments but Has Not Fully Defined Many of the 
Related Policies and Procedures: 

DOD relies on its components to execute investment management policies 
and procedures.[Footnote 33] However, while Air Force has established 
the basic management structures needed to effectively manage its IT 
projects as investments, it has not fully implemented many of the 
related policies and procedures outlined in our ITIM framework. 
Relative to its business system investments, Air Force has implemented 
three of nine practices that call for project-level structures, 
policies, and procedures, and has not defined any of the five practices 
that call for portfolio-level policies and procedures. Air Force 
officials stated that they are aware of the absence of documented 
policies and procedures, and they are currently working on guidance to 
address these areas. For example, these officials stated that they have 
drafted policies and procedures to establish portfolio-level practices 
and are currently obtaining the necessary approvals. Air Force plans to 
complete and approve these policies and procedures by December 2007. 
According to our framework, adequately documenting both the policies 
and the associated procedures that govern how an organization manages 
its IT investment portfolio is important because doing so provides the 
basis for having rigor, discipline, and repeatability in how 
investments are selected and controlled across the entire organization. 

Until Air Force has fully defined policies and procedures for both 
individual projects and the portfolio of projects, it risks selecting 
and controlling these business system investments in an inconsistent, 
incomplete, and ad hoc manner, which in turn could reduce the chances 
that these investments will meet mission needs in the most effective 
manner. 

Air Force Has Begun to Build a Foundation for Project-Level Investment 
Management but Has Not Yet Fully Defined Key Policies and Procedures: 

At ITIM Stage 2, an organization has attained repeatable and successful 
IT project-level investment control processes and basic selection 
processes. Through these processes, the organization can identify 
project expectation gaps early and take the appropriate steps to 
address them. ITIM Stage 2 critical processes include (1) defining 
investment board operations, (2) identifying the business needs for 
each investment, (3) developing a basic process for selecting new 
proposals and reselecting ongoing investments, (4) developing project- 
level investment control processes, and (5) collecting information 
about existing investments to inform investment management decisions. 
Table 3 describes the purpose of each of these Stage 2 critical 
processes. 

Table 3: Stage 2 Critical Processes--Building the Investment 
Foundation: 

Critical process: Instituting the investment board; 
Purpose: To define and establish an appropriate investment management 
structure and the processes for selecting, controlling, and evaluating 
investments. 

Critical process: Meeting business needs; 
Purpose: To ensure that investments support the organization's business 
needs and meet users' needs. 

Critical process: Selecting an investment; 
Purpose: To ensure that a well-defined and disciplined process is used 
to select new proposals and reselect ongoing investments. 

Critical process: Providing investment oversight; 
Purpose: To review the progress of investments, using predefined 
criteria and checkpoints, in meeting cost, schedule, risk, and benefit 
expectations and to take corrective action when these expectations are 
not being met. 

Critical process: Capturing investment information; 
Purpose: To make information available to decision makers to evaluate 
the impacts and opportunities created by proposed (or continuing) 
investments. 

Source: GAO. 

[End of table] 

Within these five critical processes are nine key practices required 
for effective project-level management. Air Force has fully defined the 
policies and procedures for three of these nine practices. 
Specifically, Air Force has established a management structure by 
instituting a business system Investment Review Board, called the 
Senior Working Group. This group is composed of senior executives from 
the functional business units, including the Office of the Air Force 
Chief Information Officer, and the members are responsible for 
establishing and implementing investment policies. In addition, Air 
Force has established policies and procedures for capturing information 
about its IT projects and systems and submitting, updating, and 
maintaining this information in its data repository. Finally, it has 
assigned the Certification Process Manager the responsibility of 
ensuring that specific investment information contained in the Air 
Force data repository is accurate and complete. 

However, the Air Force's policies and procedures associated with the 
remaining six project-level management practices are missing critical 
elements needed to effectively carry out essential investment 
management activities. For example: 

* Policies and procedures for directing the Investment Review Board's 
operations do not define how investments that are in operations and 
maintenance are to be governed by the Investment Review Board. In 
addition, procedures do not specify how the business investment 
management process is coordinated with other DOD management systems. 
Without clearly defined guidance and visibility into all investments 
with an understanding of decisions reached through other management 
systems, Air Force cannot be assured that consistent investment 
management decisions are being made. 

* Policies and procedures do not define how systems in operations and 
maintenance will support ongoing and future business needs. This 
increases the risk that Air Force will continue to maintain legacy 
investments that no longer support current organizational objectives. 

* Policies and procedures for selecting new systems do not specify how 
the full range of cost, schedule, and performance data are being 
considered in making selection (i.e., precertification) decisions. 
Without documenting how factors such as cost, schedule, and performance 
are considered when making precertification decisions, Air Force cannot 
ensure that it consistently and objectively selects system investments 
that best meet the department's needs and priorities. 

* Policies and procedures do not include a structured method that 
defines how the criteria will be evaluated when the precertification 
authority makes reselection decisions. In addition, policies and 
procedures do not define an approach to annually reviewing systems in 
operations and maintenance. Given that Air Force spends millions of 
dollars annually in operating and maintaining business systems, this is 
significant. Without an understanding of how the precertification 
authority is to consider these investments when making reselection 
decisions, Air Force's ability to make informed and consistent 
reselection and termination decisions is limited. 

* Policies and procedures do not specify how funding decisions are 
integrated with the process of selecting an investment. Without 
considering budget constraints and opportunities, Air Force risks 
making investment decisions that do not effectively consider the 
relative merits of various projects and systems when funding 
limitations exist. 

* Policies and procedures do not provide for sufficient oversight and 
visibility into investment management activities. Air Force has 
predefined criteria for adherence to cost, schedule, and performance 
milestones, but does not have policies and procedures that guide the 
implementation of corrective actions when program expectations are not 
met. Without such policies and procedures, the agency risks investing 
in systems that are duplicative, stovepiped, nonintegrated, and 
unnecessarily costly to manage, maintain, and operate. 

Table 4 summarizes our findings relative to Air Force's execution of 
the nine key practices that call for the policies and procedures needed 
to manage IT investments at the project level. 

Table 4: Summary of Policies and Procedures for Stage 2 Critical 
Processes--Building the Investment Foundation: 

Critical process: Instituting the investment board; 
Key practice: 1. An enterprisewide IT investment board composed of 
senior executives from IT and business units is responsible for 
defining and implementing the organization's IT investment governance 
process; 
Rating: Executed; 
Summary of evidence: Air Force has established an investment board--the 
Senior Working Group--composed of senior executives from the functional 
business units, including the office of the Air Force Chief Information 
Officer. The board is responsible for establishing and implementing 
policies governing the organization's investment process. 

Critical process: [Empty]; 
Key practice: 2. The organization has a documented IT investment 
process directing each investment board's operations; 
Rating: Not executed; 
Summary of evidence: Air Force has an IT investment process for 
directing its investment board, which explains the roles and 
responsibilities of the board and the individuals involved. Air Force 
assigns the Investment Review Board accountability for systems 
throughout the investment life cycle, including investments that are in 
the operations and maintenance phase. However, Air Force's policies and 
procedures do not define the process by which these systems will be 
reviewed by the Investment Review Board. In addition, according to our 
ITIM guidance, the department's investment guidance should specify the 
manner in which investment-related processes will be coordinated with 
other organizational plans, processes, and documents. However, Air 
Force's Investment Review Guide does not specify how the business 
investment management system is coordinated with other DOD management 
systems, such as JCIDS, PPBE, and DAS. 

Critical process: Meeting business needs; 
Key practice: 1. The organization has documented policies and 
procedures for identifying IT projects or systems that support the 
organization's ongoing and future business needs; 
Rating: Not executed; 
Summary of evidence: Although Air Force has begun to review systems in 
operations and maintenance, it does not have policies and procedures 
for ensuring that these systems support ongoing and future business 
needs. Air Force guidance dictates that investments undergoing the 
certification process must demonstrate that they support ongoing and 
future business needs by complying with the Enterprise Transition Plan 
and Business Enterprise Architecture. However, Air Force certification 
guidance does not apply to system investments in operations and 
maintenance, which account for about 60 percent of Air Force's overall 
IT budget. 

Critical process: Selecting an investment; 
Key practice: 1. The organization has documented policies and 
procedures for selecting a new investment; 
Rating: Not executed; 
Summary of evidence: While the Investment Review Guide defines Air 
Force's roles and responsibilities for certifying and approving 
investments and includes predefined criteria, such as meeting cost, 
schedule, and performance milestones, for selecting investments, it 
does not contain a structured approach for how precertification 
decisions are reached. For example, the guidance does not specify how 
factors, such as cost, schedule, and performance data, are to be used 
in making precertification decisions. 

Critical process: [Empty]; 
Key practice: 2. The organization has documented policies and 
procedures for reselecting ongoing investments; 
Rating: Not executed; 
Summary of evidence: Air Force's Investment Review Guide defines Air 
Force's approach for annually reviewing investments. However, this 
guidance does not include a structured method that defines how the 
criteria, such as meeting cost, schedule and performance milestones, 
will be evaluated when the precertification authority makes reselection 
decisions. In addition, the Investment Review Guide only addresses 
systems in development/ modernization (Tier 1 through 4) and does not 
define an approach for annually reviewing systems in operations and 
maintenance. 

Critical process: [Empty]; 
Key practice: 3. The organization has documented policies and 
procedures for integrating funding with the process of selecting an 
investment; 
Rating: Not executed; 
Summary of evidence: Air Force policies and procedures do not specify 
how investment funding and selection are to be integrated nor how the 
precertification authority is to use funding information to make 
certification or approval decisions. 

Critical process: Providing investment oversight; 
Key practice: 1. The organization has documented policies and 
procedures for management oversight of IT projects and systems; 
Rating: Not executed; 
Summary of evidence: Air Force does not have documented policies and 
procedures for overseeing the management of IT projects and systems. 
For example, while Air Force has predefined criteria and checkpoints 
for meeting cost, schedule, and performance milestones, and requires 
the development of corrective actions when a project deviates from 
milestones, it does not have policies and procedures that guide 
implementation of corrective actions. Further, Air Force may certify a 
system and impose conditions that must be met in order to obligate 
system funding. However, there are no policies and procedures that 
define the process for tracking conditions until they are resolved. 

Critical process: Capturing investment information; 
Key practice: 1. The organization has documented policies and 
procedures for identifying and collecting information about IT projects 
and systems to support the investment management process; 
Rating: Executed; 
Summary of evidence: The Air Force Investment Review Guidance describes 
the procedures for submitting, updating, and maintaining information in 
its data repository. 

Critical process: [Empty]; 
Key practice: 2. An official is assigned responsibility for ensuring 
that the information collected during project and systems 
identification meets the needs of the investment management process; 
Rating: Executed; 
Summary of evidence: The Air Force Investment Review Guidance assigns 
the Program Manager responsibility to ensure investment information 
contained in its data repository is accurate and complete. The guidance 
also assigns the Certification Process Manager with responsibility for 
verifying these data. 

Source: GAO. 

[End of table] 

Air Force officials stated that they are aware of the absence of 
documented procedures in certain areas of project-level management, and 
plan to issue new policies and procedures addressing these areas by 
December 2007. However, until Air Force fully documents IT investment 
management policies and procedures for Stage 2 activities and specifies 
the linkages between the various related processes, and describes how 
system investments in operations and maintenance are to be governed, it 
risks not being able to carry out investment management activities in a 
consistent and disciplined manner. Moreover, the Air Force risks 
selecting investments that will not effectively meet its mission needs. 

Air Force Has Assigned Responsibility but Has Not Defined the Policies 
and Procedures Associated with Effective Portfolio-Level Management: 

At Stage 3, an organization has defined critical processes for managing 
its investments as a portfolio or a set of portfolios.[Footnote 34] 
Portfolio management is a conscious, continuous, and proactive approach 
to allocating limited resources among competing initiatives in light of 
the investments' relative benefits. Taking a departmentwide perspective 
enables an organization to consider its investments comprehensively, so 
that collectively the investments optimally address the organization's 
missions, strategic goals, and objectives. Managing IT investments as 
portfolios also allows an organization to determine its priorities and 
make decisions about which projects to fund on the basis of analyses of 
the relative organizational value and risks of all projects, including 
projects that are proposed, under development, and in operation. 
Although investments may initially be organized into subordinate 
portfolios--on the basis of, for example, business lines or life-cycle 
stages--and managed by subordinate investment boards, they should 
ultimately be aggregated into enterprise-level portfolios. 

According to ITIM, Stage 3 involves four critical processes (1) 
defining the portfolio criteria; (2) creating the portfolio; (3) 
evaluating (i.e., overseeing) the portfolio; and (4) conducting post-
implementation reviews. Within these critical processes are five key 
practices that call for policies and procedures to ensure effective 
portfolio management. Table 5 summarizes the purpose of each of the 
critical processes. 

Table 5: Stage 3 Critical Processes--Developing a Complete Investment 
Portfolio: 

Critical process: Defining the portfolio criteria; 
Purpose: To ensure that the organization develops and maintains 
portfolio selection criteria that support its mission, organizational 
strategies, and business priorities. 

Critical process: Creating the portfolio; 
Purpose: To ensure that investments are analyzed according to the 
organization's portfolio selection criteria and to ensure that an 
optimal investment portfolio with manageable risks and returns is 
selected and funded. 

Critical process: Evaluating the portfolio; 
Purpose: To review the performance of the organization's investment 
portfolio(s) at agreed- upon intervals and to adjust the allocation of 
resources among investments as necessary. 

Critical process: Conducting post-implementation reviews; 
Purpose: To compare the results of recently implemented investments 
with the expectations that were set for them and to develop a set of 
lessons learned from these reviews. 

Source: GAO. 

[End of table] 

Air Force has begun to establish a governance structure for portfolio- 
level management, but it has not executed any of the five practices 
within the Stage 3 critical processes that call for policies and 
procedures associated with effective portfolio-level management. 
Specifically, Air Force has assigned the Senior Working Group the 
responsibility of establishing a governance forum to oversee business 
system portfolio activities. However, the Senior Working Group has not 
developed and approved charters outlining the roles and 
responsibilities to be assigned to subordinate Investment Review Boards 
that are intended to establish and manage the portfolios. 

In addition, Air Force has not fully defined policies and procedures 
needed to effectively execute portfolio management practices. 
Specifically, Air Force does not have policies and procedures for 
defining the portfolio criteria or creating and evaluating the 
portfolio. In addition, while DOD has policies and procedures for 
conducting post-implementation reviews for Tier 1 systems as part of 
the Defense Acquisition System, Air Force has not established policies 
or procedures for conducting post-implementation reviews for systems in 
the remaining tiers. Finally, Air Force has not established procedures 
detailing how lessons learned from these reviews are to be used during 
investment reviews as the basis for management and process 
improvements. Table 6 summarizes the rating for each critical process 
required to manage investments as a portfolio and summarizes the 
evidence that supports these ratings. 

Table 6: Summary of Policies and Procedures for Stage 3 Critical 
Processes--Developing a Complete Investment Portfolio: 

Critical process-: Defining the portfolio criteria; 
Key practice-: 1. The organization has documented policies and 
procedures for creating and modifying IT portfolio selection criteria; 
Rating-: Not executed; 
Summary of evidence: While Air Force has assigned the Major Command and 
Functional Investment Review Boards responsibility for creating and 
modifying portfolio criteria (e.g., prioritization and investment trade-
offs) for business system investments, it has not yet finalized draft 
policies, procedures, or criteria for conducting IT portfolio 
selection. According to Air Force officials, the policies and 
procedures are expected to be finalized and approved by December 2007. 

Key practice-: Critical process-Creating the portfolio: 2. 
Responsibility is assigned to an individual or group for managing the 
development and modification of the IT portfolio selection criteria; 
Rating-: Critical process-Creating the portfolio: Not executed; 
Summary of evidence: Critical process-Creating the portfolio: Air Force 
has assigned responsibility for overseeing portfolio management to the 
Senior Working Group. However, although Air Force officials stated that 
subordinate Major Command and Functional Investment Review Boards are 
to be responsible for managing specific portfolios, it has yet to 
officially assign responsibility to these groups. 

Critical process-: Creating the portfolio; 
Key practice-: 1. The organization has documented policies and 
procedures for analyzing, selecting, and maintaining the investment 
portfolios; 
Rating-: Not executed; 
Summary of evidence: Air Force is currently revising its instruction on 
the IT portfolio management process, to include policies and procedures 
for analyzing, selecting, and maintaining the investment portfolios. 
However, Air Force officials could not identify a date for when the 
instruction would be finalized and approved. 

Critical process-: Evaluating the portfolio; 
Key practice-: 1. The organization has documented policies and 
procedures for reviewing, evaluating, and improving the performance of 
its portfolio(s); 
Rating- : Not executed; 
Summary of evidence: While the Major Command and Functional Investment 
Review Board draft charters state that they are responsible for 
reviewing factors associated with portfolio management, such as 
architecture alignment, capability delivery, and risk, there are no 
policies and procedures indicating how they should use these factors 
and project indicators--such as cost, schedule, and risk--to review, 
evaluate, and improve their portfolios. 

Critical process-: Conducting post-implementation reviews; 
Key practice-: 1. The organization has documented policies and 
procedures for conducting post-implementation reviews; 
Rating-: Not executed; 
Summary of evidence: While DOD requires post-implementation reviews for 
Tier 1 systems as part of DAS, Air Force has not developed policies or 
procedures for conducting such reviews for systems in the remaining 
tiers. Moreover, there are no policies and procedures directing the Air 
Force Senior Working Group, which is accountable for Air Force business 
system investments, to consider information gathered and to develop 
lessons learned from these post-implementation reviews. 

Source: GAO. 

[End of table] 

Air Force officials are aware that they need to develop the appropriate 
portfolio management processes, and in this regard, have drafted some 
portfolio management guidance, such as the Air Force Operational 
Support Portfolio Investment Review Process. According to Air Force 
officials, this guidance is expected to be completed and approved by 
December 2007. Until policies and procedures for managing business 
systems investment portfolios are defined and implemented, Air Force is 
at risk of not consistently selecting the mix of investments that best 
supports the department mission needs and ensuring that investment- 
related lessons learned are shared and applied departmentwide. 

Conclusions: 

Given the importance of business systems modernization to Air Force's 
mission, performance, and outcomes, it is vital for the department to 
adopt and employ an effective institutional approach to managing 
business system investments. However, while the department acknowledges 
these shortcomings and the importance of addressing them and has 
established aspects of such an approach, it is lacking important 
elements, such as policies and procedures needed for project-level and 
portfolio-level investment management. This means that Air Force lacks 
an institutional capability to ensure that it is investing in business 
systems that best support its strategic needs and that ongoing projects 
meet cost, schedule, and performance expectations. Until Air Force 
develops this capability, the department will be impaired in its 
ability to optimize business mission area performance and 
accountability. 

Recommendations for Executive Action: 

To strengthen Air Force's business system investment management 
capability and address the weaknesses discussed in this report, we 
recommend that the Secretary of Defense direct the Secretary of the Air 
Force to ensure that well-defined and disciplined business system 
investment management policies and procedures are developed and issued. 
At a minimum, this should include project-level management policies and 
procedures that address the six key practices areas: 

* Specifying how systems that are in operations and maintenance will be 
reviewed and specifying how Air Force's business investment management 
system is coordinated with JCIDS, PPBE, and DAS. 

* Ensuring that systems in operations and maintenance are aligned with 
ongoing and future business needs. 

* Selecting investments, including specifying how factors, such as 
cost, schedule, and performance data are to be used in making 
certification decisions. 

* Reselecting ongoing investments, including specifying how factors, 
such as cost, schedule, and performance data are to be used in making 
reselection decisions during the annual review process and providing 
for the reselection of investments that are in operations and 
maintenance. 

* Integrating funding with the process of selecting an investment, 
including specifying how the precertification authority is using 
funding information in carrying out decisions on system certification 
and approvals. 

* Overseeing IT projects and systems, including specifying policies and 
procedures that guide the implementation of corrective actions when 
program expectations are not met. 

These well-defined and disciplined business system investment 
management policies and procedures should also include portfolio-level 
management policies and procedures that address the following five 
areas: 

* Creating and modifying IT portfolio selection criteria for business 
system investments. 

* Defining the roles and responsibilities for the development and 
modification of the IT portfolio selection criteria. 

* Analyzing, selecting, and maintaining business system investment 
portfolios. 

* Reviewing, evaluating, and improving the performance of the 
portfolio(s) by using project indicators, such as cost, schedule, and 
risk. 

* Conducting post-implementation reviews for all investment tiers and 
directing the investment boards to consider the information gathered to 
develop lessons learned from these reviews. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Business Transformation) and reprinted in 
appendix II, DOD partially concurred with the report's recommendations. 
The department further stated that our recommendations and feedback 
were helpful in guiding its business transformation and related 
improvement efforts. 

Regarding our recommendation that DOD direct Air Force to ensure that 
well-defined and disciplined business system investment management 
policies and procedures are developed and issued, DOD stated that Air 
Force has a well-defined business system investment management process 
that, while not codified into formal policy, has been documented in 
investment review guides. Further, it stated that Air Force is 
committed to formalizing its policies as the processes expand and 
mature. Our report recognizes that Air Force has drafted a business 
system investment management process. However, as our report states, 
the draft review guide lacks critical elements needed to effectively 
carry out essential investment management activities. Until Air Force 
completes and issues IT investment management policies and procedures 
that fully address these elements, it risks not being able to carry out 
investment management activities in a consistent and disciplined 
manner. 

Regarding our recommendation that DOD direct Air Force to ensure that 
the well-defined and disciplined business system investment management 
policies and procedures also include portfolio-level management 
policies and procedures, the department stated that DOD Instruction 
8115.02 defines the DOD IT portfolio management process, which Air 
Force is applying in its decisionmaking. However, as our report notes, 
Air Force has not implemented a process for managing its business 
system portfolios. Until Air Force defines and implements such a 
process, it is at risk of not consistently selecting the mix of 
investments that best supports the Air Force's mission needs and 
ensuring that investment-related lessons learned are shared and applied 
departmentwide. 

We are sending copies of this report to interested congressional 
committees; the Director, Office of Management and Budget; the 
Secretary of Defense; the Deputy Secretary of Defense; the Secretary of 
Air Force; the Air Force Chief Information Officer, and the Under 
Secretary of Defense for Acquisition, Technology, and Logistics. Copies 
of this report will be made available to other interested parties on 
request. This report will also be available at no charge on our Web 
site at [hyperlink, http://www.gao.gov]. 

Should you or your staffs have any questions on matters discussed in 
this report, please contact me at (202) 512-6304 or melvinv@gao.gov. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. GAO staff who 
made major contributions to this report are listed in appendix III. 

Signed by: 

Valerie C. Melvin: 

Director, Human Capital and Management: 

Information Systems Issues: 

List of Committees: 

The Honorable Carl Levin: 
Chairman: 
The Honorable John McCain: 
Ranking Member: 
Committee on Armed Services: 
United States Senate: 

The Honorable Daniel Inouye: 
Chairman: 
The Honorable Ted Stevens: 
Ranking Member: 
Subcommittee on Defense: 
Committee on Appropriations: 
United States Senate: 

The Honorable Ike Skelton: 
Chairman: 
The Honorable Duncan Hunter: 
Ranking Member: 
Committee on Armed Services: 
House of Representatives: 

The Honorable John P. Murtha: 
Chairman: 
The Honorable C.W. Bill Young: 
Ranking Member: 
Subcommittee on Defense: 
Committee on Appropriations: 
House of Representatives: 

[End of section] 

Appendix I: Objective, Scope, and Methodology: 

Our objective was to determine whether the investment management 
approach of the Department of the Air Force (a major Department of 
Defense (DOD) component) is consistent with leading investment 
management best practices. Our analysis was based on the best practices 
contained in GAO's Information Technology Investment Management (ITIM) 
framework and the framework's associated evaluation methodology, and 
focused on Air Force's establishment of policies and procedures for 
business system investments needed to assist organizations in complying 
with the investment management provisions of the Clinger-Cohen Act of 
1996 (Stages 2 and 3). 

To address our objective, we asked Air Force to complete a self- 
assessment of its investment management process and provide the 
supporting documentation. We then reviewed the results of the 
department's self-assessment of Stages 2 and 3 organizational 
commitment practices--meaning those practices related to structures, 
policies, and procedures--and compared them with our ITIM framework. We 
focused on Stages 2 and 3 because these stages represent the processes 
needed to meet the standards of the Clinger-Cohen Act, and they 
establish the foundation for effective acquisition management. We also 
validated and updated the results of the self-assessment through 
document reviews and interviews with officials, such as the Air Force 
Chief Information Officer and Certification Process Manager. In doing 
so, we reviewed written policies, procedures, and guidance and other 
documentation providing evidence of executed practices, including the 
Air Force Information Technology Investment Review Guide, Air Force 
Operations of Capabilities Based Acquisition System Instruction, Air 
Force IT Investment Architecture Compliance Guide, Senior Working Group 
charter and meeting minutes, the Investment Review Board Concept of 
Operations and Guidance, and the Business Transformation Guidance. 

We compared the evidence collected from our document reviews and 
interviews with the key practices in ITIM. We rated the key practices 
as "executed" on the basis of whether the department demonstrated (by 
providing evidence of performance) that it had met all of the criteria 
of the key practice. A key practice was rated as "not executed" when we 
found insufficient evidence of all elements of a practice being fully 
performed or when we determined that there were significant weaknesses 
in Air Force's execution of the key practice. In addition, we provided 
Air Force with the opportunity to produce evidence for the key 
practices rated as "not executed." 

We conducted our work at the Air Force offices in Arlington, Virginia, 
from February 2007 through September 2007 in accordance with generally 
accepted government auditing standards. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
3000 Defense Pentagon: 
Washington, DC 20301-3000: 

ACQUISITION, TECHNOLOGY AND LOGISTICS: 

October 18, 2007: 

Ms. Valerie C. Melvin:
Director, Human Capital and Management Information Systems Issues: 
U.S. Government Accountability Office: 
441 G Street, N.W.: 
Washington, DC 20548: 

Dear Ms. Melvin:

This is the Department of Defense (DoD) response to the GAO draft 
report GAO-08-52, "Business Systems Modernization: Air Force Needs to 
Fully Define Policies and Procedures for Institutionally Managing 
Investments," dated September 17, 2007 (GAO Code 310635). 

The Department partially concurs with GAO's first recommendation. The 
Air Force has established a well-defined business system investment 
management process that, while not codified into formal policy, has 
been documented in its investment review guides. Furthermore, the Air 
Force is committed to formalizing its policies as the processes expand 
and mature. 

The Department also partially concurs with the second recommendation. 
DoD Instruction 8115.02 defines the DoD Information Technology (IT) 
portfolio management process and the Air Force is applying this process 
in its decision- making. This process is also maturing. 

GAO continues to be a valuable and constructive partner in the 
Department's business transformation efforts. The recommendations and 
feedback provided will help to further guide DoD's process of continual 
improvement. We look forward to your participation in our future 
efforts. 

Signed by: 

Paul A. Brinkley: 

Deputy Under Secretary of Defense: 
(Business Transformation): 

GAO Draft Report Dated September 17, 2007 GAO-08-52 (GAO CODE 310635):  

"Business Systems Modernization: Air Force Needs To Fully Define 
Policies And Procedures For Institutionally Managing Investments": 

Department Of Defense Comments To The Gao Recommendation: 

Recommendation 1: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Air Force to ensure that well-defined and 
disciplined business system investment management policies and 
procedures are developed and issued. At a minimum, these should include 
project-level management policies and procedures that address: 

* How systems that are in operations and maintenance will be reviewed 
and specifying how Air Force's business investment management system is 
coordinated with Joint Capabilities Integration and Development System 
(JCIDS), Planning, Programming, Budgeting and Execution (PPBE), and 
Defense Acquisition System (DAS). 

* Ensuring that systems in operations and maintenance are aligned with 
ongoing and future business needs. 

* Selecting investments, including specifying how factors, such as 
cost, schedule, and performance data are to be used in making 
certification decisions. 

* Reselecting ongoing investments, including specifying how factors, 
such as cost, schedule, and performance data are to be used in making 
reselection decisions during the annual review process and providing 
for the reselection of investments that are in operations and 
maintenance. 

* Integrating funding with the process of selecting an investment, 
including specifying how the pre-certification authority is using 
funding information in carrying out decisions on system certification 
and approvals. 

* Overseeing Information Technology (IT) projects and systems, 
including specifying policies and procedures that guide the 
implementation of corrective actions when program expectations are not 
met. 

(p. 38/GAO Draft Report): 

DOD Response: Partially concur: 

The Department partially concurs. The Air Force does have a well-
defined and disciplined business system investment management system 
process documented in investment review guides. The process is maturing 
but has not been codified in formal policies. The documented investment 
review and architecture guides are in use and distributed throughout 
the Air Force. The business system investment review process is 
expanding to capture investments in the warfighting and enterprise 
infrastructure environment mission areas. These processes will be 
instituted in formal policies. 

Recommendation 2: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Air Force to ensure that the above well-
defined and disciplined business system investment management policies 
and procedures also include portfolio- level management policies and 
procedures that address: 

* Creating and modifying IT portfolio selection criteria for business 
system investments.

* Defining roles and responsibilities for the development and 
modification of the IT portfolio selection criteria. 

* Analyzing, selecting, and maintaining business system investment 
portfolios. 

* Reviewing, evaluating, and improving the performance of the 
portfolio(s) by using project indicators, such as cost, schedule, and 
risk. 

* Conduct post-implementation reviews for all investment tiers and 
directing the investment boards to consider the information gathered to 
develop lessons learned from these reviews. 

(p. 38/GAO Draft Report): 

DOD Response: Partially concur: 

DoD instruction 8115.02 defines the DoD IT portfolio management 
process. This process is maturing and the Air Force is applying 
portfolio management discipline in their information technology 
decision making. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov: 

Staff Acknowledgments: 

In addition to the contact person named above, key contributors to this 
report were Tonia Johnson, Assistant Director; Idris Adjerid; Sharhonda 
Deloach; Nancy Glover; and Freda Paintsil. 

[End of section] 

Footnotes: 

[1] Business systems are information systems that include financial and 
nonfinancial systems and support DOD's business operations, such as 
civilian personnel, finance, health, logistics, military personnel, 
procurement, and transportation. 

[2] GAO, High-Risk Series: An Update, GAO-07-310 (Washington, D.C.: 
January 2007). 

[3] GAO, Information Technology: Architecture Needed to Guide 
Modernization of DOD's Financial Operations, GAO-01-525 (Washington, 
D.C.: May 17, 2001). 

[4] See, for example, GAO, DOD Business Systems Modernization: Long- 
standing Weaknesses in Enterprise Architecture Development Need to Be 
Addressed, GAO-05-702 (Washington, D.C.: July 22, 2005); DOD Business 
Systems Modernization: Billions Being Invested without Adequate 
Oversight, GAO-05-381 (Washington, D.C.: Apr. 29, 2005); DOD Business 
Systems Modernization: Limited Progress in Development of Business 
Enterprise Architecture and Oversight of Information Technology 
Investments, GAO-04-731R (Washington, D.C.: May 17, 2004); DOD Business 
Systems Modernization: Important Progress Made to Develop Business 
Enterprise Architecture, but Much Work Remains, GAO-03-1018 
(Washington, D.C.: Sept. 19, 2003); and GAO-01-525. 

[5] Ronald W. Reagan National Defense Authorization Act for Fiscal Year 
2005, Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct. 28, 
2004) (codified in part at 10 U.S.C. §2222). 

[6] GAO, Business Systems Modernization: DOD Needs to Fully Define 
Policies and Procedures for Institutionally Managing Investments, GAO-
07-538 (Washington, D.C.: May 11, 2007); Defense Business 
Transformation: A Comprehensive Plan, Integrated Efforts, and Sustained 
Leadership Are Needed to Assure Success, GAO-07-229T (Washington, D.C.: 
Nov. 16, 2006); Business Systems Modernization: DOD Continues to 
Improve Institutional Approach, but Further Steps Needed, GAO-06-658 
(Washington, D.C.: May 15, 2006); and DOD Business Systems 
Modernization: Important Progress Made in Establishing Foundational 
Architecture Products and Investment Management Practices, but Much 
Work Remains, GAO-06-219 (Washington, D.C.: Nov. 23, 2005). 

[7] GAO-07-538. 

[8] We rated the key practices as "executed" on the basis of whether 
the agency demonstrated (by providing evidence of performance) that it 
had met all of the criteria of the key practice. A key practice was 
rated as "not executed" when we found insufficient evidence of any 
elements of a practice being fully performed or when we determined that 
there were significant weaknesses in Air Force's execution of the key 
practice. 

[9] GAO, Information Technology Investment Management: A Framework for 
Assessing and Improving Process Maturity, GAO-04-394G (Washington, 
D.C.: March 2004). 

[10] GAO-04-394G. 

[11] GAO-06-658. 

[12] According to Air Force program officials, the total amount spent 
on its business systems includes the amount spent on the Operational 
Support IT Business Mission Area, Warfighting Mission Area, and 
Enterprise Information Environment. Approximately $651 million reflects 
the budget for the Operational Support IT mission area business systems 
and does not include the amount budgeted for the Warfighting Mission 
Area and Enterprise Information Environment business systems. 

[13] The Clinger-Cohen Act of 1996, 40 U.S.C. 11101-11704. This act 
expanded the responsibilities of OMB and the agencies that had been set 
under the Paperwork Reduction Act with regard to IT management. See 44 
U.S.C. 3504(a)(1)(B)(vi) (OMB); and 44 U.S.C. 3506(h)(5) (agencies). 

[14] We have made recommendations to improve OMB's process for 
monitoring high-risk IT investments; see GAO, Information Technology: 
OMB Can Make More Effective Use of Its Investment Reviews, GAO-05-276 
(Washington, D.C.: Apr. 15, 2005). 

[15] This policy is set forth and guidance is provided in OMB Circular 
A-11 (Nov. 2, 2005) (section 300) and in OMB's Capital Programming 
Guide, which directs agencies to develop, implement, and use a capital 
programming process to build their capital asset portfolios. 

[16] See, for example, GAO-04-394G; GAO, Information Technology: A 
Framework for Assessing and Improving Enterprise Architecture 
Management (Version 1.1), GAO-03-584G (Washington, D.C.: April 2003); 
and Assessing Risks and Returns: A Guide for Evaluating Federal 
Agencies' IT Investment Decision-making, GAO/AIMD-10.1.13 (Washington, 
D.C.: February 1997). 

[17] GAO-04-394G; GAO/AIMD-10.1.13; GAO, Executive Guide: Improving 
Mission Performance Through Strategic Information Management and 
Technology, GAO/AIMD-94-115 (Washington, D.C.: May 1994); and Office of 
Management and Budget, Evaluating Information Technology Investments, A 
Practical Guide (Washington, D.C.: November 1995). 

[18] GAO-04-394G. 

[19] GAO, Information Technology: Centers for Medicare and Medicaid 
Services Needs to Establish Critical Investment Management 
Capabilities, GAO-06-12 (Washington, D.C.: Oct. 28, 2005); Information 
Technology: HHS Has Several Investment Management Capabilities in 
Place, but Needs to Address Key Weaknesses, GAO-06-11 (Washington, 
D.C.: Oct. 28, 2005); Information Technology: FAA Has Many Investment 
Management Capabilities in Place, but More Oversight of Operational 
Systems Is Needed, GAO-04-822 (Washington, D.C.: Aug. 20, 2004); Bureau 
of Land Management: Plan Needed to Sustain Progress in Establishing IT 
Investment Management Capabilities, GAO-03-1025 (Washington, D.C.: 
Sept. 12, 2003); Information Technology: Departmental Leadership 
Crucial to Success of Investment Reforms at Interior, GAO-03-1028 
(Washington, D.C.: Sept. 12, 2003); United States Postal Service: 
Opportunities to Strengthen IT Investment Management Capabilities, GAO- 
03-3 (Washington, D.C.: Oct. 15, 2002); and Information Technology: DLA 
Needs to Strengthen Its Investment Management Capability, GAO-02-314 
(Washington, D.C.: Mar. 15, 2002). 

[20] The Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11311-11313. 

[21] The National Security Strategy Report required by 50 U.S.C. 404a 
is a comprehensive report on the national security strategy of the 
United States submitted by the President to Congress. 

[22] See 10 U.S.C. 118. The Quadrennial Defense Review is a 
comprehensive examination of the national defense strategy, force 
structure, force modernization plans, infrastructure, budget plan, and 
other elements of the defense program and policies of the United States 
with a view toward determining and expressing the defense strategy of 
the United States and establishing a defense program for the next 20 
years. 

[23] GAO, Best Practices: An Integrated Portfolio Management Approach 
to Weapon System Investments Could Improve DOD's Acquisition Outcomes, 
GAO-07-388 (Washington, D.C.: Mar. 30, 2007). 

[24] As described in DOD Directive 5000.1, May 12, 2003 and DOD 
Instruction 5000.2, May 12, 2003. 

[25] A Major Defense Acquisition Program is an acquisition program that 
is estimated by the Under Secretary of Defense for Acquisition, 
Technology, and Logistics to require an eventual total expenditure for 
research, development, and test and evaluation of more than $365 
million (fiscal year 2000 constant dollars) or, for procurement, of 
more than $2 billion (fiscal year 2000 constant dollars). 

[26] A Major Automated Information System is a program or initiative 
that is so designated by the Assistant Secretary of Defense (Networks 
and Information Integration)/Chief Information Officer or that is 
estimated to require program costs in any single year in excess of $32 
million (fiscal year 2000 constant dollars), total program costs in 
excess of $126 million (fiscal year 2000 constant dollars), or total 
life-cycle costs in excess of $378 million (fiscal year 2000 constant 
dollars). 

[27] According to DOD, the milestone decision authority is the 
designated individual who has overall responsibility for an investment. 
This person has the authority to approve an investment's progression in 
the acquisition process and is responsible for reporting cost, 
schedule, and performance results. For example, the milestone decision 
authority for a Major Defense Acquisition Program, when not delegated 
to the component level, is the Under Secretary of Defense for 
Acquisition, Technology, and Logistics, and the milestone decision 
authority for a Major Automated Information System is the Assistant 
Secretary of Defense (Networks and Information Integration)/Chief 
Information Officer or a designee. 

[28] The Defense Acquisition Board--chaired by the Under Secretary of 
Defense for Acquisition, Technology, and Logistics--conducts reviews 
for Major Defense Acquisition Programs at major program milestones and 
documents the decisions resulting from the review in an Acquisition 
Decision Memorandum. 

[29] The IT Acquisition Board--chaired by the Assistant Secretary of 
Defense (Networks and Information Integration)/Chief Information 
Officer--conducts reviews for Major Automated Information System at 
major program milestones and documents the decision(s) resulting from 
the review in an Acquisition Decision Memorandum. 

[30] The four Investment Review Boards are (1) financial management, 
established by the Deputy Under Secretary of Defense for Financial 
Management; (2) weapon systems life-cycle management and materiel 
supply and services management; (3) real property and installations 
life-cycle management, both established by the Under Secretary of 
Defense (Acquisition, Technology, and Logistics); and (4) human 
resources management, established by the Under Secretary of Defense for 
Personnel and Readiness. 

[31] DOD's IT Portfolio Repository is DOD's authoritative repository 
for certain information about DOD's business systems, such as system 
names and the responsible DOD components that are required for the 
certification, approval, and annual reviews of these business system 
investments. 

[32] In addition, each component precertification authority submits a 
list of system names to the Investment Review Boards on a semiannual 
basis, to include Tier 4 systems and systems in operations and 
maintenance that have been reviewed at the component level. 

[33] These investment management policies and procedures include 
precertifying Tier 1 through 3 business system investments by the 
component. These systems are then reviewed and certified by DOD. Tier 4 
systems are certified by the components. 

[34] Investment portfolios are integrated agencywide collections of 
investments that are assessed and managed collectively on the basis of 
common criteria. 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, DC 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, Jarmong@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548: