This is the accessible text file for GAO report number GAO-03-406 
entitled 'Customs Service Modernization: Automated Commercial 
Environment Progressing, but Further Acquisition Management 
Improvements Needed' which was released on February 28, 2003.



This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO products’ accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to Webmaster@gao.gov.



Report to Congressional Committees:



February 2003:



Customs Service Modernization:



Automated Commercial Environment Progressing, but Further Acquisition 

Management Improvements Needed:



GAO-03-406:



GAO Highlights:



Highlights of GAO-03-406, a report to Congressional committees:



Why GAO Did This Study:



The U.S. Customs Service is conducting a multiyear, multibillion

dollar project, the Automated Commercial Environment (ACE), a

new trade processing system that is planned to support effective and

efficient movement of goods into the United States. By congressional

mandate, Customs’ expenditure plans for ACE must meet certain

conditions, including being reviewed by GAO. This study

addresses whether Customs’ latest plan satisfies these conditions and

provides observations about the plan and Customs’ efforts to

implement GAO’s open recommendations for improving ACE management.



What GAO Found:



Customs’ November 2002 ACE expenditure plan, the fourth in a series of

legislatively required plans, provides for certain project management 

tasks as well as the definition, design, and development of the first

 release of the second of four planned ACE increments. GAO’s analysis 

of the plan shows that it meets the legislative conditions imposed by 

the Congress.



In its series of reports on Customs’ management of ACE, GAO has made 

a number of recommendations, which Customs is currently addressing.

However, Customs has been slow to correct weaknesses in two areas

fundamental to effective acquisition management—people and processes.

These weaknesses increase the risk that ACE will be late, cost more 

than necessary, and not perform as intended.



*  While the Customs Modernization Office (CMO) has developed a human

capital strategy, neither the strategy nor supporting documentation

identifies how the commitments made in the strategy will be met,

including what steps will be taken and what resources are needed to

execute the steps. Further, based on the strategy’s timeline, it will 

be over a year before the CMO fully implements the strategy.



*  Customs’ has made slow progress in implementing key acquisition

practices, such as project office management and acquisition risk

management, which GAO first recommended Customs do in 1999. Since

that time many practices have been developed, but almost none of 

them have been implemented, as shown in the figure below.



Additionally, Customs’ ACE contractor is conducting system tests in

accordance with best practices, initial test results are positive, 

and indicators suggest the contractor is delivering a quality product. 

However, Customs is not employing independent verification and 

validation (IV&V) in overseeing ACE testing, which is one way to 

mitigate the acquisition weaknesses cited above. Without IV&V, Customs 

states that it is relying on the contractor’s reputation and maturity 

level as guarantors of system quality, which is not adequate for a 

complex and risky program like ACE.



What GAO Recommends:



To ensure that Customs is positioned to effectively manage

the acquisition of ACE, GAO is making recommendations to the

commissioner aimed at improving Customs’ acquisition management

capabilities. Customs concurred with GAO’s recommendations and

described specific actions that it is taking to respond to each.



Contents:



Letter:



Recommendations for Executive Action:



Agency Comments and Our Evaluation:



Appendixes:



Appendix I: Customs’ Fourth Automated Commercial Environment (ACE) 

Expenditure Plan:



Appendix II: Comments from the U.S. Customs Service:



Abbreviations:



ACE: Automated Commercial Environment:



CIO: Chief Information Officer:



CMM: Capability Maturity Model:



CMO: Customs Modernization Office:



IV&V: independent verification and validation:



OMB: Office of Management and Budget:



SA-CMM: Software Acquisition Capability Maturity Model:



SAT: system acceptance tests:



SEI: Software Engineering Institute:



February 28, 2003:



The Honorable Ben Nighthorse Campbell

Chairman

The Honorable Byron L. Dorgan 

Ranking Minority Member

Subcommittee on Treasury and General Government

Committee on Appropriations

United States Senate:



The Honorable Ernest J. Istook, Jr.

Chairman

The Honorable Steny H. Hoyer

Ranking Minority Member

Subcommittee on Transportation, Treasury, Postal Service,

 and General Government

Committee on Appropriations

House of Representatives:



In November 2002, the U.S. Customs Service submitted to Congress its 

fourth expenditure plan, seeking release of $314 million for its 

Automated Commercial Environment (ACE) project. ACE is to be Customs’ 

new trade processing system and the first project under the Customs 

Modernization Program. As required by Customs’ fiscal year 2002 

appropriation and continuing resolutions,[Footnote 1] we reviewed the 

expenditure plan. Our objectives were (1) to determine whether the 

fourth ACE expenditure plan satisfies certain legislative conditions, 

(2) to determine whether the plan is consistent with our open ACE 

recommendations, and (3) to provide observations about the plan and 

Customs’ management of ACE.



On December 27, 2002, we briefed your offices on the results of this 

review. This report transmits the results of our work. The full 

briefing, including our scope and methodology, is reprinted as appendix 

I.



Concerning our first objective, Customs’ expenditure plan satisfies the 

legislative conditions specified in Customs’ appropriations act. That 

is, the plan provides for (1) meeting the capital planning and 

investment control review requirements of the Office of Management and 

Budget (OMB); (2) complying with Customs’ enterprise 

architecture;[Footnote 2] and (3) complying with federal acquisition 

rules, requirements, guidelines, and systems acquisition management 

practices. Further, the plan was reviewed and approved by the Joint 

Capital Investment Review Board[Footnote 3] and OMB.



Concerning our second objective, Customs is making progress in 

addressing our open recommendations, which are as follows:



* Justify and make investment decisions incrementally.



* Before building each ACE release, certify to Customs’ appropriations 

subcommittees that Customs’ enterprise architecture has been 

sufficiently extended and updated.



* Develop and implement a rigorous and analytically verifiable cost 

estimating program that embodies the tenets of effective estimating as 

defined in the Software Engineering Institute’s (SEI) institutional and 

project-specific estimating guidance.



* Immediately develop and implement a human capital management strategy 

for the Customs Modernization Office (CMO).



* Limit future expenditure plan requests for management reserve funds 

to 10 percent of the total funds requested for the program, or 

adequately justify any management reserve requests in excess of 10 

percent.



* Develop and implement process controls consistent with SEI’s Software 

Acquisition Capability Maturity Model (SA-CMM[Footnote 4]), and by 

September 30, 2002, assess and report to Customs’ appropriation 

subcommittees on the maturity of CMO’s software acquisition process.



* Address the risks associated with the accelerated ACE acquisition 

strategy and report on the strategy going forward and plans for 

mitigating the risks associated with this strategy.



While Customs has taken actions to address each of these 

recommendations, its actions to develop and implement a human capital 

strategy and acquisition management process controls were only 

partially consistent with our recommendations. Specifically, while the 

CMO developed a human capital strategy that specified high-level 

commitments and satisfied some best practices, there are some notable 

omissions, including how these commitments will be met. Further, based 

on the timeline in the strategy, it will be over a year before key 

practices are implemented.



Also, while Customs reported to its House and Senate appropriations 

subcommittees on its development of SA-CMM processes, it did not report 

on implementation. Further, while many of the acquisition processes 

have been developed, Customs has not yet implemented most of them.



Finally, we made the following observations:



* System integration and acceptance testing performed by the ACE 

contractor is being conducted in accordance with best practices. The 

contractor has established a test organization and built an 

infrastructure for test activities. It has completed system integration 

and system acceptance tests (SAT) for ACE increment 1, release 1.1, and 

these test activities were consistent with test management best 

practices.



* The results of SAT performed by the ACE contractor are positive. All 

but seven ACE increment 1, release 1.1 requirements successfully passed 

their test cases for SAT. For those that did not pass, only noncritical 

system defects resulted, and acceptable workarounds were identified.



* Data provided by the ACE contractor on all unresolved system defects 

for ACE increment 1, release 1.1, show a recent downward trend, 

suggesting that the contractor is delivering a quality product.



* Customs’ oversight of contractor testing does not provide for 

independent verification and validation (IV&V). Industry best practices 

include IV&V as a part of effective managerial oversight and control.



* The potential for ACE infrastructure to support other Department of 

Homeland Security applications could affect both cost and schedule for 

ACE. Customs is one of several agencies to be merged into the 

Department of Homeland Security, and it has proposed that the ACE 

infrastructure be used by these agencies. Because a meaningful 

understanding of the effect of the new department’s use of the ACE 

infrastructure is not yet known, there is the potential for existing 

ACE cost and schedule commitments to change.



Recommendations for Executive Action:



To ensure that Customs has the requisite capability to manage its ACE 

acquisition, we recommend that the Customs Service Commissioner 

designate strengthening CMO human capital and acquisition processes as 

priority matters. To this end, we recommend that the Commissioner 

direct the Chief Information Officer (CIO) to immediately:



* develop and implement each of the missing SEI SA-CMM practices for 

the key process areas discussed in this report, and until this is 

accomplished, report to its appropriations subcommittees quarterly on 

the progress of its efforts to do so;



* develop and implement the missing human capital management practices 

discussed in this report, and until this is accomplished, report to its 

appropriations committees quarterly on the progress of its efforts to 

do so; and:



* establish an IV&V function to assist Customs in overseeing contractor 

efforts, such as testing.



Additionally, we recommend that the Commissioner take steps, as 

appropriate in light of Customs’ merger into the Department of Homeland 

Security, to have future ACE expenditure plans specifically address any 

proposals or plans, whether tentative or approved, for extending and 

using ACE infrastructure to support other homeland security 

applications, including any impact on ACE of such proposals and plans.



Agency Comments and Our Evaluation:



In written comments on a draft of this report signed by the acting 

director, Office of Planning, Customs stated that it was pleased with 

the report, agreed with our recommendations, and described specific 

actions that are being taken to implement each. Customs’ comments are 

reprinted in appendix II.



Customs also provided additional information regarding our 

recommendation that it establish an IV&V function to assist in 

overseeing contractor efforts, such as testing. Specifically, Customs 

stated that it wanted to modify its earlier statement that it relies on 

the contractor’s reputation and maturity level as guarantors of system 

quality. Customs said that its modified position is based on a number 

of practices that it believes are elements of an integrated approach to 

IV&V. However, the practices that Customs described as elements of 

IV&V, while additive to system quality, are not independent from 

program cost and schedule pressures, and therefore do not constitute 

IV&V.



We are sending copies of this report to the Chairmen and Ranking 

Minority Members of other Senate and House committees and subcommittees 

that have authorization and oversight responsibilities for the Customs 

Service. We are also sending copies to the Secretary of Homeland 

Security, the Secretary of the Treasury, the Commissioner of the 

Customs Service, and the Director of OMB. We also will make copies 

available to others upon request. In addition, the report will be 

available at no charge on the GAO Web site at http://www.gao.gov.



Should you or your staff have any questions on matters discussed in 

this report, please contact me at (202) 512-3439. I can also be reached 

by E-mail at HiteR@gao.gov. Key contributors to this report were Mark 

Bird, Harold Brumm, Jr., Barbara Collier, Scott Farrow, Joanne Fiorino, 

Tamra Goldstein, Michael Holland, Freda Paintsil, Madhav Panwar, Karen 

Richey, Randolph Tekeley, and Aaron Thorne.



Signed by Randolph C. Hite:



Randolph C. Hite

Director, Information Technology Architecture

 and Systems Issues:



[End of section]



Appendixes:



[End of section]



Appendix I: Customs’ Fourth Automated Commercial Environment (ACE) 

Expenditure Plan:



[See PDF for image] - graphic text:



[End of figure] - graphic text:



[End of section]



Appendix II: Comments from the U.S. Customs Service:



U.S. Customs Service Memorandum:



DATE: February 13, 2003:



MEMORANDUM FOR RANDOLPH C. HITE:



U.S. GENERAL ACCOUNTING OFFICE:



FROM: Acting Director Office of Planning:



SUBJECT: General Accounting Office (GAO) Draft Report: Customs Service 

Modernization Automated Commercial Environment (ACE) Progressing, but 

Further Acquisition Management Improvements Needed:



Thank you for providing us with a copy of your draft report and the 

opportunity to discuss the issues in this report.



Overall, we are pleased with the GAO report. GAO concludes that Customs 

has met the legislative requirements imposed by Congress and continues 

to progress in addressing prior GAO recommendations. GAO also reports a 

positive assessment of the testing practices and results of the ACE 

contractor.



GAO characterizes progress in implementing a human capital strategy and 

key acquisition management practices as slow and recommends quarterly 

reporting of progress to the appropriations committees. While we 

certainly agree with the recommendations, we briefly note actions taken 

that show that Customs continues to work diligently to address these 

two critical program management dimensions.



Customs continues to implement its Modernization Strategic Human 

Capital Management Plan (HCMP) in accordance with the GAO principles. 

Priority attention was given to gaining Commissioner approval for an 

expanded Customs Modernization Office (CMO) organizational structure 
and 

filling these positions. This has been accomplished, and a talented 
cadre 

of executives is trained and functioning to oversee the Modernization 
task 

orders. In addition, our quarterly report will show that we have taken 

responsive actions across the range of GAO principles, such as the 

following:



* Integration of human capital functional staff into management teams 

through the inclusion of the OF Resources Management Group (RMG) in the 

RCMP implementation and through participation of the Director of the 

RMG and Customs Office of Human Resources Management in the 

Modernization Management Team.



* Issuance of the new “Customs Leadership Development Guide” in 
December 

2002 that clearly identifies a comprehensive leadership competency 

model, as well as a Customs Succession Management System.



* Establishment of performance plans for the CMO staff in October 2002, 

and the inclusion of Modernization objectives in the performance plans 

of an array of OIT executives and managers. *CMO recognition of staff 

for their contributions at a quarterly ceremony with both cash and non-

traditional awards.



The CMO is also continuing to implement key acquisition practices. Its 

January assessment found substantial progress in implementing key 

practices, but Customs agrees that much more progress is needed. To 

ensure that progress is made:



* CMO and other Customs managers and personnel have been assigned 

responsibilities to manage and implement the acquisition processes and 

have been trained on the process improvement goals, timeframes, and 

their roles in achieving these goals.



* The CMO continues to internally assess progress against the CMO 

process improvement goals. Additionally, Customs is contracting with 

the Software Engineering Institute (SEI) to conduct both rated and non-

rated Software Acquisition -Capability Maturity Model (SA-CMM) 

assessments to measure progress towards becoming a Level 2 
organization.



* Deficiencies identified in the process asset audit have been 
distilled 

into Corrective Action Requests (CARs) that are tracked in an OF 

database. The CMO is aggressively working to close these CARs and 

continues to move as rapidly as possible through process 

implementation.



These are examples of the progress Customs will include in its 

quarterly report.



OF also agrees to implement actions responsive to GAO’s recommendation 

relative to an independent verification and validation function (IV&V). 

However, OF wishes to modify the characterization of its IV&V function 

provided to GAO in the short timeframe we had for a response. To state 

that we rely upon the contractor’s reputation and maturity level as 

guarantors of system quality is not reflective of the range of 

practices the CMO has implemented as part of its integrated 

approach to IV&V. Such practices have included;



* Independent review of the e-Customs Partnership (eCP) requirements 

products by end users.



* Involvement of the OIT Software Development and Infrastructure 

Services Divisions in conducting IV&V of the technical design and 

implementation products.



* Engagement of the MITRE Corporation, as the CMO’s Federally Funded 

Research and Development Center (FFRDC), to provide technically 

knowledgeable and independent reviews of all critical eCP products.



These practices represent elements of an Integrated Independent 

Verification and Validation (12V2) function that enables us to 

accomplish the needs of a complex Modernization program. Customs 

recognizes that these functions could benefit from being formalized 

into a process. As a result, where appropriate, Customs will strengthen 

and formalize this process, as well as establish formal accountability 

for the 12V2 function, including formal sign-off. MITRE, as the Customs 

FFRDC, will take the lead in reviewing, documenting, and strengthening 

the 12V2 process.



If you have any questions regarding these comments, please contact Ms. 

Michele Donahue at (202) 927-0957.



Signed by Brenda B. Smith:



Brenda B. Smith:



Attachment:



U.S. Customs Service General Accounting Office (GAO) Review of Fourth 

Automated Commercial Environment (ACE) Expenditure Plan:



Recommendation 1: Develop and implement the missing human capital 

management, practices discussed in this briefing, and until this is 

accomplished, report to its (Customs) appropriations committees 

quarterly on the progress of its (Customs) efforts to do so.



Response:



Customs concurs with GAO’s recommendation to implement the actions 

identified in the Modernization Strategic Human Capital Management Plan 

(HCMP) and further develop the Customs Modernization Office (CMO) human 

capital practices. Customs also shares GAO’s desire to speed 

implementation and thereby mitigate program risks. Customs will report 

quarterly on its progress in implementing its HCMP.



Milestone Date:March 31, 2003 for CMO concurrence on Job and Core 

Competencies:



Recommendation 2: Develop and implement each of the missing Software 

Engineering Institute (SEI) Software Acquisition-Capability Maturity 

Model (SA-CMIM) practices for the key process areas discussed in this 

briefing, and until this is accomplished, report to its (Customs) 

appropriations subcommittees quarterly on the progress of its efforts 

to do so.



Response:



Customs agrees with GAO on the immediate need to develop and implement 

all of the SEI SA-CMM practices in the key process areas, and believes 

substantial progress is being made toward that end. Customs will 

provide quarterly progress reports to Customs appropriations committees 

in accordance with Congressional direction.



Milestone Date:April 2003 for updated, process asset audit:



Recommendation 3: Establish an independent verification and validation 

(IV&V) function to assist Customs in overseeing contractor efforts, 

such as testing.



Response:



The opportunity afforded by the comment period on the GAO draft report 

leads the Office of Information and Technology (OIT) to modify the 

position it took in responding to GAO’s short turnaround request of 

December 26, 2002 for a characterization of its IV&V activities. OIT 

believes it has in place a number of elements of an Integrated 

Independent Verification and Validation (12V2) function that conforms 

to contemporary practices and has provided value in terms of reduced 

risks. However, Customs recognizes that these functions could benefit 

from being formalized into a process. As a result, where appropriate, 

Customs will strengthen and formalize this process, as well as 

establish formal accountability for the 12V2 function, including formal 

sign-off. MITRE, as the Customs Federally Funded Research and 

Development Center FFRDC), will take the lead in reviewing, 

documenting, and strengthening the I V2 process.



Milestone Date:May 30, 2003 for an 12V2 strategic plan July 31, 2003 

for documented 12V2 process:



Recommendation 4: Take steps, as appropriate in light of Customs merger 

into the Department of Homeland Security, to have future ACE 

expenditure plans specifically address any proposals or plans, whether 

tentative or approved, for extending and using ACE infrastructure to 

support other homeland security applications, including any impact on 

ACE of such proposals and plans.



Response:



Customs recognizes the importance of apprising the appropriations 

committees of any changes in ACE plans that might arise from merger 

into the Department of Homeland Security. However, Customs questions 

the appropriateness of the GAO recommendation that “tentative proposals 

or plans” be included. Customs will continue its practice of providing 

the appropriations committees with expenditure plans that are well 

justified and are vetted and approved across the Executive Branch. When 

the ACE infrastructure is used or extended to support other Homeland 

Security applications, this will be noted in the appropriate 

Expenditure Plans.



Milestone Date:August 2003 for release of Expenditure Plan to the 

Congress and GAO:



FOOTNOTES



[1] For example, P. L. 108-2 appropriated such amounts as may be 

necessary under the authority and conditions provided in the applicable 

appropriations act for fiscal year 2002 for continuing projects or 

activities that were conducted in fiscal year 2002, at a rate for 

operations not exceeding the current rate, and for which 

appropriations, funds, or other authority was made available in the 

Treasury and General Government Appropriations Act, 2002, P. L. 107-67, 

115 Stat. 514, 520 (2001). These appropriations shall be available to 

the extent and in the manner which would be provided by the fiscal year 

2002 Treasury Appropriations Act. 



[2] An enterprise architecture is an institutional blueprint for 

guiding and constraining investments in business process change and 

systems.



[3] To expedite reviews of ACE expenditure plans, the Customs and 

Treasury Investment Review Boards were consolidated to form the Joint 

Capital Investment Review Board.



[4] Capability Maturity Model (CMM) is a service mark of Carnegie 

Mellon University, and CMM is registered in the U.S. Patent and 

Trademark Office. The SA-CMM identifies key process areas that are 

necessary to effectively manage software-intensive system 

acquisitions. Level 2 is the second level of the SA-CMM’s five-level 

scale; achieving this level means that an organization has the software 

acquisition rigor and discipline to repeat project successes.



GAO’s Mission:



The General Accounting Office, the investigative arm of Congress, 

exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability 

of the federal government for the American people. GAO examines the use 

of public funds; evaluates federal programs and policies; and provides 

analyses, recommendations, and other assistance to help Congress make 

informed oversight, policy, and funding decisions. GAO’s commitment to 

good government is reflected in its core values of accountability, 

integrity, and reliability.



Obtaining Copies of GAO Reports and Testimony:



The fastest and easiest way to obtain copies of GAO documents at no 

cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains 

abstracts and full-text files of current reports and testimony and an 

expanding archive of older products. The Web site features a search 

engine to help you locate documents using key words and phrases. You 

can print these documents in their entirety, including charts and other 

graphics.



Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its 

Web site daily. The list contains links to the full-text document 

files. To have GAO e-mail this list to you every afternoon, go to 

www.gao.gov and select “Subscribe to daily E-mail alert for newly 

released products” under the GAO Reports heading.



Order by Mail or Phone:



The first copy of each printed report is free. Additional copies are $2 

each. A check or money order should be made out to the Superintendent 

of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. 

Orders should be sent to:



U.S. General Accounting Office



441 G Street NW,



Room LM Washington,



D.C. 20548:



To order by Phone: 	



	Voice: (202) 512-6000:



	TDD: (202) 512-2537:



	Fax: (202) 512-6061:



To Report Fraud, Waste, and Abuse in Federal Programs:



Contact:



Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov



Automated answering system: (800) 424-5454 or (202) 512-7470:



Public Affairs:



Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.



General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.



20548: