Processes to Effectively Manage
the Development of the Custodial Accounting Project Are Improving
June 2002
Reference
Number: 2002-20-121
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
June
28, 2002
MEMORANDUM
FOR DEPUTY COMMISSIONER FOR MODERNIZATION &
CHIEF INFORMATION OFFICER
FROM: (for) Pamela J. Gardiner /s/ Scott E.
Wilson
Deputy Inspector General for
Audit
SUBJECT: Final Audit Report - Processes to
Effectively Manage the Development of the Custodial Accounting Project Are
Improving (Audit # 200120042)
This
report presents the results of our review of the development of the Custodial Accounting
Project (CAP). The overall objective of
this review was to determine whether the
Business Systems Modernization Office (BSMO) and TRW had procedures in place to
ensure that the CAP meets the needs of users and is completed reasonably within
the estimated costs and schedule.
The
CAP is one of several projects designed to correct longstanding weaknesses in
the Internal Revenue Service’s (IRS) financial accounting system. The CAP is a multi-year effort with an
estimated cost of about $61 million.
The BSMO and TRW are currently in the middle of development activities
for the CAP and expect to complete the project by May 2003.
In
summary, we found that while the BSMO estimates that the actual cost and
completion date for the CAP will significantly exceed initial estimates, the
cost and schedule slippages were primarily caused by events that occurred prior
to our audit. More importantly, the
BSMO and TRW addressed the issues that caused the cost and schedule slippages
and implemented procedures to track and monitor the project’s cost and
schedule. In addition, the project team
had established and generally followed risk management and change management
procedures. Lastly, the project team is
resolving concerns related to planning activities and users of the system were
involved in development activities for the project.
Overall,
the BSMO and TRW have implemented processes to effectively manage the
development of the CAP. Nevertheless,
our audit did reveal some areas where additional improvements could be
made. We communicated our concerns to
BSMO and TRW officials during our review, and they implemented several
corrective actions to address our concerns.
Because most corrective actions were taken during our review, we are
only recommending that the project team follow-up to ensure the corrective
action related to risk management adequately corrects the identified
weakness. The corrective actions taken
by BSMO and TRW officials during our review are discussed in detail in the body
of the report.
Management’s Response:
Management’s response was due on June
27, 2002. As of June 28, 2002,
management had not responded to the draft report.
Copies of this report are also being sent to the IRS
managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you
have questions or Scott E. Wilson, Assistant Inspector General for Audit
(Information Systems Programs) at (202) 622-8510.
Schedule
Delays and Cost Overruns Were Adequately Monitored and Controlled
The Project Team Continues to Make Improvements to Change Management Procedures
Exit Conditions From the Project Planning Phase Are Being Resolved
Users Were Involved in Development Activities
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
The Internal Revenue Service (IRS) is currently modernizing its outdated, paper-intensive tax processing systems. This multi-billion dollar effort, known as Business Systems Modernization (BSM), is projected to last up to 15 years. One of the initial BSM projects, known as the Custodial Accounting Project (CAP), is designed to correct the IRS’ longstanding custodial accounting weaknesses.
The General Accounting Office (GAO) has reported numerous material weaknesses in the IRS’ custodial financial management practices. These weaknesses include:
· Deficiencies in controls to properly manage unpaid assessments, resulting in both taxpayer burden and lost revenue to the Government.
· Deficiencies in controls over tax refunds, permitting the disbursement of improper refunds.
· Inadequacies in the financial reporting process that result in the lack of timely and reliable information for decision-making.
These weaknesses have forced the IRS to expend tremendous resources to prepare reliable financial statements and could adversely affect any decision by IRS management and/or the Congress that is based on information obtained from the custodial reporting systems. The GAO recommended that the IRS develop a solution that would resolve weaknesses in the custodial financial reporting system.
The IRS’ initial efforts to resolve the weaknesses discussed above included the Financial Reporting Release (FRR) and the Payment Information Database (PIDB) projects, which were initiated in 1997 and 1998, respectively. The IRS expended approximately $15 million on FRR and PIDB. In the fall of 1999, IRS management decided to combine these projects into a single BSM project known as the CAP. The IRS transferred many of the objectives of FRR and PIDB to the CAP and used work completed under FRR and PIDB as building blocks for the CAP.
The IRS contracted with TRW to assist in designing and developing the CAP. The CAP was designed to provide subsidiary ledgers for taxpayer accounts and collections, which directly feed the IRS’ general ledger and financial statements. Implementation of the system would improve IRS’ ability to store, analyze, and report taxpayer accounts and collections information. The ultimate goal of the CAP was to provide an automated revenue accounting and collections allocation system that is compliant with the Federal Financial Management Improvement Act of 1996.
The Business Systems Modernization Office (BSMO) initially planned to complete the CAP in four separate phases. Two phases were to be completed under the Taxpayer Accounts Subledger (TASL) application, and two phases were to be completed under the Collections Subledger (CSL) application. The BSMO planned to complete the CAP by September 2004. However, during Fiscal Year 2001, the BSMO revised several projects, and CAP was restructured to include only one phase under the TASL (Build 1). The remaining three phases would be completed as part of the Enterprise Data Warehouse (EDW) project.
The BSMO currently expects to complete
the revised CAP (TASL Build 1) by May 2003 at an estimated cost of $61
million. The BSMO and TRW are currently planning the scope and
estimating the cost of completing the remaining three phases of the original
CAP as part of the EDW. The initial
estimates were that the remaining three phases would be completed by September
2004 at a cost of almost $77 million.
The IRS has and will continue to expend a significant amount of resources to modernize its custodial financial reporting system. IRS costs to modernize its custodial financial reporting system include FRR, PIDB, CAP (TASL Build 1) and EDW (TASL Build 2 and CSL). The cost and schedule time line for these projects are shown in the table below.
Cost and Schedule Time Line for the
IRS Custodial Financial Reporting System (Cost in Millions)
Project
|
Cost
|
Time Line
|
|
FRR and PIDB |
$ 15 |
Apr 97 – Nov 1999 |
|
CAP (TASL 1) |
$ 61 |
Nov 1999 – May 2003 |
|
EDW |
$ 77 |
Mar 2002 – Sept 2004 |
|
TOTAL: |
$153 |
|
Source: The
IRS provided the cost and schedule data for the FRR and the PIDB projects. The
$15 million does not include costs incurred during Fiscal Year 1997. TASL Build 1 costs and the start date for
TASL Build 2 and CSL are based on spending plans and a recent adjustment to the
estimated cost. The cost and schedule
figures for TASL Build 2 and CSL are based on estimates contained in the CAP
business case.
Our audit was conducted at the BSMO facilities in New Carrollton, Maryland, and the TRW offices in Merrifield, Virginia. The audit was conducted between October 2001 and February 2002 in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
Project officials have adequate procedures in place to effectively measure and monitor project performance. Project officials compare actual cost and schedule results to budgeted results. Additionally, project officials routinely monitored cost and schedule variances during biweekly and monthly project meetings.
While cost and schedule delays were well controlled during our audit period, the CAP’s estimated cost and schedule have increased significantly. The majority of the cost and schedule increase occurred prior to our audit period.
The BSMO initially planned to deploy TASL Build 1 by May 2002 at an estimated cost of about $47 million. In August 2000, the BSMO and TRW revised plans developed during the CAP design phase. The revisions delayed the completion of the design phase by about 2 months.
In November 2000, the BSMO addressed the concerns of the Congress. However, the project could not deploy for another fiscal year due to the delays. As a result, the cost of completing TASL Build 1 increased by nearly $13 million and the deployment date was pushed back to March 2003.
Prior to completing our audit work, the IRS temporarily reassigned critical CAP project team members. The IRS removed critical personnel from the project in order to make them available to complete activities related to the 2002 tax filing season changes. The BSMO and TRW estimated that the temporary reassignment of project resources would delay the deployment of TASL Build 1 by nearly 2 months and increase costs by about $1.5 million. The IRS approved this variance.
Because of the above situations, the scheduled completion for the CAP could be delayed by nearly one year, and the cost of designing, developing, and deploying TASL Build 1 could increase by over $14 million to a total of $61 million. The graph below summarizes the cost increases and schedule delays related to TASL Build 1.
Schedule Delays and Cost Increases for CAP (TASL Build 1)
The graph as
removed due to its size. To see the
graph, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Over 85 percent of the cost increases and schedule delays relate to events that occurred prior to our review. Furthermore, the BSMO and TRW have addressed the deficiencies that caused the variances.
A risk is a potential event that, if it occurs, will adversely affect the project’s cost, schedule, and/or technical performance. Risk management is the process of identifying, analyzing, and tracking risks; and assessing the probability that risks will occur and their potential impacts. Effective risk management includes collecting timely information on the status of a risk and providing that information to the appropriate program/project personnel to support reassessing risk exposure and managing the risk.
To determine if risk management for the CAP was adequate, we reviewed risk meeting minutes, a sample of documented risks, and access rights to the TRW risk database.
Risk meeting minutes
The CAP Risk Management Plan
requires a meeting to review and monitor the status of all risk management
activities, track the progress of actions, and to record minutes of these
meetings. Project officials periodically
meet to review the status of each risk to determine whether risks should be
closed, remain open, or be converted to issues. These meetings are referred to as Risk Review Board
meetings. We initially found that the
Risk Review Board did not meet on a consistent basis. Additionally, the project did not adequately document the results
of Risk Review Board meetings. If
meetings are not held on a consistent basis, risks may not be monitored
timely. Without formal meeting minutes,
it is difficult to determine what risk decisions have been made. BSMO and TRW officials stated that risk
meetings were combined with other meetings early in the development phase and
were not documented.
Management Action: During our review, the Risk Review Board began to hold meetings consistently and document the results of those meetings.
Risk sample
We reviewed a judgmental sample of 6 of 24 documented CAP risks. We determined that risk management procedures are generally being followed; however, risk reduction actions could be timelier.
Risk management procedures require that risk reduction plans be prepared for certain risks, reduction actions be assigned and have a scheduled start and completion date, and reduction plans be completed or converted to an issue before the risk impact date.
We reviewed the risk reduction plans for all 6 risks and determined that risk reduction plans had been prepared and risk reduction actions were reasonable. We also noted that all risk reduction actions had been assigned with a scheduled due date.
However, we noted that:
· 5 of the 6 risks did not have a risk impact date.
· 1 risk included an impact date, but was not converted to an issue timely.
· 8 of the 14 risk reduction actions were not completed timely.
If risk reduction actions are not completed timely, the likelihood that risks will have an adverse effect on the project increases. Also, actions taken to reduce risks are often different from actions taken to handle an adverse event that has already occurred. Therefore, misclassification of an issue as a risk could lead to inappropriate actions taken by the project team.
Per CAP risk management procedures, project officials should estimate when risk reduction actions should be completed in order to avoid or prevent a risk. During our review, TRW Quality Assurance noted that project officials did not routinely complete risk reduction actions before the estimated completion date. TRW project officials also stated that the risk impact date was not a meaningful field within the risk database since some risks had multiple impact dates. Also, TRW project officials stated that the risk database was not always updated timely.
Management Action: TRW modified its database to include a field entitled “next review date.” The risk database will generate risks for review by the Risk Review Board based on the date that is inserted in the “next review date” field.
TRW risk database access
The CAP project uses a risk management database to record, monitor, and report the status of identified project risks. To ensure that documented risks are not accidentally deleted or the status is not changed without proper approval, the risk database should prevent overwriting of risk numbers and ensure that only approved officials can update the risk status field. We reviewed the TRW risk database and determined that access controls were in place to prevent accidental deletion and unapproved status changes.
To ensure that planned corrective actions have the intended effect, the CAP project officials should:
1. Review the timeliness of risk reduction actions in the future to ensure that the addition of the “next review date” field in the TRW database is effective.
Changes on software development projects can occur either when a request is received to enhance or change a baseline, or because a problem was recognized during testing. For software development projects, an effective change management process is key to success. Change management is a systematic process that ensures that changes to controlled items in each baseline are properly identified, documented, evaluated for impact, approved by an appropriate level of authority, incorporated, and verified.
During our review, we determined that
· Change management procedures were generally being followed.
· Change management improvements were made to correct identified issues.
To determine if change management procedures for the CAP were being followed, we reviewed Capability Maturity Model evaluation results and change management documentation. In May 2001, an independent evaluator commissioned by the Software Engineering Institute determined that TRW had institutionalized the key processes necessary for performing repeatable software configuration and change management activities.
We judgmentally selected samples of 3 baseline files and 18 change documents. Our review of these samples determined that:
· The project team established baselines for all 3 baseline files. The baseline files and versions were maintained in TRW’s configuration repository.
· BSMO and TRW personnel performed assessments as required to identify the impact that implementation of the changes would have on the project.
· The appropriate boards approved changes prior to implementation.
· Board approval documentation was prepared to implement changes.
·
Test teams validated solutions
implemented in response to testing problem reports.
Change management improvements
were made to correct identified issues
Based on our judgmental sample, we
determined that change documents were not always signed as required by change
management procedures. However, TRW
corrected these issues during the audit.
· The change originator’s manager did not sign two change requests and four data update requests included in the sample. Change management procedures require that the originator’s manager sign the change forms. If the originating manager does not sign change forms, the review boards could expend resources on changes that are not authorized, incomplete, and/or inaccurate. TRW management stated that in some cases the database had not been updated in the non-mandatory sign off field.
· On four data update requests, completed change directives did not contain a sign off to indicate whether the TRW Quality Assurance group had reviewed the changes. The data update procedures require that change directives be signed by the TRW Quality Assurance group to indicate that the changes have been reviewed. If the Quality Assurance group does not sign change directives, the approving board does not have independent assurance that the changes were implemented correctly. TRW management stated that this problem is due to a lag in updates to the change management database by the CAP Data Architect.
· One data update request contained a completed change directive that had not been signed by the implementer. The data update procedures require that the implementer sign change directives. If the implementer does not sign change directives, the approving board does not have assurance that changes were implemented. TRW management stated that this problem is due to a lag in updates (or maintenance) of the change management database.
Management Actions: TRW management added the missing signatures
on the change and data update requests, and updated the database. TRW management also initiated a review of
other change requests and data update requests to correct any other similar
occurrences, and initiated a change to the CAP change management database to
make the originating manager’s sign off field mandatory for completed data
update requests and change requests. In
addition, TRW management revised the Data Model Update procedures to require
the CAP Data Group to perform accuracy checks on all updates to the database.
Since TRW took corrective actions
to the issues we raised in our sample and scheduled change management process
reviews in the future, we are not making any additional recommendations.
In September 2000, IRS officials reviewed
the final CAP planning documents. The
IRS agreed that the CAP could begin development work; however, certain
conditions would need to be satisfied.
A condition is a change required for contractual acceptance of a
deliverable or work product. Unresolved
conditions to planning documents could result in incomplete business and
technical requirements, which could contribute to cost overruns and schedule
delays.
The IRS made 5 recommendations and placed 52 conditions on the CAP team at the end of the project’s planning phases. While BSM procedures for resolving conditions were not in place when the CAP team completed planning activities, the BSMO and TRW had adequate interim controls in place to ensure that conditions were resolved.
We
reviewed a judgmental sample of six conditions and one recommendation resulting
from the exit of planning activities.
We selected conditions that we believed were most significant and
reviewed the conditions to determine whether the IRS approved the resolution of
each condition.
During our review, we determined that there was one outstanding condition concerning performance measures. The BSMO set a resolve-by date of October 2000 for the open condition. However, the condition was still unresolved in March 2002, about 18 months after the resolve-by date. Also, the IRS had not required resolution of the outstanding condition in the terms of the contract with TRW.
Management
Action:
The BSMO and TRW established
procedures for reporting project performance data consistent with IRS’
reporting requirements. These new requirements
were scheduled to be implemented in May 2002.
Once implemented the outstanding condition should be resolved.
Since IRS and TRW have agreed on a schedule to resolve this
condition, we are not making any further recommendations.
Adequate user involvement in the development of a new system helps ensure that the system delivered will meet the requirements of the user. We found that IRS users routinely participated in development activities. These activities included meetings to review and approve system requirements and changes to requirements previously established during the planning phases. Users also actively participated in project and program meetings designed to discuss the status of the project. During those meetings, users and project managers discussed issues that could impact the cost, schedule, and/or quality of the project deliverables.
While we found adequate evidence of user involvement, project officials did not routinely document the results of several meetings that occurred during the first few months of the development phase.
Management Action: During our review, project officials began to routinely document the results of meetings.
Since TRW has taken corrective action to document meetings, we are not making any additional recommendations.
Appendix I
Detailed Objective, Scope, and Methodology
Our overall objective was to determine whether the Business Systems Modernization Office (BSMO) and TRW had adequate procedures in place to ensure that the Custodial Accounting Project (CAP) meets the needs of users and is completed reasonably within the estimated cost and schedule. To achieve this objective, we performed the following audit tests.
I.
Determined whether
the BSMO had adequate controls in place to ensure that all conditions related
to the milestone (MS) 3 exit were or will be resolved, reviewed, and approved
by the Internal Revenue Service (IRS).
A.
Documented
procedures and identified controls in place to ensure that conditions for the
MS 3 exit were adequately resolved.
B.
Determined
whether resolution of an outstanding condition was included in the terms and
conditions of the contract for the next fiscal year.
C.
Determined
whether a schedule existed for the resolution of the outstanding condition.
D.
Determined
whether the BSMO approved the resolution of each item based on a judgmental
sample. NOTE: The IRS made 5 recommendations and placed 52
conditions on the CAP team at the end of the project’s planning phases. We selected a judgmental sample of 6
conditions and 1 recommendation so that we could concentrate on the conditions
that we considered most significant.
II.
Determined whether the BSMO and TRW had adequate
controls in place to ensure that stakeholders were involved and their concerns
were resolved during the development, testing, and integration activities.
A.
Determined whether users were adequately involved in
development activities.
1.
Reviewed the minutes and
supporting documentation from key project meetings held during our period of
review.
2.
Determined whether risks and issues related to a lack
of user involvement had been reported and adequately resolved.
B.
Determined whether change management procedures and
processes were in place.
1.
Determined whether change management activities were
properly coordinated between the BSMO, PRIME, and TRW.
2.
Determined whether controls were in place to ensure
that change management procedures were being followed during the life cycle of
the project.
III. Determined whether the BSMO and TRW were meeting project cost and schedule goals and identified the cause and effect of any significant cost variances or schedule delays.
A. Attempted
to collect and analyze on-line project cost and schedule data. NOTE: Data was not available due to the fact that TRW was not reporting
the requested cost and schedule data to the BSMO.
B. Constructed
a timeline for the CAP cost and schedule.
C. Interviewed
TRW and BSMO officials to identify the cause and effect of any significant cost
variances.
D.
Interviewed
TRW and BSMO officials to identify the cause and effect of any significant
schedule delays.
IV.
Determined
whether the BSMO and TRW had adequate risk management controls in place.
A.
Determined
the cause and effect of the lack of warning flags (trigger points) to identify
potential risks.
B.
Determined
the cause and effect of a meeting not being held at the beginning of MS 4
activities to identify risks in accordance with PRIME Risk Procedures.
C.
Determined
who had access to make changes to the TRW risk database.
D.
Determined
if the risk tracking identification number field was a primary key field in the
TRW risk database.
E.
Determined
if the Risk Review Board met on a consistent basis.
F.
Determined,
based on a judgmental sample of project risks, if:
1. All open risks had mitigation plans.
2. The plans effectively addressed the risks.
3. Mitigation actions were tracked and on schedule.
4. The risk repository contained relevant, complete, accurate, and timely information. NOTE: At the time of our sample, there were 24 project risks. We judgmentally selected 6 project risks for review that we considered the most significant.
G.
Determined
if risks were converted to issues timely.
Appendix II
Major Contributors to This Report
Scott Wilson, Assistant Inspector
General for Audit (Information Systems Programs)
Scott Macfarlane, Director
Troy
Paterson, Audit Manager
James
Douglas, Senior Auditor
Wallace
Sims, Senior Auditor
George
Franklin, Auditor
Sylvia Sloan-Copeland, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Chief Financial Officer N:CFO
Associate Commissioner, Business Systems Modernization M:B
Deputy Associate Commissioner, Program Management M:B:PM
Director, Internal Management Modernization M:B:IM
Director, Configuration Management M:B:SI:CM
Chief Counsel CC
National Taxpayer Advocate TA
Director,
Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls
N:CFO:F:M
Audit Liaison:
Associate Commissioner, Business Systems Modernization M:B