Electronic Intermodal Supply Chain Manifest
Freight ITS Operational Test Evaluation
Final Report
PDF Version 1.26MB
Notice This document is disseminated under the sponsorship Mark Jensen |
Field Operational Test Evaluation
Final Report
FREIGHT & ITS WEB RESOURCES
USDOT ITS Joint Program Office:
http://www.its.dot.gov
USDOT Office of Intermodalism
http://www.dot.gov/intermodal/freight.html
FHWA Office of Freight Management
http://ops.fhwa.dot.gov/freight/
ITS Cooperative Deployment Network (ICDN):
http://www.nawgits.com/jpo/icdn.html
ITS Electronic Document Library (EDL):
http://www.its.fhwa.dot.gov/cyberdocs/welcome.htm
USDOT ITS Joint Program Office
USDOT Office of Intermodalism (OST)
FHWA Office of Freight Management and Operations
U.S. Department of Transportation
Federal Highway Administration Operations Unit
400 7th Street, S.W., HOP
Washington, DC 20590
Toll-Free "Help Line" (866) 367-7487
1. Report No. FHWA-OP-02-XXX |
2. Government Accession No. |
3. Recipient’s Catalog No. |
||
4. Title and Subtitle Electronic Intermodal Supply Chain Manifest Field |
5. Report Date December 2002 |
|||
6. Performing Organization Code |
||||
7. Authors Mark Jensen (SAIC), Mike Williamson (Cambridge Systematics), Robert Sanchez (SAIC), Carol Mitchell (SAIC) |
8. Performing Organization Report No. |
|||
9. Performing Organization Name and Address Science Applications International Corporation 8301 Greensboro Drive M/S E-7-6 McLean, VA 22102 |
10. Work Unit No. (TRAIS) 11. Contract or Grant No. DTFH61-96-C-00098; Task 9811 |
|||
12. Sponsoring Agency Name and Address United States Department of Transportation ITS Joint Program Office, HVH-1 400 7th Street SW Washington, DC 20590 |
13. Type of Report and Period Covered 14. Sponsoring Agency Code HOIT-1 |
|||
15. Supplementary Notes Dr. Joseph I. Peters (COTR) |
||||
16. Abstract This report presents the results of a 2.5 year freight ITS evaluation of an air cargo security and logistics system which was deployed at O’Hare and JFK international airports. In September 1999, the Federal Highway Administration (FHWA) and the Federal Aviation Administration (FAA) jointly funded a field operational test to develop an electronic supply chain manifest system for air cargo. The primary objectives of this test were to increase the security of air cargo operations, while providing shippers and carriers with improved efficiencies in their operations. Several key conclusions of this evaluation report are summarized as follows:
|
||||
Key Words Intelligent Transportation Systems, ITS, Freight, Intermodal, Electronic Chain Supply Manifest, Biometrics, Smart Cards, Field Operational Test, Evaluation |
18. Distribution Statement No restrictions. This document is available to the public from: The National Technical Information Service, Springfield, VA 22161. |
|||
19. Security Classif. (of this report) Unclassified |
20. Security Classif. (of this page) Unclassified |
21.No of Pages 82 |
22. Price N/A |
|
Form DOT F 1700.7 (8-72) Reproduction of completed page authorized.
TABLE OF CONTENTS
LIST OF TABLES
LIST OF FIGURES
ABBREVIATIONS
EXECUTIVE SUMMARY
1. INTRODUCTION
2. ESCM SYSTEM Deployment Overview
3. Technical Approach
4. Operational Impact of The ESCM System
4.1 Description of Participants’ Use of ESCM System
4.2 Comparison of Manual and Automated Systems
4.3 Customer Satisfaction and Institutional Challenges
5. Overview of the Technical Effectiveness of the ESCM System
5.1 Integrated air cargo system OPERATIONAL PERFORMANCE
5.1.1 Positive Association of Driver with Electronic Manifest
5.2 ESCM Computer System TECHNICAL Performance
5.3 Electronic Manifest Technical Performance
5.4 Biometric Fingerprint Identification technical Performance
6. Conclusions and Recommendations
REFERENCES
APPENDIX
LIST OF TABLES
Table ES-1. ESCM Operational Test Participants
Table ES-2. Efficiency Improvements for Trucking Companies
Table 4-1. ESCM System vs. Manual Process Times per Shipment for Manufacturers
Table 4-2. ESCM System vs. Manual Process Times per Shipment for Trucking Companies
Table 4-3. ESCM System vs. Manual Process Times per Shipment for Airlines
Table 5-1. Total Manifest/Cargo Cycle Time
Table 5-2. Physical Cargo Cycle Times
LIST OF FIGURES
Figure ES-1. Cargo Operations at O’Hare Experienced Rapid Growth in 1990’s
Figure ES-2. ESCM System Design Concept
Figure ES-3. ESCM System Processes Overview
Figure 2-1. Biometric Fingerprint Identification Technology.
Figure 2-2. ESCM System Smart Card Reader and Smart Cards.
Figure 2-3. Data Entry of ESCM System Manifest at Manufacturer Location.
Figure 2-4. Centralized ESCM System Server Enabling Internet Access.
Figure 2-5. Shipment Acceptance and Processing at Participating Airline.
Figure 2-6. ESCM System Login Screen for All Authorized Participants.
Figure 2-7. ESCM System Manifest Search Screen after Manufacturer Login.
Figure 2-8. ESCM System Manifest Maintenance Screen Before a Manufacturer Creates an Air Waybill.
Figure 2-9. ESCM System Manifest Maintenance Screen After a Manufacturer Creates an Air Waybill.
Figure 2-10. Truck Driver’s Screen After Login (Air Waybill #14 Created).
Figure 2-11. Air Cargo Clerk’s Screen View After Login.
Figure 2-12. Air Cargo Clerk’s Screen View After Air Waybill Number Selection.
Figure 2-13. ESCM System Logout Screen for All Participants.
Figure 3-1. Phase II Pre-Deployment Data Collection.
Figure 3-2. Phase II Deployment Data Collection.
Figure 5-1. Frequency of ESCM Transactions
Figure 5-2. Frequency of Elapsed Times for Manifest Creation to Release
Figure 5-3. Frequency of ESCM System Errors
Figure 5-5. Second-Try Outcome
Figure 5-6. False Acceptance Rate as a Function of Threshold
AMS |
Automated Manifest System |
ATIS |
Advanced Traveler Information Systems |
COTR |
Contracting Officers Technical Representative |
CVO |
Commercial Vehicle Operations |
DOT |
US Department of Transportation |
ESCM |
Electronic Supply Chain Manifest |
FAA |
Federal Aviation Administration |
FHWA |
Federal Highway Administration |
FOT |
Field Operational Test |
ITS |
Intelligent Transportation Systems |
JPO |
Joint Program Office |
MOE |
Measures of Effectiveness |
SAIC |
Science Applications International Corporation |
EXECUTIVE SUMMARY
Introduction
Air transport is the fastest growing freight transportation mode today with both volume and revenue projected to double by 2006. Although airfreight comprises only 1% of total freight moved worldwide by weight, it accounts fort 38% of total freight by value. Airfreight can therefore be classified as high value/low density products with a heightened requirement for timely delivery.
Figure ES-1. Cargo Operations at O’Hare Experienced Rapid Growth in the 1990’s
Security concerns and time pressures to deliver air cargo more quickly than ever are focusing attention on the ground-to-air intermodal link. Truck-to-air cargo movements grew at a rapid rate in the 1990’s, yet this logistics link today is still largely maintained by industry and regulated by government using paper- and telephone-based information exchanges. This situation is exacerbated by the fact that much of the air cargo is transported on passenger planes, a major safety concern that is now even more critical following the events of September 11, 2001. To support the needs of the marketplace and to ensure the security of air passengers and cargo shipments, new tools and processes are being encouraged by the U.S. Department of Transportation.
The ESCM Field Operational Test
In support of the Office of the Secretary of Transportation, the Federal Aviation Administration (FAA), the FHWA Office of Freight Management and Operations, and others, over the past three years the ATA Foundation led the development of a pubic-private partnership to develop and test the first operational electronic air cargo manifest and security system in the United States. The goal of this test was to demonstrate the improvements in efficiency and security of an Internet-based electronic air cargo security system compared to traditional processes and paper-based manifest systems. The operational test was conducted in conjunction with manufacturing, trucking, and airline participants in the Chicago-O’Hare International Airport and New York City-JFK International Airport service areas. A summary of test participants and their roles is presented in Table ES-1.
Table ES-1. ESCM Operational Test Participants
|
ROLE |
||||||||
Project Management |
System Development |
System Deployment |
Participant Recruitment/ |
System Participant |
Evaluation |
Project Oversight |
|||
Public Sector Partners: |
|||||||||
USDOT & FHWA |
● |
||||||||
Federal Aviation Administration |
● |
||||||||
State of Illinois |
● |
||||||||
NY Department of Transportation |
● |
||||||||
Chicago Dept. of Aviation (O’Hare Airport) |
● |
● |
|||||||
Port of NY-NJ (JFK Airport) |
● |
● |
|||||||
Private Sector Partners: |
ATA Foundation |
● |
● |
● |
● |
SecurCom (System Engineer) |
● |
● |
● |
Identix (Biometrics) |
● |
● |
Manufacturers (2 @ O’Hare) |
● |
MotorCarriers (4 @ O’Hare, 4 @ JFK ) |
● |
Air Cargo Carriers (4 @ O'Hare, 5 @ JFK) |
● |
SAIC & Cambridge Systematics |
● |
This test builds on an earlier FAA test at Chicago’s O’Hare Airport. That test used biometric “smart cards” to confirm the identity of the driver using a stored thumbprint and to provide information about the seal on the cargo the driver was transporting. The purpose was to improve security of freight movement into and out of the airport. Building on this experience, this operational test deployed and tested a secure electronic manifest system -- utilizing leading edge security technologies including encrypted internet transactions, 8K smart cards and biometric fingerprint readers. An overview of the system design concept is presented in Figure ES-2.
Figure ES-2. ESCM System Design Concept
This system is designed to allow only authorized users to enter and monitor cargo movement and access valuable shipment information at specific points in time and in the logistics process. However, this system also provides for substantially more freight management functions, all of which differ depending on whether the user is a manufacturer, a motor carrier, and an air freight consolidator or airline. For example this system his could allow an airline advance notice of incoming freight and reduce consolidation time in planning specific flight loads. In the unfortunate case of an air transport incident, the cataloging of electronic manifests can provide immediate access cargo content records by public sector agencies to aid in incident reconstruction.
This system allows manufacturers to send cargo information real-time along the distribution channel in advance of pick-ups and deliveries. And the electronic manifest offers a secured identity process through biometric imprints (fingerprint recognition) in addition to the reduction in information errors due to electronic processing at all times. An overview of the system processes utilized in this test is presented in Figure ES-3.
Figure ES-3. ESCM System Processes Overview
Evaluation Overview
An Evaluation Team led by SAIC was selected in January 2000 by USDOT to develop and implement an independent evaluation of the ESCM Operational Test. The objective of this evaluation was to identify goals and “lessons learned” with respect to implementing the technologies. The evaluation focused on the four areas:
- Intermodal Freight System Operations
- Technology Effectiveness
- Institutional Issues
- Participant Satisfaction
A key component of the evaluation approach was the teamwork and coordination that existed between the ATA Foundation (the deployers) and the Evaluation Team. This was a critical component to the evaluation as repeated on-site collections had the potential for unnecessary disruption of participants’ operating routines.
There were two groups of objectives developed for the evaluation: operational and technical. The operational objectives addressed the participants' experiences with the ESCM system, and the technical objectives addressed the ESCM system functionality. The objectives of the evaluation consisted of the following:
- To measure the impact of the ESCM system on overall freight system operations (including shipment security) of the participants, to document the participants’ level of satisfaction with the performance of the system, and identify key institutional challenges associated with the deployment.
- To measure the technical performance of the ESCM technologies in their operating environment, in terms of both the integrated air cargo system operational performance and the individual technology technical performance for electronic manifests, smart cards and biometric technologies.
Participant Expected Benefits
During pre-ESCM system deployment interviews, participants were specifically asked by the Evaluation Team about the type of results they hoped the test would provide. Their responses are aggregated into the following three categories of operational impacts:
- Efficiency. One major opportunity for achieving this was the ability of the ESCM system to automate the shipment transfer procedures (biometric/Smart Card authorization vs. manual duplication/photocopying of all paperwork).
- Security. A secure, closed system that ensures an individual entity is responsible for a load at all times, coupled with the ability to track the ownership of shipments was viewed as an enormously valuable tool by all participants
- Regulatory Compliance. The Participants were hopeful that a system of this type would be a step in the right direction for streamlining and improving their ability to meet what they currently see as cumbersome paper-based FAA requirements.
System Usage
In most cases, although the system was used, it was not used as fully as it could have been – most participants did not successfully integrate the ESCM system into their daily operations for all transactions. This was primarily the result of the duplicate processing (they had to still use the FAA-mandated paper-based system), limited staff resources, and working with intermodal partners who were not part of the test..
Efficiency Analysis
The Evaluation Team conducted a comparison of existing manual security functions versus the operational test systems automated processes for manufacturers, trucking companies and airlines. This comparison showed significant time savings in every category measured. For example, Table ES-2 shows the time savings calculated for trucking companies. These savings are substantial when multiplied by total number of shipments per year for a typical trucking company.
Table ES-2. Efficiency Improvements for Trucking Companies
Trucking Company Activity |
Percent Reduction in Time |
Order acceptance over the phone and data input |
100% |
Load acceptance at manufacturer |
93% |
Input to create master manifest |
66% |
Reproduction of manifests |
100% |
Paperwork error correction |
100% |
Contact airline and arrange shipping |
100% |
Delivery to airlines |
94% |
System Technology Effectiveness
The evaluation of the system operational performance showed that the integrated air cargo system (consisting of the electronic manifest, Smart Cards, and biometric identification technologies) completed the required tasks expeditiously. ESCM system transactions were recorded by the server and errors were captured in a timely manner. Although the limited number of project participants resulted in only eight shipments completing the entire electronic manifest cycle (create, release, pick up, delivery), the ESCM server did capture the transactions for subsequent analyses.
Over a 6 month test period, the ESCM computer system and network was found to be reliable. Reliability was measured in terms of system up time, unexpected errors, availability of system resources (and not working near capacity), and lack of unauthorized access attempts. Five metrics of system availability were examined: all indicated that the computer resources (computer processor, network activity, disk space, and physical memory) were working below capacity and were readily available.
Participant Satisfaction and Institutional Challenges
The evaluation found that heavy users of the system reported continued interest and commitment to the system throughout the test. These participants had operations staff that found the system easy to use and they remained committed in many cases in hopes that the system would be expanded and further deployed. They reported that once the ESCM became the primary system for their and their supply chain partners operations the true benefits could be measured. This being said, a major concern documented by the Evaluation Team with this FOT was the number of participants that dropped out of the test over its duration. This occurred because they could no longer afford to operate two duplicate systems. While the ESCM was designed to replace existing FAA regulatory paper-based processes, it did not replace these processes as part of this test -- it simply added a new set of procedures that duplicated an existing processes. Staffing constraints and profitability became the ultimate priority of these companies, as should be expected of for-profit organizations. Given the serious nature of air cargo security, and the fact that not all supply chain partners of these companies were involved in the test, it would have been difficult to eliminate the existing and regulated process. However, without doing so, the test was necessarily restricted to partial use of the system as resources allows.
Conclusions
The time savings estimates developed in this report show the potential for substantial industry time savings by the implementation of this system. Many of these savings were estimated to come from replacement of manual processes with automated ones like notification of load pickup or acceptance.
Few shipments were processed through the entire ESCM system. Over the defined test period, only eight shipments moved completely through the system from manufacturer to airline using the ESCM system. This was largely due to participant staff resource problems and interactions with non-test logistics partners.
Participants have reported overall satisfaction with the ESCM system. Some participants felt the ESCM system would be significantly more useful with wider deployment to more of their supply chain partners.
Significant outreach activities were conducted to build support for the ESCM system. Nearly 100 ESCM system demonstrations were provided for participants.
Drivers were very interested and supportive of a system that provides a single Smart Card that replaces their commercial driver’s license (CDL) and paper work. One Smart Card that eliminates the need for cumbersome paperwork and duplicate copies of their CDLs was welcomed.
Recruitment of participants was an ongoing challenge. Identifying complete supply chains was difficult; starting with motor carriers proved the best way to build supply chains; getting the managers to personally “buy in” to the test was critical.
Several participants used the system for 18 months as opposed to the originally planned 6-month test period. “When will this system become fully operational so that I can discontinue the manual processes,” was a common theme.
The test was successfully developed and deployed within the confines of a complicated organizational structure. This created the need for a high level of coordination by ATA Foundation and SecurCom staff to ensure that all stakeholders were provided with the appropriate level of service, while not negatively impacting the operational participants.
Recommendations
The following highlights some of the most important recommendations of this evaluation final report which have been developed for consideration by FHWA and FAA as these technologies are expanded to LAX and Toronto International airports:
Acquire a partial waiver from FAA to allow full testing of the ESCM system to document actual changes/improvements in operations based on full deployment. This would require participants to fully embrace the system and rely on it. This would take the test to the next step as lessons learned would be on actual system use as opposed to partial testing as time permits.
Expand the current follow-on phase of this operational test to include the second half of the freight movement. This FOT covered manufacturer to motor carrier to airline. This should be expanded to include final delivery of the shipment (airline to motor carrier to customer).
Identify an appropriate participant (high volumes and multiple partners) with a legacy system and develop an interface between it and the ESCM system to test the ease of customization. Of the participants with automated systems, they were frustrated the inability to integrate their electronic systems.
Develop an organizational structure that streamlines access to funding and facilitates decision making from an operations perspective. The Phase II test had multiple agencies playing lead roles and multiple funding sources that impacted ease of implementation early in the development and deployment.
Consider developing an incentives program that stimulates broader system testing. If regulatory agencies are unable to provide waivers during FOTs, then reimbursing participants for time spent on test activities to stimulate system testing should be considered – the could be done by tax incentives, staff incentives, etc.
1. INTRODUCTION
1.1 Background/Purpose of Field Operational Test
In September 1999, the Federal Highway Administration (FHWA) and the Federal Aviation Administration (FAA) jointly funded a field operational test to develop an electronic supply chain manifest system for air cargo. Improving the existing system is necessary as air transport is the fastest growing freight transportation mode today with both volume and revenue projected to double by 2006[1]. Although airfreight comprises only 1 percent of total freight moved worldwide by weight, it accounts for 38 percent of total freight by value[2].
The primary objectives of this test were to increase the security of air cargo operations, while providing shippers and carriers with improved efficiencies in their operations. The development of the electronic supply chain manifest (ESCM) system was the second phase of the project. Phase I specifically focused on establishing a Smart Card/ biometric-based driver security system to improve the efficiency of truck access to airports for the delivery of air cargo by automating the transfer process, which historically consisted of manually photocopying the driver’s license for each bill of lading. The Phase I system allowed the driver to communicate his/her identity via a personalized Smart Card and biometric reading. In addition, each truck trailer was sealed at the point of origin and the seal number was loaded onto the Smart Card.
Phase II consisted of the development of an Internet-based manifest system that was access-restricted/managed using the biometric and Smart Card technology developed in Phase I. The use of trailer seals was not continued in Phase II. Phase II focused on testing the manifest system and its access controls across three node supply chains (manufacturers, motor carriers, and airlines). This test had many participants and was led by the American Trucking Association Foundation (ATA Foundation). This test was deployed initially at Chicago’s O’Hare International Airport and was expanded to New York City’s J.F.K. International Airport. Initially, there were plans to acquire a waiver from the FAA to allow this system to be the official system for participants over the finite period of the test. However, this waiver was not enacted as part of the test, therefore, the test required dual systems maintenance for the old and new systems employed.
In support of the United States Department of Transportation’s (USDOT) intermodal Intelligent Transportation Systems (ITS) program, an Evaluation Team lead by Science Applications International Corporation (SAIC), under the direction of the USDOT Joint Program Office (JPO), was selected in January 2000 to develop and implement an evaluation of the Electronic Intermodal Supply Chain Manifest ITS Field Operational Test (FOT). The ultimate goal of this evaluation, as defined by the JPO, was to identify “lessons learned” with respect to implementing intermodal ITS technologies for four study areas: system operational processes, technology applications, institutional agreements, and user acceptance. The Evaluation Team has worked closely with the deployers over the course of this FOT to develop a comprehensive evaluation.
In addition to the independent evaluation described in this document, the ATA Foundation, as the system developer and deployer, conducted its own “self-evaluation” activities in support of FAA and State of Illinois requirements. It was agreed upon by all parties early in the process that all data collected in support of evaluation activities would be coordinated and shared. This has occurred and is described in more detail throughout this document.
1.2 Organization of the Final Report
This draft final report has been organized to describe the activities undertaken by the deployers and the Evaluation Team, and to document lessons learned and recommendations for future research. The remaining sections and content of this document is described as follows:
Section 2. ESCM System Deployment Overview. Section 2 provides a detailed description of the ESCM system FOT, including a summary of Phase I and an overview of the Phase III expansion activities already underway.
Section 3. Technical Approach. Section 3 provides a review of the methodology used to conduct the evaluation activities. This includes a review of key objectives and the data collection and analysis efforts.
Section 4. Operational Impact of System. Section 4 defines the participants' use of the ESCM system, compares the available manual and automated data, and identifies key customer satisfaction and institutional challenges.
Section 5. Overview of the Technical Effectiveness of the System. Section 5 describes the technical performance of the ESCM system at two levels – integrated system operational performance and major technology technical performance.
Section 6. Findings, Conclusions, and Recommendations. Section 6 highlights the findings and conclusions of the evaluation and provides recommendations for consideration by FHWA and FAA.
2. ESCM SYSTEM Deployment Overview
The ESCM system evaluated by the Evaluation Team is the second phase of a three-phase project. The first phase was completed prior to the Evaluation Team's involvement, and the third phase currently is ongoing and is not part of this evaluation. The following provides descriptions of each phase.
2.1 Highlights from Phase I
Phase I of this project was sponsored by the FAA and the State of Illinois, and consisted of establishing a Smart Card/ biometric-based driver security system to improve the efficiency of truck access to airports for the delivery of air cargo. This was accomplished by automating the transfer process, which historically consisted of manually photocopying the driver’s license for each bill of lading. The Phase I system allowed the driver to communicate his/her identity via a personalized Smart Card and biometric reading. In addition, each truck trailer was sealed at the point of origin and the seal number was loaded onto the Smart Card. This phase also allowed the FAA to review its “known shipper” regulations and protocols.
Phase I involved over 500 drivers and 11 airlines and/or freight forwarders. Results from this project indicated that biometric identification and Smart Card systems can provide tangible improvements in air cargo security and greater efficiencies for motor carrier operational processing. Further results determined that comprehensive training and communication programs must be developed to ensure user acceptance. Additionally, it was determined that technology upgrades must be performed regularly to ensure high system performance.
In Phase I, a computer equipped with a Smart Card reader and software was installed at each participating trucking company. A participating driver placed a numbered seal on the trailer door and entered the seal number and cargo information into the computer prior to delivery at the airport. The Smart Card was created with the seal and driver information on it. The driver then proceeded to the air cargo loading area, where the air cargo attendant scanned and read the card along with the driver’s thumbprint. Based on the card information, the attendant’s computer then retrieved a picture of the driver, driver information, and seal number. The computer then displayed an approval, denial, or request for additional information. Based on an approval, the attendant checked the seal on the container and allowed the driver to proceed to the unloading area.
2.2 Overview of Phase II
Phase II built upon the Phase I technologies. Using the truck driver/cargo access system developed in Phase I, Phase II integrated a newly developed biometrically secured electronic manifest. This system was accessed through the Internet, which provided all supply chain participants access to the load information in their respective supply chains.
The shipment was first originated in the system when the manufacturer entered the load information in the ESCM. This data entry process was secured by Smart Card and biometric technology. Once the manifest was complete, it was uploaded to the manifest database on the server. At this time, emails were automatically sent to the downstream supply chain partners. At each subsequent transfer, emails were sent to all three supply chain participants. This communicated to the motor carrier that the shipment was ready for pickup and to the airline that the shipment could be tracked. When the truck driver arrived to pick up the load, he/she accepted the load electronically via Smart Card and biometric confirmation. The shipment status was electronically transferred from the shipper to the motor carrier via this process. The shipment was then transported to the airline. When the driver entered the air cargo office, both the clerk’s and the driver’s identities were confirmed with Smart Cards and biometric scan. This completed the supply chain as the shipment was electronically transferred to the airline and the shipment was accepted.
Figures 2-1 through 2-13 illustrate the ESCM process. Figure 2-1shows the biometric and Smart Card technologies used to secure the system. At right, the fingerprint is scanned and converted to a “minutiae” template as shown below. The pattern displays a mathematical representation of the fingerprint, which can be easily stored and retrieved in identification information systems. This method of biometric fingerprint identification and storage is the most commonly used format.
Figure 2-1. Biometric Fingerprint Identification Technology.
Figures 2-2 through 2-5 show various components of the participant process and equipment installations, ranging from the Smart Card reader and Smart Cards that contain shipment information, to data entry of the manifest at the manufacturer’s location, to the centralized ESCM system main server that allows authorized users access for shipment tracking via the Internet, to the final shipment acceptance and processing with the destination airline.
Figure 2-2. ESCM System Smart Card Reader and Smart Cards.
Figure 2-3. Data Entry of ESCM System Manifest at Manufacturer Location.
Figure 2-4. Centralized ESCM System Server Enabling Internet Access.
Figure 2-5. Shipment Acceptance and Processing at Participating Airlines.
Figures 2-6 through 2-13 provide screen shots of the ESCM system as used by the authorized supply chain participants.
Figure 2-6. ESCM System Login Screen for All Authorized Participants.
Figure 2-7. ESCM System Manifest Search Screen after Manufacturer Login.
Figure 2-8. ESCM System Manifest Maintenance Screen Before a
Manufacturer Creates an Air Waybill.
Figure 2-9. ESCM System Manifest Maintenance Screen After a
Manufacturer Creates an Air Waybill.
Figure 2-10. Truck Driver’s Screen After Login (Air Waybill #14 Created).
Figure 2-11. Air Cargo Clerk’s Screen View After Login.
Figure 2-12. Air Cargo Clerk’s Screen View After Air Waybill Number Selection.
Figure 2-13. ESCM System Logout Screen for All Participants.
The ESCM system provided continuous tracking capabilities, provided one automated paperless shipment history, and accounted for shipment responsibility throughout the supply chain. The system required that all three participants use the ESCM system. There were two Phase I components that were not included in Phase II. First, the use of seals was eliminated in Phase II due to the limited success of using seals during Phase I in a multi-stop truck loading environment that is common in air-to-truck cargo operations. And second, the drivers’ photographs were not loaded onto the Smart Cards electronically due to concerns expressed by drivers during the Phase I test.
There were three main groups of participants involved in this FOT:
- The first group consisted of the participants responsible for developing and deploying the systems (i.e., ATA Foundation, SecurCom).
- The second group consisted of the regulatory agencies responsible for reviewing and approving the FOT components and facilitating the deployment process (i.e., FHWA, FAA).
- The third group consisted of the operational participants who were responsible for incorporating the FOT into their operations (manufacturers, motor carriers, airlines/air freight forwarders).
In addition, there were representatives from other entities that provided support and input over the course of this FOT (i.e., O’Hare International Airport, New York and New Jersey Port Authority, Chicago Area Transportation Study). The preceding list of participants/ representatives is meant to illustrate the diversity of the group – it does not accurately reflect all participants (public or private).
In addition to the operational test, it was anticipated that the data derived from this system could be used to provide real-time traffic information to regional Advanced Traveler Information Systems (ATIS) in the Chicago area, such as the Gary-Chicago-Milwaukee Priority Corridor project. This did not occur as part of this FOT; however, it could be an opportunity for development in future applications of these technologies.
There were expectations for a Phase III project that would expand this test to include an automatic cargo profiling function to assist regulatory agencies in identifying dangerous goods and thus “red-flag” specific loads for additional inspection. Although there was a Phase III in operation during the development of this report, this phase is focusing on geographic expansion, including expanding to an airport outside the United States. A description of Phase III is provided in the next section.
2.3 Description of Phase III
Phase III of the ESCM system was originally envisioned to be an expansion of the Phase II system to provide for additional cargo screening capabilities. Specifically, it was to provide cargo information to the regulatory agencies making decisions on which shipments should receive additional screening, such as hazardous materials or other shipments that were categorized as high risk. Although the system has been expanded beyond the parameters of the Phase II test, it was not expanded in this manner.
The current Phase III expansion effort consists of geographically expanding the system to cover two additional airports (four in total): the Los Angeles International Airport and Toronto International Airport. There were two primary objectives driving this expansion. The first was to continue testing the Phase II system in a broader market place. The second was to test the system in the international arena. As part of this second objective, discussions have taken place with the U.S. Customs Service to investigate integration of the ESCM system with the AMS system. Data are being collected currently for analysis by the deployer, the ATA Foundation.
3. Technical Approach
The technical approach to the evaluation focused on two elements: proof of concept that the technologies functioned as designed; and the impact the ESCM system had on the operations of the participants. The impact on operations evaluation addressed quantitative differences between the existing conditions and the automated conditions provided by the system. In addition, the impact evaluation qualitatively measured customer satisfaction and identified institutional challenges. From a systems perspective, the technical performance of the system also was evaluated.
A key component of the evaluation approach was the teamwork and coordination that existed between the ATA Foundation (the deployers) and the Evaluation Team. It was determined at the start of Phase II that there would be an enormous amount of duplication between these two teams regarding data collection. In an effort to mediate the potential negative impact this would have had on the industry participants, it was agreed that the two would work closely together to collect the data, and then each would conduct their own analyses. This was a critical component to the evaluation as repeated on-site collections had the potential for unnecessary disruption of participants’ operating routines. To have doubled this intrusion by conducting nonparallel duplicate data collection activities would have resulted in an extremely negative situation. In addition, this gave the deployers and the evaluators the opportunity to work together and benefit from each others' perspectives.
The following provides a summary of the activities undertaken to conduct the evaluation. For a more detailed description of these activities, readers are referred to the following two documents:
Electronic Intermodal Supply Chain Manifest ITS Field Operational Test Evaluation Plan, U.S. Department of Transportation and Federal Highway Administration by Science Applications International Corporation, July 12, 2000.
Electronic Intermodal Supply Chain Manifest ITS Field Operational Test Evaluation Detailed Test Plans, U.S. Department of Transportation and Federal Highway Administration by Science Applications International Corporation, June 8, 2001.
3.1 Objectives
- To measure the impact of the ESCM system on overall freight system operations (including shipment security) of the participants, to document the participants’ level of satisfaction with the performance of the system, and identify key institutional challenges associated with the deployment.
- To measure the technical performance of the ESCM technologies in their operating environment, in terms of both the integrated air cargo system operational performance and the individual technology technical performance for electronic manifests, smart cards and biometric technologies.
3.2 Approach
The overall freight system impacts were evaluated by identifying the changes in operations and information flow between the pre-test conditions and test conditions associated with the ESCM system deployment. This required the Evaluation Team to gain a more complete understanding regarding the operational characteristics of the participants, documenting their experiences with the system, identifying their level of satisfaction with the system, and documenting the deployment/institutional challenges encountered.
Interview guides and surveys were designed to collect data on operational impacts, levels of customer satisfaction, and identification of any institutional challenges. Data were collected multiple times over the course of the FOT to capture changes in perceptions and document any problems encountered. In addition, data were collected from the ESCM system deployers to document any challenges associated with the deployment activities, including any ongoing troubleshooting required over the test period. Figures 3-1 and 3-2 illustrate the changes in operations the Evaluation Team attempted to measure pre- and post-deployment of the ESCM system.
The technical performance of the system components in their operating environment were examined to assess the effectiveness of the technologies and to identify needed system improvements. This performance evaluation focused primarily on system data provided by the automated system administration reports. More qualitative user performance data were incorporated as available. The activities focused on both the operational performance of the integrated air cargo system itself (manifest transaction statistics, etc.), as well as specific functionality of system components (system downtime, etc.), which include electronic manifests, Smart Cards, and biometric fingerprint identification.
Figure 3-1. Phase II Pre-Deployment Data Collection.
Figure 3-2. Phase II Deployment Data Collection.
3.3 Work Steps Completed
The following work steps were implemented to support the technical approach:
- Data collection protocols were established with the ATA Foundation for interaction with the participating carriers, airlines, freight forwarders, and manufacturers.
- Data collection tools were developed to support interviews, surveys, and timing activities on-site at participants' places of business.
- Data collection tools were used to collect data from participants before and during the ESCM system test in coordination with the ATA Foundation.
- Participated in the ATA Foundation’s pre-deployment interview process.
- Participated in the ATA Foundation’s close-out interview process.
- The ATA Foundation's draft final report was reviewed and incorporated into this evaluation as appropriate.
- Qualitative and quantitative data were analyzed to identify key findings and conclusions.
- Data collection protocols were established with the ATA Foundation to schedule exports of archived data from the ESCM system.
- Phase I system analyses were reviewed and summarized as part of the ESCM system evaluation.
- Quantitative system data were summarized and analyzed.
Implications of these technologies were assessed in regard to the National ITS Architecture and Standards.
4. Operational Impact of The ESCM System
A key focus of this evaluation was to document, and measure where possible, the impact the ESCM system had on daily operations of manufacturers, motor carriers, and airlines. These three entities work together to move products from point of manufacture to customers using an intermodal system. It was difficult to identify and recruit representatives from each of these categories because they had to be based on existing supply chains. One supply chain “participant” consisted of a manufacturer, motor carrier, and an airline that already worked together. Without this relationship in place, there could be no test of the system.
In addition, there were many differences among the participants involved in this test. Manufacturers producing different products; motor carriers providing LTL service with fixed routes, variable routes, cross-dock operations, and consolidation operations; and airlines or air freight forwarders that served varying markets.
Within this group of participants, the existing level of automation also varied immensely. Some companies had completely manual paper-based systems with no Internet access, while others had well-established automated systems that were used for all business transactions. This diversity in manual vs. automated operational systems presented a range of challenges for the ESCM system developers and deployers. While some companies had no technological infrastructure (e.g., no Internet access), others had sophisticated systems that could not be integrated or accessed as part of this FOT based on incompatible hardware or software applications.
Regardless of these differences, the participants had similar expectations or hopes for what the system could provide their industry. During pre-ESCM system deployment interviews, participants were specifically asked about the type of results they hoped the test would provide. Their responses are aggregated into the following three categories:
Efficiency. The ability to do their jobs better and more efficiently was the universal theme identified. Although each company has its own operational system in place, whether manual or automated, they all continue to look for new ways to streamline and simplify their operations. One major opportunity for achieving this was the ability of the ESCM system to automate the shipment transfer procedures (biometric/Smart Card authorization vs. manual duplication/photocopying of all paperwork).
Security. A secure, closed system that ensures an individual entity is responsible for a load at all times, coupled with the ability to track the ownership of shipments was viewed as an enormously valuable tool by all participants. Manufacturers reported wanting to able to know the shipment was secure from the time of pick up at the factory to the moment of delivery to their customer. Airlines wanted to know that the load they accept for transport has not been tampered with and is in the same condition in which it left the manufacturer.
Regulatory Compliance. The FAA has established protocols and regulations that must be met for air cargo transport. These requirements have resulted in the cumbersome, paper-intensive system used today. Participants were hopeful that a system of this type would be a step in the right direction for streamlining and improving their ability to meet these requirements.
This section summarizes the data collected in support of this evaluation. This summary includes a description of how the system was used, quantifies comparisons of the manual and automated systems, presents the level of satisfaction experienced by the participants, and identifies the institutional challenges.
4.1 Description of Participants’ Use of ESCM System
The participants involved in this FOT were recruited to test the ESCM system over a finite period of time in coordination with their established supply chain partners. This was a voluntary commitment and their agreement to participate was the result of their interest in improving their operations.
It was hoped that the participants would be able to incorporate the system into their daily operations for regular daily use. In most cases, although the system was used, it was not used as fully as it could have been. Following is a summary describing the manner in which participants used the ESCM system. The subsequent customer satisfaction and institutional challenges section further explains their use patterns.
The ESCM system-generated manifests were used in parallel with the existing manual processes, as time allowed. Few manifests were actually entered into the system due to the limited number of customers on the system.
Manifests were processed simultaneously in both systems. For one clerk interviewed, nearly every manifest created during the test period was entered into the system.
Manifests were handled simultaneously in both systems; however; use of the ESCM system was very limited due to removal and reinstallation of the systemThe ESCM system processes were conducted concurrent with the current processes as staff and time allowed; relatively few were entered because creating manifests using the ESCM system was seen as time consuming for the limited staff (existing system was automated with imbedded customer data).
Manifests were primarily pass-throughs from shipper to airline. Emphasis was placed on driver pick up and delivery procedures; enrolled drivers found the system easy to use.
ESCM system use was limited because primary intermodal partners were difficult to recruit or maintain, or were not part of the FOT.
Current system uses EDI; ESCM is a standalone system, which requires separate data entry; few manifests were processed due to lack of time and staff for data entry.
Due to heavy layoffs and staffing limitations, shipment information was only entered a few times. However, the system was used some for shipment tracking.
The preceding interview data illustrates that most participants did not successfully integrate the ESCM system into their daily operations for all transactions. Most were able to test the system, and in some cases provide ongoing data entry, but not for the majority of their shipments. This was primarily the result of duplicate activities, limited staff resources, and working with established intermodal partners who were not part of the FOT. Interestingly, those participants with manual systems and those with automated systems both had the same issue with duplication.
4.2 Comparison of Manual and Automated Systems
The Evaluation Team, in coordination with the ATA Foundation and SecurCom, timed the manual activities at several participants’ operations. These data were collected and compared to ESCM system-generated reports, which provided the details for the same activities as conducted with the ESCM system. The following summary presents data from this comparison.
The manual “time on-task” data were collected at several participant sites. These data represent a variety of tasks identified to represent those activities present in the ESCM system. Both the manual timings and the ESCM system data represent aggregates of the data collected: the sum of all manual timings were compared to the sum of all ESCM system data. However, manual timings were not collected from all participants; the ESCM system data were collected from all participants that used the system. Comparison of these data, therefore, represents an estimate of time savings based on the manual timings of a subset of the participants.
The quantitative data presented here supports the findings of the qualitative analyses. A comparison of the manual and automated processes shows time savings in every category measured. Tables 4-1 through 4-3 show these time saving estimates.[3] In instances where the ESCM system replaces/eliminates human activity, a null time value of “zero” is assumed (e.g., e-mails automatically generated to communicate load transfer information). In addition, other major savings occur with key activities like load acceptance. For example, load acceptance at an air cargo facility was timed at 3 minutes, 3 seconds less than the timing for the manual process. This same activity using the biometric/Smart Card verification process took just 11 seconds, which represents a 94 percent reduction in time.
Table 4-1. ESCM System vs. Manual Process Times
per Shipment for Manufacturers
Activity |
Manual Time |
ESCM Time |
Time Savings |
Percent Reduction |
Filling out manifests |
2:35 |
1:07 |
1:28 |
54% |
Contacting motor carriers (from carrier order acceptance) |
0:51 |
- |
0:51 |
100% |
Search out documentation, load verification, driver sign-off |
4:12 |
0:18 |
3:54 |
93% |
Paperwork error correction |
0:12 |
- |
0:12 |
100% |
Table 4-2. ESCM System vs. Manual Process Times per Shipment
for Trucking Companies
Activity |
Manual Time |
ESCM Time |
Time Savings |
Percent Reduction |
Order acceptance over the phone and data input |
0:51 |
- |
0:51 |
100% |
Load acceptance at manufacturer |
4:12 |
0:18 |
3:54 |
93% |
Input to create master manifest |
2:08 |
0:43 |
1:25 |
66% |
Reproduction of manifests |
1:03 |
- |
1:03 |
100% |
Paperwork error correction |
1:03 |
- |
1:03 |
100% |
Contact airline and arrange shipping |
4:09 |
- |
4:09 |
100% |
Delivery to airlines |
3:03 |
0:11 |
2:52 |
94% |
Table 4-3. ESCM System vs. Manual Process Times per Shipment for Airlines
Activity |
Manual Time |
ESCM Time |
Time Savings |
Percent Reduction |
Order taking/contact motor carriers |
4:09 |
- |
4:09 |
100% |
Load acceptance |
3:03 |
0:11 |
2:52 |
94% |
Clerical time for creating airplane load documentation |
3:00 |
- |
3:00 |
100% |
Paperwork error correction |
0:41 |
- |
0:41 |
100% |
Copy and file for FAA audits |
1:10 |
- |
1:10 |
100% |
The time savings estimates presented in the preceding tables show significant reductions by percent for these tasks. These savings are substantial when multiplied by total number of shipments per year for a given company. The ATA Foundation’s final report has converted these time savings into dollar savings. As part of the independent evaluation, percent reduction has been used to focus more on direct operational metrics as opposed to financial feasibility.
Given the qualitative results, which suggest that the system was well received but underutilized, the time savings should be viewed as a conservative estimate of potential savings. However, conversely, the test parameters required that the existing manual and ESCM system processes be completed. Without fully testing the ESCM system as “the system”, it is difficult to predict with confidence that all these benefits would remain. In summary, these results show extreme potential for time and cost savings through ESCM system deployment.
4.3 Customer Satisfaction and Institutional Challenges
As with many FOTs, the experiences of the participants over the course of the deployment provide valuable lessons learned for future deployments of this type. In fact, with all of the challenges faced by this test with participant turnover and dropping out, the qualitative data provided by the participants became extremely important. Participants dropped out for a variety of reasons documented throughout this final report. The major factor was the events of September 11, 2001 combined with the overall slowdown in the economy. This led companies to pull back from their voluntary commitments to tests of this type to focus on their bottom lines.
As the following section documents, participants did not necessarily drop out because of system performance, but rather because they could no longer afford to operate two duplicate systems. Staffing constraints and profitability became their ultimate priority, as should be expected of for-profit organizations. Unfortunately, this limited the quantity of quantitative data, however, a lot was learned by discussing with the participants what they thought of the system and having them explain why they dropped out.
The following section summarizes the level of satisfaction experienced by the participants, and documents the challenges faced over the course of the test by the participants and the deployers.
4.3.1 Customer Satisfaction
Customer satisfaction was measured qualitatively via interviews. The interviews were conducted throughout the course of the operational test and focused on collecting perceptions on training, functionality offered by the system, ease of system use, and Internet accessibility in tracking shipment information.
Heavy users reported continued interest and commitment to the system throughout the FOT. These participants had operations staff that found the system easy to use and they remained committed in many cases in hopes that the system would be expanded and further deployed. They reported that once the ESCM became the primary system for their and their supply chain partners operations the true benefits could be measured. In addition, other less heavy users acknowledged that they expected the benefits would more easily manifest themselves with broader deployment of the system.
The ESCM system successfully duplicated existing systems, providing all the functionality required. The participants were very satisfied with the ability of the ESCM to duplicate all necessary business functions and in some instances reported finding the system easier to use than their traditional processes.
Training activities were “very thorough”; the system was simple and easy to use. Many reported the ESCM system would be easier for new employees to learn than the existing system. Most participants found that not much training was required; those that did were provided the necessary level of detail and found SecurCom staff easy to work with and responsive.
Access to shipment status was a major benefit; it could be integrated with carrier websites to provide customers with additional shipment tracking capabilities. Given the relatively small number of shipments that traversed the complete system from shipper to receiver, the support documented for this system capability indicates the overall importance of shipment tracking to the transportation industry.
Very few participants reported technical problems; when they did, it almost always consisted of Internet access issues. Some companies did not have Internet access prior to the test, were using dial-up modems, or were unable to install the ESCM into a secure environment, which is where their Internet access was provided. The participants were satisfied with the efforts put forth by SecurCom to provide them with the necessary connections.
Technical support was timely and effective; there was rarely a need to ask for help as the system was dependable and simple to use. The dependability of the system, combined with the professional support provided is likely what motivated many of the participants to remain involved for most or all of the operational test.
The system provides a more effective way to exchange information among supply chain partners; it provides a “one-stop shop” for shipment information. Many of the participants relied on traditional communications tools (telephone, fax, and e-mail) to access shipment status. Under those conditions there were times that it was unclear exactly what the status was. The ESCM provided the ability to provide real time information on who had possession of the shipment at any given time.
The system provides improved security because there is limited access and documentation of all individuals that had access to the load. This characteristic addresses the primary reason the FAA sponsored this operational test. The existing process relies heavily on a paper trail, which is labor intensive and cumbersome. The ESCM automatically accomplishes this. In addition, driver validation is especially useful for any clerks that are not familiar with regular drivers by face.
The ESCM system is a faster system to use than existing manual procedures and was reported to take less time than some in-house automated systems due to screen layout and format. Therefore, for some participants the ESCM did more than duplicate the existing processes – it improved upon them.
Drivers were very interested in and supportive of reducing pickup/drop off delays. For a driver, the time spent at a terminal is unproductive time. This is especially true if he/she is simply waiting for paperwork. The ability to walk in, log in to the system and have all the paperwork completed electronically was seen as a major strength of the system.
The ESCM system was well received by the participants. They felt it was easy to learn and use, it provided all the required functionality of the existing system, and it was believed to provide improved operational efficiency when more fully deployed. Their major frustrations focused primarily on the incompleteness of the system for two reasons. First, it did not replace their daily routine, it added to it. And second, not all of their customers (supply chain partners) were online.
4.3.2 Institutional Challenges
The institutional challenges were also addressed through qualitative interviews completed throughout the operational test. This process allowed the participants to comment on the factors they viewed as barriers to system acceptance. The following lists the challenges identified by this process:
- Duplicate systems created need for multiple data entry; in many instances this restricted the number of manifests entered into the system as participants reported this as too time-consuming given their staff resources. This situation created a bottleneck for testing the system, as it greatly restricted the number of manifests generated. In some instances, use of the system was limited to those clerks that were interested in the system. This factor was further exacerbated because for a manifest to travel completely through the entire system there had to be interested users at three or more locations.
- Staff turnover and layoffs restricted the resources available for a duplicate system. As described under the customer satisfaction findings, staffing played a major role in use of the system as well as in determining whether or not a company remained involved in the test.
- Lack of integration between the ESCM and legacy systems restricted use and acceptance of the system. Many of the participants had existing systems for processing shipments. The existing system had to be maintained as this was only a test. Therefore, the ESCM was only used when time allowed. System integration could not be provided on a test of this scale but was considered absolutely necessary if the system was to ever be fully deployed.
- Lack of participation by enough supply chain partners restricted use of system. Many participants reported that there simply were very few of their transportation partners enrolled in the system; without a larger number of participants the benefits are difficult to quantify.
- Several participants stated that they did not have enough air freight shipments to further test the system. This factor is the result of a decrease in shipments caused by current events, inappropriate participants included because they were very interested in the system, or the result of an excuse by the participant to explain their low volume of manifests.
- Slow connection time to access the Internet was frustrating; some participants felt this was an internal problem, while others simply said faster access was required. As discussed in the customer satisfaction findings, some participants had restricted Internet access for one reason or another. SecurCom ensured that all participants were in fact connected, but most participants felt high speed access was required for the system to be successful in the operating environment.
- Driver aversion to biometric fingerprinting remained a topic for discussion throughout the FOT; however, it diminished over time. As with any technology there was some apprehension with the technology. However, at specific times during the operational test, the biometric technology received serious scrutiny from the participating truck drivers.
The key institutional challenge that impacts most of the preceding specific issues is available resources. Although the ESCM was designed to replace existing processes, it did not replace any system as part of this operational test. It simply added a new set of procedures that duplicated an existing process. Given the serious nature of air cargo security, and the fact that not all supply chain partners were involved in the test, it is difficult to eliminate the existing and regulated process. However, without doing so, the test is restricted to partial use of the system as resources allows.
5. Overview of the Technical Effectiveness
of the ESCM System
The technical effectiveness evaluation was conducted to measure the performance of the ESCM system and its components. This evaluation was organized and focused on the following four areas:
- Integrated Air Cargo System Operational Performance
- ESCM Computer System Technical Performance
- Electronic Manifest Technical Performance
- Biometric Fingerprint Identification Technical Performance
In addition, a questionnaire was developed and administered to trucking companies, manufacturers and airlines to qualitatively assess the technical performance of hardware and software, and the participants’ satisfaction with the system performance and features. These findings are discussed in the Customer Satisfaction and Institutional Challenges evaluation provided in Section 4.
5.1 Integrated air cargo system OPERATIONAL PERFORMANCE
The integrated Air Cargo Information System (consisting of the electronic manifest, Smart Cards, and biometric identification technologies as shown in Section 2) was evaluated using ATA Foundation System administration reports and statistics, and Evaluation Team survey results. The ATA System administration reports and statistics provided ESCM website transactions and computer system/network performance data. These reports produced detailed logs of ESCM system performance data, which enabled the detailed assessments to evaluate the technical effectiveness of the technologies.
The ESCM website recorded predefined transactions to study the effectiveness and efficiency of the ESCM system. The ESCM system automatically recorded all transactions including the date, time, manifest access type, transaction events and measures, and user login types.
5.1.1 Positive Association of Driver with Electronic Manifest
Over the past year, from April 11, 2001 to May 2, 2002, the ESCM system server recorded 374 transactions. Of all the transactions, creating and releasing a manifest were the most commonly recorded types (129 and 126 occurrences, respectively). Figure 5-1 shows the frequency of ESCM transactions. Of the 129 manifests created, 126 corresponded with releases to the respective trucker and receiver (airline). Three manifests were created, but had not yet been released. There were 23 shipments that were picked up and recorded by the ESCM system. Of these 23 shipments, the ESCM system recorded eight of these as being delivered to the receiver (airline).
Although the ESCM system server data only had eight shipments during the defined time period that represented the complete electronic manifest cycle (create to release to pick up to delivery), this is not surprising considering the deployment parameters. There were a limited number of complete supply chains, and given the limited staff resources, not all shipments were entered into the system.
For example, manufacturers could be entering manifests that were subsequently being picked up by nonparticipating motor carriers, or the motor carrier may deliver the shipment to a nonparticipating airline. What the data does show, however, is that the ESCM system successfully recorded and tracked electronic manifests, the movement of cargo along the supply chain, and the overall transfer of responsibility from a given manufacturer to motor carrier to final recipient (airline).
Figure 5-1. Frequency of ESCM Transactions
5.1.2 Total Manifest/Cargo Cycle Time
The eight manifests/cargo shipments that completed the entire electronic manifest cycle were examined to measure the total manifest/cargo cycle time. The manifest and cargo cycle time is the time elapsing from manifest creation to physical delivery of the cargo to the final recipient (airline). The time begins when a manufacturing clerk completes filling out the ESCM web-based manifest form. Once the truck driver makes the delivery, the driver logs into the ESCM system and verifies that the cargo has been delivered. The cycle is complete and the ESCM system server records the date and time of delivery.
To compute the total time elapsing between manifest creation and physical delivery of the cargo to the final recipient, the date and time the manifest was created was subtracted from the recorded date/time for shipment delivery to the receiver (airline). Table 5-1 presents the date/time of creation and delivery and elapsed times for the eight manifests.
Table 5-1. Total Manifest/Cargo Cycle Time
Manifest |
Create |
Delivered |
Elapsed Time (hours:min:sec) |
77 |
5/31/2001 9:32 |
5/31/2001 9:59 |
00:27:00 |
84 |
7/2/2001 11:42 |
7/3/2001 10:09 |
22:26:41 |
85 |
7/2/2001 13:21 |
7/3/2001 10:09 |
20:47:28 |
86 |
7/2/2001 13:37 |
7/3/2001 10:07 |
20:29:57 |
89 |
7/18/2001 9:38 |
2/5/2002 11:34 |
7 months, 20 days, 1:56:01 |
143 |
2/5/2002 17:19 |
2/5/2002 17:21 |
0:01:44 |
144 |
2/6/2002 10:42 |
2/14/2002 14:46 |
8 days, 4:04:06 |
145 |
2/6/2002 11:42 |
2/6/2002 11:49 |
0:07:47 |
Excluding Manifests # 89 and 144, the average elapsed time was 10 hours, 43 minutes, and 26 seconds. However, elapsed times appear to fall into three groups: short (less than 30 minutes); medium (elapsed times of about one day); and long (elapsed times taking numerous days, even months). Those manifests in the less than 30 minutes group averaged just over 12 minutes with the shortest elapsed time being one minute and 44 seconds (manifest 143). For the manifests lasting about one day the average time was about 21 hours and 15 minutes. Finally, the manifest elapsed times that were long appear to be shipments that were either delayed or the ESCM transactions were not entered at the actual time of the event.
Irrespective of the underlying cause resulting in the elapsed times, the ESCM system did function correctly by capturing the transactions upon entry by the user.
5.1.3 Physical Cargo Cycle Time
The eight manifests/cargo shipments that completed the entire electronic manifest cycle were also examined to assess the physical cargo cycle time. The physical cargo cycle time is the time elapsing from pickup of the cargo (at the manufacturing site) by the trucker to physical delivery of the cargo to the final recipient. The time begins after the ESCM system verifies a truck driver as being in the system and registers the individual as the assigned truck driver. Once the truck driver makes the delivery, a truck driver logs into the ESCM system and verifies that the cargo has been delivered. The cycle is complete and the ESCM server records the date and time of delivery.
The physical cargo cycle time is calculated using the difference in date and time between when the trucker picked up the cargo at the manufacturing site and physically delivers the cargo to the final recipient. Eight entries were available. For each manifest, the date/time of pick up and delivery and elapsed times are shown in Table 5-2.
Table 5-2. Physical Cargo Cycle Times
Manifest ID # |
Picked Up Date/Time |
Delivered Date/Time |
Elapsed Time (hours:min:sec) |
77 |
5/31/2001 9:58 |
5/31/2001 9:59 |
0:00:44 |
84 |
7/3/2001 10:08 |
7/3/2001 10:09 |
0:01:10 |
85 |
7/3/2001 10:08 |
7/3/2001 10:09 |
0:00:27 |
86 |
7/2/2001 13:40 |
7/3/2001 10:07 |
20:26:48 |
89 |
7/18/2001 9:40 |
2/5/2002 11:34 |
7 months, 20 days, 1:54:01 |
143 |
2/5/2002 17:20 |
2/5/2002 17:21 |
0:01:02 |
144 |
2/14/2002 14:16 |
2/14/2002 14:46 |
0:29:43 |
145 |
2/6/2002 11:47 |
2/6/2002 11:49 |
0:02:01 |
Average Time = 3:00:16[4] |
|||
The elapsed times varied in length from less than 1 minute to over 7 months and 20 days. Excluding the elapsed time for manifest # 89, the average elapsed time was 3 hours, 16 seconds. However, five of the eight entries were recorded at approximately 2 minutes or less, suggesting that the picked up and delivered entries were completed sometime after the actual events occurred.
5.2 ESCM Computer System TECHNICAL Performance
The overall performance of the ESCM computer system was assessed using the Microsoft NT Performance Monitor, a software application that reports system performance metrics. Used by the ATA Foundation System Administrator, the NT Performance Monitor measured the performance of the system network computers. Each of the computer’s processes and components has an associated set of counters that provide information about device usage, queue lengths, delays, and information used to measure throughput and internal congestion. Performance Monitor provides charting, alerting, and reporting capabilities that reflect both current and ongoing activity – allowing users to open, browse, and chart log files of current and past activity.
The computer system performance was assessed in three areas: system reliability, system availability, and errors, as described in the following sections.
5.2.1 System Reliability
To assess overall system reliability, two metrics were investigated: system up time and unscheduled system downtime. Using the system up time metric, which measures the total time the computer is operational, the computer was in continuous operation for 182 days. There were no occurrences of unscheduled downtime over 6 months of testing. The most recent scheduled downtime was a computer reboot needed to perform planned maintenance (i.e., service pack installation) on the computer. Consequently, the ESCM system server appears to have functioned reliably for the past 6 months.
5.2.2 System Availability
The ESCM computer resources were also examined to assess the availability of computer resources. Five NT Performance Monitor metrics were investigated:
- Processor Availability
- Interrupts per Second
- Bytes Transferred per Second
- Disk Space Available
- Pages per Second
Following is a description of each performance metric:
Processor Availability indicates the percentage of elapsed time that the processor is busy working on a task (i.e., non-idle thread). The NT Performance Monitor for the ESCM system server reported the average percentage of processor utilization time was 2.6 percent. This value indicates that the server processor was working 2.6 percent of the time and was readily available to perform ESCM system functions.
Interrupts per Second measures how busy the computer processor is in receiving and servicing hardware interrupts. Normal thread execution is suspended during interrupts, so this value is an indirect indicator of system-wide activity of devices, such as the system clock, mouse, disk drivers, data communication lines, network interface cards, and other peripheral devices. These devices normally interrupt the processor when they have completed a task or require attention. The number of interrupts per second for the ESCM system server ranged from a low of 65.99 to a high of 177.24. The average interrupts per second was 89.17, which is not particularly high, and indicates a relatively modest level of hardware activity.
Bytes Transferred per Second provides an overall indication of how busy the server is sending and receiving data across the network. The number of bytes transferred per second ranged from a low of 0 (zero) to a high of 590.99. The average number of bytes transferred per second was relatively low at 19.27 bytes per second.
Disk Space Available measures how much of the total disk space remains. Having little or no disk space can affect the server’s speed in processing tasks and indicates the need as to when to make (or acquire) more disk space. The average percentage of free space available was calculated and found to be 35.1 percent. While it appears that an adequate amount of disk space is available, however, depending on the total disk capacity and rate of disk space consumption, more disk space may need to be added to maintain optimum server performance.
Pages per Second measures the number of pages read from or written to disk to resolve hard page faults. Hard page faults occur when a process requires information that must be retrieved from disk. When physical memory becomes full, virtual memory uses the hard disk to transfer information back and forth. Although the size of a page depends on the amount of physical memory, disk space available, and system settings, this “paging traffic” is a primary indicator of the type of faults that cause system-wide delays. It is counted in numbers of pages and is the sum of memory pages input/second and memory pages output/second.
The pages per second ranged from a low of 0 (zero) to a high of 34.00. The average number of pages per second was found to be 0.21 pages. At 0.21 pages per second, there does not appear to be excessive paging traffic resulting from excessive pressure (or demand) on the physical memory.
5.2.3 Errors
ESCM system errors were also investigated to assess the frequency of access-denied errors and unexpected system errors. The NT Performance Monitor provided measures of both types of errors.
Errors Access Permissions indicates the number of times a request to open a file failed because the user did not have the proper access privilege. Access privileges are created when the user’s account is created and can only be changed by the system administrator. If someone is randomly attempting to access files in hopes of getting at something that was not properly protected, the NT Performance Monitor records the attempt and counts it as a STATUS_ACCESS_DENIED error. There were no access permission errors during the reporting period.
Errors System indicates the number of times an unexpected internal server error is detected. Unexpected errors usually indicate a problem with the server. There were no unexpected system errors reported, indicating that unexpected problems with the ESCM system errors did not occur during the reporting period.
5.3 Electronic Manifest Technical Performance
This portion of the evaluation focuses on the electronic manifest. Two topics are addressed here, the manifest processing time and error rates.
5.3.1 Manufacturer Manifest Processing Time
Manufacturer manifest processing time was assessed to investigate the time required for the ESCM system to release a manifest. The Manufacturer Manifest Processing Time is the time elapsing from manifest creation to releasing the manifest to the respective trucker and receiver. As previously mentioned, manifest creation time begins when a manufacturing clerk completes filling out the ESCM web-based manifest form. Once the manifest is created, the ESCM system accepts the manifest form and electronically notifies the truck driver and receiver of the pending shipment.
To compute the total time elapsing between manifest creation and release to the respective truck driver and receiver, the date and time of manifest creation is subtracted from the recorded date/time for manifest release. ESCM system data was available for 126 manifests. To show the frequency of manifest processing times, duration intervals or time groups were used to group the times. Figure 5-2 shows the frequency of processing times as a function of the time group. For example, elapsed times between 15 seconds but less than 30 seconds were grouped and counted. For this group, 39 of the 126 manifests were processed between 15 and 30 seconds.
Figure 5-2. Frequency of Elapsed Times for Manifest Creation to Release
It was recorded that 74 percent of the manifests required less than 1 minute and 15 seconds to be processed before being released. In addition, 86 percent were processed in less than 2 minutes and 92 percent were released in less than 3 minutes.
5.3.2 Error Rates
An examination of the occurrence of transfer errors between the ESCM system server and Smart Card system was not performed. The ESCM system administration reporting and statistics did not record any transfer errors to allow assessment of the transfer error rates between the ESCM and Smart Card system. However, the ESCM Server did capture a number of user-related errors. As shown in Figure 5-3, the most frequently occurring error was Error 5, “Carrier is not assigned to the selected manifest.” However, of the 20 occurrences, 17 were from the same manifest and occurred after the manifest was released to the respective truck driver and receiver. In reviewing the frequency as a percentage of total manifests created, Error 5 only occurred in 3 of 126 or 2 percent of the manifests created.
Figure 5-3. Frequency of ESCM System Errors
The second most common error was Error 1, “No identification supplied.” In contrast to Error 5, Error 1 was distributed over 11 different manifests. Thus, 9 percent of the manifests created had a “No identification supplied” error. Similarly, but to a lesser extent, Error 4, “User is not defined as a Trucker”, had 6 of the 9 errors distributed over different manifests. This represented 5 percent of the total number of manifests created. Error 2, “User does not exist “, was relatively rare in that it occurred in 2 percent of manifests.
Only two types of errors did not occur in the ESCM system data: Error 3, “User is not an active use”, and Error 6, “The manifest has already been delivered.”
5.4 Biometric Fingerprint Identification technical Performance
Dr. James Wayman of the U.S. National Biometric Test Center at San Jose State University independently evaluated the biometric fingerprint identification system and Smart Card technology in Phase I of the Electronic Intermodal Supply Chain Manifest ITS Field Operational Test. Results from both qualitative and quantitative evaluations were originally reported in June 2000.[5] The analysis of false rejection error rates and false acceptance rates reported in the following section is a summary of Dr. Wayman’s system performance test results.
5.4.1 Analysis of the False Rejection Error Rate
In 1998 commercial drivers were asked to verify their identity by providing fingerprint information upon entering the O’Hare International Airport cargo area. Data logs from the system were later analyzed to examine the “false rejection” rates of the O’Hare Air Cargo Security Access System. The test data consisted of 2,239 transactions between August 8, 1998 and July 14, 1999. The transaction data contained driver identification, transaction location, transaction date/time, action, operator number, and gate. There were 13 types of actions recorded: Driver Added, Driver Deleted, Driver Updated, Verify Pass, Verify Fail, Invalid Driver ID, Manifest Added, Manifest Accepted, Manifest Comments Up, Operator Enrolled, Operator Deleted, Operator Logged, and Shipment Updated.
The analysis focused on 497 “Verify Pass” or “Verify Fail” transactions. These transactions indicated that a fingerprint image was taken and compared to a stored fingerprint image. The outcomes (either “Pass” or “Fail”) were used to calculate the system’s false rejection rate.
The data were adjusted to compensate for three factors that bias the computation of error rates. The three factors are “transaction inflation” (a few users generating a large number of transactions), which tends to decrease the error rate; “multiple failures in the same session”, which causes an overestimation of the errors; and “template aging” (long elapsed time since fingerprint enrollment), which can increase errors.
After compensating for bias, 126 sessions from 65 users were used to derive the false rejection error rate. As shown in Figure 5-4, of these 126 sessions, 91 verifications resulted in “Pass” and 35 had “Fail” outcomes in the first attempt. Consequently, the first try rejection rate was 28 percent (35 / 126).
Figure 5-4. First-Try Outcome
In Figure 5-5, of the 35 first-attempt failures, 28 users chose to use another form of identification and seven users made a second attempt to verify their fingerprint. Of the seven users attempting a second time, five passed and two failed. (Neither user of the two failed cases attempted a third try, apparently opting to use other means of identification.) Overall, false rejections after two attempts represented 1.6 percent of the total number of sessions (2 / 126).
Figure 5-5. Second-Try Outcome
Although the rejection rate for a second attempt was two out of seven (29 percent), the acceptance rate was five out of seven (71 percent). At this rate of acceptance (71 percent), had all 35 first-attempt failed users chose to re-try, 25 of the 35 would have been expected to be accepted after two fingerprint verification attempts.
Conversely, with a second-try failure rate of 29 percent, 10 of the 35 users would be expected to be rejected after two verification attempts. However, the 10 users expected to be rejected after two attempts represents an overall two-try failure rate of only 8 percent of the total (10 out of 126).
According to Dr. Wayman’s analysis, both 28 percent (first-try rejection) and 8 percent (expected two-try rejection):
…are not at all unreasonable for a system with inexperienced users and limited visual feedback…. Fingerprint systems in general require significant training of and feedback to the user. The primary challenge to the user is the consistent placement of the fingerprint "core" on the scanner. During enrollment, the user is instructed and supervised in this placement. In verification trials, however, the user is expected to remember and repeat this placement. Users with limited experience with the system, such as the 27 in this study with one use only, are believed not to perform well.[6]
5.4.2 Analysis of the False Acceptance Rate
Dr. James Wayman of the U.S. National Biometric Test Center and Dr. John Scott of Identix performed the analysis of the false acceptance rate. Since all user transactions during the 1998 O’Hare test period were presumed to be by enrolled users, no special impostor testing was performed at that time. Also, the operationally submitted fingerprints were not stored for later artificial impostor comparisons. Therefore, the false acceptance rate analysis of the fingerprinting technology was performed using standard databases unrelated to the O’Hare project.
The data used for the false acceptance test were derived from a standardized data set collected for the Republic of the Philippines Social Security System (SSS) Identification Card Project Automatic Fingerprint Identification System (AFIS) benchmark test conducted in May 1997[7]. The data were collected voluntarily from employees of the Republic of the Philippines Social Security System[8]. During data collection, three separate images (a triplet) of the same fingerprint were scanned and stored in a database. For this analysis, a total of 200 triplets were processed by the Identix algorithm to develop a feature template of the image. From the original images, the algorithm developed 165 templates (83 percent of the 200 images)[9].
Using the 165 templates and non-matching single images, 22,260 comparisons were made to determine the false match rate as a function of threshold score. According to Dr. Wayman, the purpose of pattern matching in an AFIS is to determine a similarity score between a presented fingerprint image and the stored templates in the database.
If the score is larger than some arbitrary threshold, a "match" is declared. If smaller, a "non-match" is declared. It is clear that the chances of a match or non-match are dependent upon the chosen threshold. As the threshold is decreased, there is a trade-off between a decrease in false non-matches and an increase in false matches[10].
The threshold is adjusted either by the vendor or the system administrator during implementation and can vary based on the allowable or desired error rates.
The false acceptance rate as a function of threshold score is shown in Figure 5-6. The O’Hare test used a threshold of 100. At this threshold level, approximately one false match per ten thousand comparisons would be expected.
This false acceptance error rate, according to Dr. Wayman, is completely consistent with finger-print systems used for access control applications and is considerably better than many biometric systems currently in use for government applications. This system false acceptance error rate is competitive with 4 - digit PIN systems[11] .
Figure 5-6. False Acceptance Rate as a Function of Threshold
6. Conclusions and Recommendations
6.1 Conclusions
A set of conclusions has been developed based on the quantitative and qualitative data collected over the course of this FOT. Many of the observations are based on the experiences of the participants and the deployers and are summarized as follows:
This FOT successfully demonstrated the use of technology to create a secure intermodal electronic manifest system. The ESCM was designed to be a standalone system that provided the secure transfer of information from manufacturer to motor carrier to airline. This was accomplished for multiple supply chains at two separate geographic locations.
The time savings estimates developed in this report show the potential for substantial industry time savings by the implementation of this system. These comparisons estimated that there are in fact operational time savings with the ESCM system. Many of these savings were estimated to come from replacement of manual processes with system generated processes, like automatic notification of load pickup or acceptance.
The calculated benefits should be viewed as illustrative of potential benefits under full deployment. Given that the ESCM system did not become fully operational, that is, use of the system only captured specific supply chains within a company, and the manual/existing system was maintained, the results of this evaluation represent potential opportunities. Without full deployment testing, these benefits cannot be assumed, only estimated.
Few shipments were processed through the entire ESCM system. Over the defined test period, only eight shipments moved completely through the system from manufacturer to airline using the ESCM system. This is the result of limited system use as a predictable outcome due to the overall lack of staffing resources and partner participants (not all of a manufacturer’s trucking companies were participating, so some manifested loads ended up moved by unequipped trucks, terminating the ESCM system continuity).
Participants have reported overall satisfaction with the ESCM system. Some participants felt the ESCM system would be significantly more useful with wider deployment to more of their supply chain partners.
Several participants used the system for 18 months as opposed to the originally planned 6-month test period. Participants using the ESCM system over the longer duration asked the deployer regarding what is the next step toward full integration and deployment, and when will the ESCM system become fully operational so that they can discontinue their manual processes.
Drivers were very interested and supportive of a system that provides a single Smart Card that replaces their commercial driver’s license (CDL) and paper work. Drivers were very supportive of a system that would streamline their interaction with clerks at transfer points. One Smart Card that eliminates the need for cumbersome paperwork and duplicate copies of their CDLs was welcomed.
The FOT required that the existing manual systems be maintained, making participation in this test a duplicative effort for the participants. This challenge of duplicating the manifest creation process resulted in sporadic use of the ESCM system. If a company had an extremely busy day, or staff constraints, they did not use the ESCM system – no time or resources to do the same job twice.
Operations staff were a critical part of the FOT. Once the ESCM system was installed at a participant’s location, the operations staff were intensively involved from that point forward.
Significant outreach activities were conducted to build support for the ESCM system. Nearly 100 ESCM system demonstrations were provided for participants. Reactions ranged from “that’s cool” to “you’re not getting my finger print”. SecurCom was available any time for additional training and support.
Recruitment of participants was an ongoing challenge. The FOT was presented as an opportunity for industry to influence a future mandated system; identifying complete supply chains was difficult; starting with motor carriers was the best way to build complete supply chains; getting the managers to personally “buy in” to the test was key for stimulating staff.
Training activities worked very well and consisted of multiple formats. Training was provided in a variety of ways. When the ESCM system was initially installed, the system integrator met with participant staff for basic training and participant use. This included one-on-one sessions, group sessions, and the case study approach using hands-on training. Staff picked up the material quickly; this was attributed to their existing familiarity with the Internet and Microsoft Windows platform. The transportation focus of the material was more of an issue for manufacturing staff.
Staff turnover at participant companies dramatically impacted using the ESCM system. Turnover in personnel affected both management and operations staff. Several participants dropped out of the test as their internal ESCM champion departed the company and their replacement did not have time or interest in participating. Additionally, some personnel wanted an increase in pay based on added technical skills (learning and using the ESCM system); it was also typical that at least one staff person at every company made an issue of the biometric fingerprinting.
Existing automation primarily consisted of customized or proprietary systems. Many companies had automated systems; the ESCM system duplicated these systems (or some part of them) and did not integrate with them (somewhat due to hardware and software incompatibility). In addition, the industry still relies heavily on the use of facsimile throughout their supply chains for communication.
ESCM system use varied by company and also within the staff of a company. Successful pairings were necessary for regular/routine use. There were sets of clerks/drivers that were supportive and interested in the system and did use it regularly. Others used the system sporadically, or initially used it, and then their use and interest dissipated over time.
Establishing Internet access was a factor for some participants. Several participants required use of dial-up modems, while others had Internet access through their internal networks, but would not allow access, which required that a separate connection needed to be established. In some cases, SecurCom facilitated the connection by providing additional phone lines and resources to get the connection established.
The impact of the September 11, 2001 terrorist attacks initially brought this FOT to the forefront of national security. Many expected the FOT to move forward and be expanded rapidly. However, the airline industry was immediately focused 100 percent on daily operations, security, and profitability. These challenges dramatically impacted their ability to participate in a voluntary test that required duplicate activities.
The test was successfully developed and deployed within the confines of a complicated organizational structure. The FOT was developed and deployed by multiple funding sources and multiple lead agencies. This created the need for a high level of coordination by ATA Foundation and SecurCom staff to ensure that all stakeholders were provided with the appropriate level of service, while not negatively impacting the operational participants. At times, this added layer of coordination among funding sources and agencies distracted SecurCom staff from the primary goal of operating the ESCM system.
The ESCM system had to keep operating regardless of institutional challenges. This condition created situations where SecurCom, supported by the outstanding FOT leadership displayed by the ATA Foundation, took on personal risks and absorbed additional costs to maintain momentum during the FOT. Consolidation of funding and appropriate Memorandum of Understanding (MOUs) could have prevented or better managed these issues.
Examination of the ATA System Administration Reports and Statistics showed that the integrated air cargo system (consisting of the electronic manifest, Smart Cards, and biometric identification technologies) completed the required tasks expeditiously. ESCM system transactions were recorded by the server and errors captured in a timely manner. Although the limited number of project participants resulted in only eight shipments completing the entire electronic manifest cycle (create, release, pick up, delivery), the ESCM server did capture the transactions for subsequent analyses. Total manifest/cargo cycle times and physical cargo cycle times were calculated. The cycle times revealed instances of both short (less than 1 minute) and long elapsed times (over 7 months).
In over 6 months of testing, the ESCM computer system and network was found to be reliable. Reliability was measured in terms of system up time, unexpected errors, availability of system resources (and not working near capacity), and lack of unauthorized access attempts. Five metrics of system availability were examined: all indicated that the computer resources (computer processor, network activity, disk space, and physical memory) were working below capacity and were readily available.
The ESCM system server processed the electronic manifest in a timely manner. In general, upon creation of the manifest the ESCM system server released the manifest within a few minutes:
- 74 percent were processed and released in less than 1 minute,15 seconds;
- 86 percent were released in less than 2 minutes;
- 92 percent were released in less than 3 minutes.
6.2 Recommendations
The Evaluation Team prepared the following list of recommendations for consideration by FHWA and FAA as they consider future tests of this type, as well as expansion of the ESCM system:
Acquire a partial waiver from FAA to allow full testing of the ESCM system to document actual changes/improvements in operations based on full deployment. This would require participants to fully embrace the system and rely on it. This would take the test to the next step as lessons learned would be on actual system use as opposed to partial testing as time permits.
Expand the current follow-on phase of this operational test to include the second half of the freight movement. This FOT covered manufacturer to motor carrier to airline. This should be expanded to include final delivery of the shipment (airline to motor carrier to customer). This should be able to be accomplished by piecing together additional supply chain participants. This expanded test parameter addresses operational efficiencies more than safety/security issues as the FAA is more concerned with what is placed on a plane. However, for an integrated supply chain management system, the information flow must be complete.
Focus resources on a small number of supply chains where the system is fully (or more fully) integrated into the daily operations. This should include elimination of requirements to maintain the existing manual system (FAA waiver for test period). This would provide the opportunity to more fully explore the strengths and weaknesses of the system as it would become the system the staff and regulatory agencies relied on for secure freight movement.
Develop operational parameters that require participants to process a specific number of electronic manifests through the entire supply chain. The Phase II test proved the technology worked, but from a statistical standpoint, it did not successfully prove the operational stability of the system. With only 8 complete manifest movements (from manufacturer to airline) the system needs additional testing. Future versions of this FOT should ensure this number goes up substantially through recruitment, incentives, and support.
Identify an appropriate participant (high volumes and multiple partners) with a legacy system and develop an interface between it and the ESCM system to test the ease of customization. Of the participants with automated systems, they were frustrated the inability to integrate their electronic systems. This resulted in limited testing of the ESCM system. This challenge also signifies one of the leading obstacles to full deployment and should be tested further.
Develop customized views of the electronic manifest for each participant type. The existing electronic manifest did not duplicate the manufacturers’ shipping forms as well as it did the motor carrier and airline forms. In a more advanced system, the manifest should be able to be prepared in a few formats or templates to facilitate business as usual for all participants.
Develop an organizational structure that streamlines access to funding and facilitates decision making from an operations perspective. The Phase II test had multiple agencies playing lead roles and multiple funding sources that impacted ease of implementation early in the development and deployment. Day-to-day development and deployment activities were managed by SecurCom, who had multiple agencies to answer to on a not always coordinated schedule. This became apparent with multiple demonstrations and site visits that impacted participant satisfaction. In addition, SecurCom had to cover significant funding shortfalls for the period of time that it took to rectify funding issues.
Develop an incentives program that stimulates broader testing of the system. The recurring themes throughout this evaluation were the limited staff resources and the duplication of effort. If regulatory agencies are unable to provide waivers during FOTs (which is understandable), then the funding agency should consider reimbursing the participants for time spent on test activities to stimulate system testing. This could be promoted by tax incentives, staff incentives, etc.
REFERENCES
Draft Final Report, Phase II: Developing and Testing an Electronic Supply Chain Manifest, prepared by The ATA Foundation, prepared for the Federal Aviation Administration and U.S. Department of Transportation Office of Intermodalism, September 2002.
Electronic Intermodal Supply Chain Manifest ITS Field Operational Test Evaluation Plan, prepared for the U.S. Department of Transportation and Federal Highway Administration by Science Applications International Corporation, July 12, 2000.
Electronic Intermodal Supply Chain Manifest ITS Field Operational Test Evaluation Detailed Test Plans U.S. Department of Transportation and Federal Highway Administration by Science Applications International Corporation, June 8, 2001.
O’Hare Air Cargo Security Access System: Testing the Effectiveness of Biometric Smart Card Security, prepared for the Federal Aviation Administration, William J. Hughes Technical Center, June 2000.
Biometric Identification Standards Research Final Report (1997) by Dr. James Wayman.
APPENDIX
O’Hare Electronic Manifest Participant Pre-Test Questions
Activities To Be Timed At Motor Carriers/Freight Forwarders
Activities To Be Timed At Manufacturers
Activities To Be Timed At Air Cargo Carriers
Electronic Supply Chain Manifest
O’Hare Electronic Manifest Participant Pre-Test Questions
Shipper Questions
- Briefly describe your operation.
What do you manufacture?
What are your key raw materials?
Where are your primary suppliers located?
Where are your customers located?
What motor carrier(s) do you use?
What airline(s) or air freight forwarder(s) do you use?
How many air cargo shipments do you ship per day/week/month? How do you define a shipment?
What is your average shipment value?
For pick-ups, in general how much product goes out with a given truck? Do you ship truckload or less-than-truckload quantities?
How do you manage your inventory (JIT, VMI, customized orders, warehouse)? How many days of inventory to you maintain?
- What percent of your outbound shipments use air transportation and why?
- Do you currently have an automated manifest generation process? Yes or No. Please describe your system or process.
- What type of computer system do you use (operating system, software, hardware, etc.)?
- How do you communicate with your customers and your carriers? (e.g., what is the shipment status, when was the product shipped? Where is it now? What is the estimated time of arrival?)
- Do you have a back-up system for any or all of your communication network? Yes or No. If yes, please describe.
- How many requests do you get on average from customers requesting shipment status/location information?
- Have you experienced an increase in requests for shipment status over the last 2 years? Yes or No. If yes, how do you measure it? Why do you believe this is occurring?
- Briefly describe the shipment pick-up process. What are the primary activities undertaken to prepare the shipment? How long is the shipment ready before it is picked up? Is the pick-up a regularly scheduled stop? How long does it take to load the truck?
- Is there any part of this process that is frustrating or inefficient? Yes or No. If yes, what part?
- What security protocols do you have in place to ensure shipment security from pick-up to delivery at the air cargo operation?
- Are you aware of any delays at the point of transfer from truck to air? Yes or No. If yes, please describe.
- What shipping/inventory processes do you collect performance data on? What do you do with the data? Is there any additional data you would like to have access to?
- Do you have any plans for investing in new technologies to facilitate your operation? Yes or No. If yes, what will you invest in? [NOTE: “New” is defined as a product that changes the way you do business; it does NOT include standard upgrades in software and/or hardware]
- What benefits do you believe might arise from this pilot test?
Motor Carrier Questions
- Briefly describe your operation
- What is the primary type of freight you haul?
- Do you have fixed routes? Yes or No.
- How many pick-ups do you make per day? What percent of your pickups are destined for O’Hare for air freight forwarding?
- Where are your customers (manufacturers) geographically located who use air cargo services out of O’Hare (town/city and state)?
- Do you provide service to more than one airline or air freight forwarder? Yes or No. If yes, how many? Who are they?
- What is the primary type of freight you haul?
- Do you currently have an automated process to handle bills of lading? Yes or No. Please describe your system or process.
- Has this process changed significantly in the last 2 years? Yes or No. If yes, how has it changed?
- What type of computer system do you use (operating system, software, hardware, etc.)? Do you have mobile data terminals in your power units?
- Briefly describe the shipment pick-up (at manufacturer’s site) and drop-off (air freight forwarder’s site) processes. How long do they take? What do they involve?
- How do you communicate with your customers (manufacturers and freight forwarders)? (e.g., what is the shipment status, when was the product shipped? where is it now? what is the estimated time of arrival?)
- Do you have a back-up system for any or all of your communication network? Yes or No. If yes, please describe.
- How many requests do you get on average from customers (manufacturers) requesting shipment status/location information?
- Do you communicate with your drivers en-route? If yes, how? (telephone, fax, email, radio, OBC, etc.)?
- Do you have any security protocols in place to ensure shipment security from pick-up to delivery at the air cargo operation? Yes or No. If yes, what are they?
- Are you aware of any delays and/or frustrations at the point of transfer from either the manufacturer to truck or truck to air? Yes or No. If yes, please describe.
- Do you use any shipping data to track or evaluate performance? Yes or No. If yes, please describe them. If no, are there areas of your operation you would like to track?
- Do you have any plans for investing in new technologies to facilitate your operation? Yes or No. If yes, what will you invest in? [NOTE: “New” is defined as a product that changes the way you do business; it does NOT include standard upgrades in software and/or hardware]
- What do you think the potential benefits might be from participating in this pilot test?
Airline/freight forward Questions
- Briefly describe your operation.
- [Airline specific] Do you use freighters or passenger planes for your freight shipments? If you use both, are there differences between the two operations? Yes or No. If yes, please describe.
- Where are your shippers and receivers located?
- How many different trucking companies provide service to your Chicago facilities and who are they?
- How many air cargo shipments do you ship per day/week/month? How do you define a shipment?
- How do you schedule carriage of freight in planes? What is the service schedule (daily, hourly, etc.)?
- Do you currently have an automated air bill generation process? Yes or No. Please describe your system or process.
- Has this process changed significantly in the past 2 years? Yes or No. If yes, how has it changed?
- What type of computer system do you use (operating system, software, hardware, etc.)?
- Describe the shipment drop-off process (truck to air facility). How long does it take? What does it involve?
- Who are your customers (manufacturers, motor carriers, airlines, receiver)?
- How do you communicate with your customers (as defined above)? (e.g., what is the shipment status, when was the product shipped? where is it now? what is the estimated time of arrival?)
- Do you have a back-up system for any or all of your communication network? Yes or No. If yes, please describe.
- Has this changed significantly in the past 2 years? Yes or No. If yes, how has it changed?
- How many requests do you get on average from customers requesting shipment status/location information?
- Briefly describe your security protocols. Do you have any security protocols in place in addition to FAA requirements? Yes or No. If yes, please describe.
- Are you aware of any delays and/or frustrations at the point of transfer from truck to air? Yes or No. If yes, please describe.
- Do you use any shipping data to track or evaluate performance? Yes or No. If yes, please describe them. If no, are there areas of your operation you would like to track?
- Do you have any plans for investing in new technologies to facilitate your operation? Yes or No. If yes, what will in invest in? [NOTE: “New” is defined as a product that changes the way you do business; it does NOT include standard upgrades in software and/or hardware]
- What do you think the potential benefits might be from participating in this pilot test?
Activities To Be Timed At Motor Carriers/Freight Forwarders
Company Name: _______________________________________________________
Follow-Up Contact Name/Phone: __________________________________________
Date: ________________ Collected By: ____________________________________
Activity: Bill of lading input by routing clerk to create master manifest for delivery to airline (completion of master manifest). Describe process and degree of automation, etc. _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
Time to Conduct Activity |
|
|
Activities To Be Timed At Motor Carriers/Freight Forwarders
Company Name: ______________________________________________________
Follow-Up Contact Name/Phone: _________________________________________
Date: ________________ Collected By: ___________________________________
Activity: Load/paperwork acceptance at motor carrier terminal. Also note overall time from arrival at motor carrier of pickup to outbound shipment to airline (operational process, timing for context). _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
Time to Conduct Activity (Time from Arrival of Driver to Acceptance |
|
|
Activities To Be Timed At Motor Carriers/Freight Forwarders
Company Name: _______________________________________________________
Follow-Up Contact Name/Phone: __________________________________________
Date: ________________ Collected By: ____________________________________
Activity: Reproduction of Manifests. Also describe whether the process is batch or on-demand-operational process, time gaps-for context. _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
|
|
|
Activities To Be Timed At Motor Carriers/Freight Forwarders
Company Name: _______________________________________________________
Follow Up Contact Name/Phone: __________________________________________
Date: _____________________________ Collected By: _______________________
Activity: Contact Airline and Arrange Shipping
Telephone _____ Fax _____ Email _____
Is there any confirmation procedure for non-telephone options? If yes, describe. _______________________________________________________________________________________________________________________________________________________________________________________________________________
Describe whether the process is batch or on-demand-operational process, time gaps, confirmations-for context.
_______________________________________________________________________________________________________________________________________________________________________________________________________________
|
|
|
|
Activities To Be Timed At Motor Carriers/Freight Forwarders
Company Name: _______________________________________________________
Follow-Up Contact Name/Phone: __________________________________________
Date: ________________ Collected By: ____________________________________
Activity: Paperwork Errors – Time to correct. How are errors detected, at what stage of the process, and how are they corrected? _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
|
Number and Types of Errors |
|
Activities To Be Timed At Manufacturers
Company Name: _______________________________________________________
Follow-Up Contact Name/Phone: __________________________________________
Date: ________________ Collected By: ____________________________________
Activity: Filling Out Manifest. Describe process and degree of automation.
_______________________________________________________________________________________________________________________________________________________________________________________________________________
Observation # |
(Time to Complete One Manifest) |
Number of Forms or Orders Processed |
Notes |
Activities To Be Timed At Manufacturers
Company Name: _______________________________________________________
Follow-Up Contact Name/Phone: __________________________________________
Date: ________________ Collected By: ____________________________________
Activity: Shipper contacts Motor Carrier(s) to arrange load pick-up.
Telephone ______ Fax______ Email ______
Is there any confirmation procedure for non-telephone options? If yes, describe.
_______________________________________________________________________________________________________________________________________________________________________________________________________________
Observation # |
(Time Required to Contact one Motor Carrier) |
Number of Forms or Orders Processed |
Notes |
Activities To Be Timed At Air Cargo Carriers
Company Name: _________________________________________________
Follow-Up Contact Name/Phone: ____________________________________
Date: ________________ Collected By: ______________________________
Activity: Load Acceptance – Driver at Counter to Acceptance of Load. Describe process and record any stack-up or queuing of drivers at peak times. _______________________________________________________________________________________________________________________________________________________________________________________________________________
Observation # |
Time to Conduct Activity |
Blank |
Notes |
Activities To Be Timed At Air Cargo Carriers
Company Name: _________________________________________________
Follow-Up Contact Name/Phone: ____________________________________
Date: ________________ Collected By: ______________________________
Activity: Clerical Time for creating airplane-load documentation. Describe process and degree of automation, etc. _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
Time to Conduct Activity |
Number of Manifests to be Processed (To create |
|
Activities To Be Timed At Manufacturers
Company Name: _________________________________________________
Follow-Up Contact Name/Phone: ____________________________________
Date: ________________ Collected By: ______________________________
Activity: Driver arrives, shipping clerk produces shipping documentation, verifies driver/carrier identification, and driver signs off accepting load. Describe process. _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
Time to Conduct Activity (Time from Arrival to Identification, then time for Driver to Accept Load; Note Overall Time Driver is There) |
Number of Forms or Orders Processed |
|
Activities To Be Timed At Manufacturers
Company Name: _________________________________________________
Follow-Up Contact Name/Phone: ____________________________________
Date: ________________ Collected By: ______________________________
Activity: Paperwork errors - Time to Correct. How are errors detected, at what stage of the process, and how are they corrected? _______________________________________________________________________________________________________________________________________________________________________________________________________________
|
Time to Conduct Activity (Time to Complete One Manifest) |
Type of Error |
|
Electronic Supply Chain Manifest:
Participant Perceptions – Interview Guide
Purpose of interview: to collect user perceptions about the ESCM, their experiences during participation in the operational test, and any comments or recommendations they may have regarding improvements in the system or test process.
Expected time to conduct interview: 15-20 minutes.
Interview Questions:
- Describe overall deployment at your company. Were manifests handled simultaneously in both existing and test systems? If not, explain typical daily operation.
- How many air cargo manifests/shipments does your company process?
a. ____ per ______ (day/week/month)
- How many manifests have you entered into the system? (Total? Last month? Last week? Did interest in the system wane over the course of the test? What are the primary reasons more weren’t entered (lack of time or staff, lack of satisfaction, system errors, etc.)?
- Has your operations staff (or you) found the system easy to incorporate into the staff’s (your) daily routine? If not, what was their biggest complaint?
- How much additional work (how is this measured? Man-hours?)Have they (you) had to undertake to participate in this test? (Differentiate between extra time spent in training and extra time spent operationally).
- Please describe the major differences between this system and your existing process (different steps, time to process, transfer of shipments, etc.).
- Did the system adequately duplicate or improve your existing system or process? If yes, how? If not, why/how not?
- Can you identify any specific examples that illustrate the success and/or failure of the system?
- Were there any times you were aware of that the system did not function (server crashes, PC problems, Internet access issues, etc.)? If so, what was the problem? Generally how long did it take to have the problem resolved?
- Did you track shipments by viewing manifests in the system? If yes, how often? If no, why not?
- Do you feel the system has allowed for better or easier scheduling of shipments? Why? If it hasn’t made scheduling easier, what elements of the system can be changed to do so? If you had a new employee responsible for scheduling, would this system be easier to learn and master than your current system? Why or why not?
- Do you feel the system has allowed (or could allow, if expanded) for better service to your customers? If yes, how? If no, why not?
- Do you feel the system has been secure in protecting information? If yes, why? If no, why not? Is the system more or less secure than your current system? Why or why not?
- Do you feel the system can save time/effort? If yes, why? If no, why not?
- Do you feel the system can improve cargo security? If yes, how? If no, why not?
- What were your expectations for the system at the beginning of your participation in the operational test?
- Based on your participation in this operational test, has the system met your expectations? If yes, how? If no, why not?
- Was the training adequate? Why or why not? Describe training received.
- Was the technical support adequate? If not, why? Describe any instances where support was requested or provided.
- Do you have any specific recommendations for how the system can be improved? If yes, what recommendations? If no, why not?
[1] Electronic Intermodal Supply Chain Manifest ITS Field Operational Test Evaluation Plan, prepared for the U.S. Department of Transportation and Federal Highway Administration by Science Applications International Corporation, July 12, 2000.
[2] Ibid.
[3] Draft Final Report, Phase II: Developing and Testing an Electronic Supply Chain Manifest, September 2002, prepared by ATA Foundation, for the FAA and FHWA.
[4] Average Time excludes Manifest # 89.
[5] O’Hare Air Cargo Security Access System: Testing the Effectiveness of Biometric Smart Card Security prepared for the Federal Aviation Administration, William J. Hughes Technical Center, June 2000.
[6] O’Hare Air Cargo Security Access System: Testing the Effectiveness of Biometric Smart Card Security, prepared for the Federal Aviation Administration, William J. Hughes Technical Center, June 2000.
[7] The International Fingerprint Benchmark Test is described in the Biometric Identification Standards Research Final Report (1997) by Dr. James Wayman. (The report is available online at: http://www.engr.sjsu.edu/biometrics/publications_fhwa.html.)
[8] The optical scanner used to capture the fingerprint images was compliant with the U.S. Criminal Justice Information Services "Interim IAFIS Fingerprint Image Quality Specifications for Scanners," CJIS-RS-0010v4, Appendix G.
[9] Identix has stated that this feature template generation rate is consistent with other tests using images acquired from an Identix scanner.
[10] In this regard, minutiae-based fingerprinting is different from other biometric methods that use distance measures in place of "similarity" scores. Opposite to the similarity scores, the distance measure decreases with increasing similarity of the compared images.
[11] Biometric Identification Standards Research Final Report (1997) by Dr. James Wayman.