The National Training Evaluation Program (NTEP) will enable the Office of Training and Development (OTD) to provide valuable information about the quality of Customs-based training programs. The training evaluation model being implemented is based on the work of Dr. Donald L. Kirkpatrick, a former national president of the American Society for Training and Development. This model - a four-level hierarchy - is widely used in government and private industry.

During FY 2002, OTD will pilot the use of Level 1, Reaction, data collected from students immediately after taking a Customs training course, and Level 3, Training Transfer, data collected between 3 to 6 months after the training event has occurred.

• Level 1, Reaction, data typically contains information on the students' thoughts and/or feelings about the training course, including the training materials and content, instructors, facilities, and methodology.
• Level 3, Training Transfer, data involves both students' and their managers' evaluation of training and measures on-the-job changes in work behaviors and performance as a result of the training event.

Outlining NTEP
The NTEP evaluation model will initially collect Level 1 and Level 3 data through a combination of scannable paper evaluation forms, electronic data input, telephone response, and transcribed hand written comments provided by students and their managers. The data inputs will be stored in a relational database, and reports will be generated and distributed via email, a secure Internet Web site, and other means available to OTD.

The program involves the following steps:

1. Customs training coordinators and instructors will request that students complete the Level 1 evaluation, batch the completed surveys by class number, and mail them to the evaluation contractor. Blank forms will be stocked by the National Distribution Center.
2. Level 3, Training Transfer, evaluation surveys will be sent electronically (via e-mail) to the students of pre-selected courses of interest and their supervisors 3 to 6 months after attending training. Students and managers will be able to supply answers to the contractor anonymously through an e-mail, Internet, or telephone format.
3. An evaluation team will process and analyze the forms. The information will be entered into a centralized database that will allow for the reporting of evaluation results to training customers through paper and/or electronic methods including e-mail and an access-only Internet Web site.

Going beyond a typical evaluation program
A trend analysis of traditional evaluation data is not always sufficient for capturing and presenting all of the impacts of training on the Customs workforce. Once NTEP is up and running, OTD will be looking into various new ways to determine the effect of training on performance throughout the organization. For example, future evaluation programs studying the effect of training on numerous performance criteria within Customs will allow us to analyze the results of our training programs on how well we do business as an organization. Some of these results can include costs savings, work output improvements, and quality changes.

OTD is always interested in determining new ways to measure the benefit of training to the organization. Therefore, it is very important to hear success stories from employees, the consumer of Customs training. Please send an e-mail to the author in the Office of Training and Development and state how training has helped to improve your skills or the skills of your work group. Let us know how training has helped you do your job better and make Customs a more efficient and productive organization.

Based on training evaluation data, instructors can improve skills, eliminate outdated material, and develop new training. In the long run, NTEP will have a significant impact on the overall cost-benefit analysis of training, helping to ensure that Customs receives the greatest return on its investment. Table 1. The Kirkpatrick Model of Training Evaluation Level and Evaluation Data Obtained 1. Reaction Do the training materials, course content, and teaching methods contribute to learning what is needed on the job? Are instructors prepared to teach and do they exhibit good teaching skills? Are the training facilities suitable for learning? 2. Learning Was there an increase in participant knowledge obtained from training, as measured by testing the participants? 3. Training Transfer Are the participants able to use the training on the job? Did the participants change their job-related behavior based on what was learned at the training? Did the training have an impact on work performance? 4. Results Did the change in work behavior positively impact the organization?