Test Plan For Validating A Context Driven Integrated Model (CDIM) For Sheet Metal Die Design August 26, 1991 Kevin K. Jurrens National Institute of Standards and Technology (NIST) Factory Automation Systems Division CDIM SM1 Test Team Leader for the PDES, Inc. Sheet Metal Project Key Words: CDIM, Die Design, PDES, Product Data, Sheet Metal, SM1, STEP, Test Plan Acknowledgement This work was performed for the National PDES Testbed, which was established at the National Institute of Standards and Technology (NIST) in 1988 under sponsorship of the U.S. Department of Defense Computer-aided Acquisition and Logistic Support (CALS) program. The plan outlined in this document pertains to the PDES, Inc. Sheet Metal Project CDIM SM1 effort, for which NIST serves as a government associate. The author also wishes to acknowledge and thank the following Sheet Metal Project and CDIM SM1 Test Team members for reviewing this document and providing numerous suggestions for improvement: Frank Demasek - Electronic Data Systems Al Montano - Electronic Data Systems Phil Rosche - South Carolina Research Authority Gloria Roth Robinson - Electronic Data Systems Bill Shaffer - Boeing Jack Skeels - Product Data Integration Technologies, Inc. Shyhdong (Dong) Tsuei - Electronic Data Systems No approval or endorsement of any commercial product by the National Institute of Standards and Technology is intended or implied. The work described was funded by the United States Government and is not subject to copyright. Table of Contents 1.0 Introduction and Document Scope . . . . . . . . . 1 2.0 Background - CDIM SM1 Overview . . . . . . . . . 2 2.1 CDIM Purpose . . . . . . . . . . . . . . 2 2.2 CDIM Scope . . . . . . . . . . . . . . 2 2.3 CDIM Activities . . . . . . . . . . . . . 3 3.0 Test Purpose . . . . . . . . . . . . . . . 4 3.1 Test Scope . . . . . . . . . . . . . . 4 3.2 Test Objectives . . . . . . . . . . . . . 4 4.0 Test Methodology . . . . . . . . . . . . . . 5 4.1 Elements of CDIM Testing . . . . . . . . . . 5 4.2 Express-Based Testing Software Tools . . . . . . 7 4.3 B-Rep and Form Feature Testing Strategy . . . . . 9 4.4 IDEF-Based Testing . . . . . . . . . . . . 10 4.5 Logistics of CDIM SM1 Testing . . . . . . . . 11 4.6 Revision Control System (RCS) . . . . . . . . 12 5.0 Test Constraints . . . . . . . . . . . . . . 13 6.0 Test Requirements . . . . . . . . . . . . . . 15 6.1 Model Team Deliverables . . . . . . . . . . 15 6.2 CDIM Deliverable Evaluation Methods . . . . . . 16 7.0 Test Procedure . . . . . . . . . . . . . . . 19 7.1 Test Suites . . . . . . . . . . . . . . 19 7.1.1 Test Suite Context Diagrams . . . . . . 19 7.1.2 Test Scenarios . . . . . . . . . . . 20 7.1.3 Test Data . . . . . . . . . . . . 20 7.1.4 Test Cases . . . . . . . . . . . . 20 7.2 Test Execution . . . . . . . . . . . . . 21 7.3 Test Evaluation Criteria . . . . . . . . . . 21 8.0 Resources . . . . . . . . . . . . . . . . 23 8.1 People . . . . . . . . . . . . . . . . 23 8.2 Facilities . . . . . . . . . . . . . . 24 8.3 Software Tool Requirements . . . . . . . . . 25 8.4 Training Requirements . . . . . . . . . . . 26 9.0 Test Deliverables . . . . . . . . . . . . . . 28 9.1 CDIM SM1 Test Plan . . . . . . . . . . . . 28 9.2 Test Procedure Documentation . . . . . . . . 28 9.3 Test Result Documentation . . . . . . . . . 28 9.4 Test Issue Reports . . . . . . . . . . . . 29 9.5 Test Demonstration Capability . . . . . . . . 29 9.6 CDIM SM1 Testing Final Report . . . . . . . . 29 10.0 Issue Procedures . . . . . . . . . . . . . . 30 10.1 Issue Reporting . . . . . . . . . . . . . 30 10.2 Issue Tracking . . . . . . . . . . . . . 30 10.3 Issue Resolution Method . . . . . . . . . . 31 11.0 References . . . . . . . . . . . . . . . . 32 Appendix A: Testing Project Schedule . . . . . . . . . 34 Appendix B: CDIM SM1 Test Issue Log . . . . . . . . . 38 Appendix C: Test Suite Example . . . . . . . . . . . 40 Appendix D: List of Acronyms . . . . . . . . . . . 47 1.0 Introduction and Document Scope The Standard for the Exchange of Product Model Data (STEP) is an evolving international standard developed under sponsorship of the International Organization for Standardization (ISO). The objective of the STEP standard is to provide a complete, unambiguous, computer interpretable definition of the physical and functional characteristics of a product throughout its life cycle. PDES (Product Data Exchange Using STEP) is the name given to the United States development activity in support of this international standard. In order to accelerate the development, validation, and implementation of STEP, a joint industry/government consortium was formed in 1988 and incorporated in the state of Delaware as PDES, Inc. South Carolina Research Authority (SCRA) has been awarded the Host Contract to provide technical management for the PDES, Inc. program. The methodology used by PDES, Inc. to validate the STEP standard consists of testing the STEP resource models (i.e., information data models) from a very specific application point of view. This application viewpoint requires development of a Context-Driven Integrated Model (CDIM). A CDIM is an integrated model composed of data definitions and data construct relationships, constraints, etc. from the STEP resource models, which satisfy data requirements for a specified application context. The National Institute of Standards and Technology (NIST) has contributed to the initial development, execution, evaluation, and subsequent refinement of this STEP validation methodology [7,9]. It is anticipated that this methodology will form the basis for validation of future STEP Application Protocols (AP) [5]. This document defines the Test Plan that will govern testing of the first CDIM developed by the PDES, Inc. Sheet Metal Project (i.e., CDIM SM1). This Test Plan specifies the test objectives, methodology, constraints, requirements, evaluation criteria, resources, deliverables, and issue procedures for this testing activity. In addition, this document provides a brief introductory overview of CDIM SM1 for those readers unfamiliar with this Sheet Metal Project effort. 2.0 Background - CDIM SM1 Overview Due to the interest of PDES, Inc. member companies, the PDES, Inc. Sheet Metal Project was initiated during the first half of 1990. This project has focused on the creation of an international standard for the exchange and sharing of product data pertaining to design and manufacture of sheet metal parts and related tooling [14]. The first major effort of this project is named Context- Driven Integrated Model (CDIM) SM1. Full-scale development of CDIM SM1 began in September 1990, with participation from General Motors (GM)/Electronic Data Systems (EDS), Boeing, National Institute of Standards and Technology (NIST), Product Data Integration Technologies, Inc. (P.D.I.T.), and South Carolina Research Authority (SCRA). 2.1 CDIM Purpose The purpose of CDIM SM1 is to test and validate the portions of the STEP resource models that are within the scope of the defined CDIM application context. Within the CDIM SM1 effort, the Sheet Metal Project will produce a CDIM data model, testing criteria, and documented test results. These items will be used to further the development of STEP, as well as to provide much of the needed information for one or more sheet-metal-focused STEP Application Protocols (AP) [6]. 2.2 CDIM Scope The scope for CDIM SM1 is specifically defined as follows: CDIM SM1 will focus on standardization of the product data for a shared data environment to support the design of dies/blocks for sheet metal parts. This includes the interrelationships between the dies/blocks and the parts they fabricate. The CDIM will utilize existing STEP resource models. [16] This scope includes: Exchange of product definition data as it relates to die design, from receipt of a die request to the providing of corresponding die manufacturing data. Management of design change control data within the die design process. A focus on a shared database environment to reflect internal PDES, Inc. member company needs for product design and manufacturing automation support. Candidate STEP resource models to satisfy data requirements for this application context include Geometry, Topology, Tolerance, Product Structure Configuration Management (PSCM), Shape Representation, Form Features, Material, Manufacturing, and the resource model subsets that exist in PDES, Inc. CDIMs A1, A2, and B3/B4 [8,9,10,12,17]. In addition, CDIM SM1 will use the STEP Integration Framework models (i.e., Generic Product Data Model (GPDM) and Generic Enterprise Data Model (GEDM)) [1,2]. 2.3 CDIM Activities Before initiation of this Test Plan document, the following CDIM SM1 activities were completed: Development of an activity model for the process of "Prepare for Sheet Metal Part Production" [15]. This model is roughly equivalent to the Application Activity Model (AAM) of a STEP Application Protocol [6]. Development of a data planning model for the in-scope data as identified through the activity modeling process [15]. This model is roughly equivalent to the Application Resource Model (ARM) of a STEP Application Protocol [6]. Both of these models were reviewed extensively by sheet metal die design application experts from within PDES, Inc. and other automotive and aerospace companies. At the conclusion of the above activities, the CDIM SM1 project team then split into two parallel efforts: the Model Team, to develop the CDIM data model from existing STEP resource models, and the Test Team, to develop the test strategy as documented in this Test Plan, develop test suites, test scenarios, and test cases, obtain test data from industry sheet metal die design application experts, and perform the CDIM testing. The remainder of this document is concerned strictly with CDIM SM1 Test Team activities. For further information on other CDIM SM1 activities, the reader is directed to the CDIM SM1 Project Plan [16]. 3.0 Test Purpose This section outlines the scope and objectives of the CDIM testing process. The testing purpose and the CDIM purpose (as given in Section 2.1) are essentially equivalent. The testing activity provides the basis for evaluating the validity of the STEP resource models within the CDIM scope from an application viewpoint. Development of the CDIM model provides the viewpoint or context from which to test the STEP resource models. It is important to note that evaluation of the STEP resource models cannot occur without some context on which to base data requirements. 3.1 Test Scope The scope of the CDIM testing will include those activities defined as being in-scope by the CDIM SM1 Activity Model and the data elements defined by both the in-scope Inputs, Controls, Outputs, and Mechanisms (ICOMs) from this activity model and the CDIM SM1 Data Planning Model [15]. All attempts will be made to provide maximum test coverage of the CDIM data model within the constraints placed upon the testing team (see Section 5.0, Test Constraints). In a practical sense, sufficient test cases cannot be obtained or developed to completely test all possible aspects of the sheet metal die design process from every company's viewpoint. The task that is possible, however, is to test from several viewpoints using sheet metal die design application experts from different companies and industries to obtain a representative cross-section of sheet metal die design data requirements. This approach, along with a coordinated selection of test suite subject areas, will ensure that the test team will achieve maximum coverage of the CDIM data model. 3.2 Test Objectives The CDIM SM1 Test Team has identified several objectives for the CDIM SM1 testing activity. These objectives are provided below. Provide data to further STEP resource model validation. Identify deficiencies in the STEP resource models through analysis of the CDIM SM1 data model. Determine how well the CDIM SM1 model satisfies member company/industry data requirements (in the area of sheet metal die design). Identify areas of the CDIM SM1 model that have no corresponding industry data. 4.0 Test Methodology This section provides an overview of CDIM testing methodology, descriptions of the software tools used in the testing activity, and CDIM SM1 Test Team decisions regarding the logistics of testing and version control of test data. 4.1 Elements of CDIM Testing There are six primary elements involved in CDIM testing. These elements are described below in roughly the order in which they are performed. 1) Map Data. This activity is largely a manual operation which consists of mapping test case data to entities and attributes within the CDIM model. This mapping operation can be performed using either the IDEF1X wall chart or the Express version of the data model. A two-step mapping approach is used as described below to enable testers to validate high- level entity relationships before mapping detailed attribute- level test data. Step 1 - High Level Mapping - Mapping is performed at the entity level. Basic entity relationships are analyzed with the primary question to be answered during this step being, "Can I generally find entities within the CDIM model which correspond to my test case data, particularly for key test case elements?" This step concludes with a review of the mapping results by both the testing and modeling teams to validate the relationships of the test data to the entity definitions. The modeling team will comment on the mappings and the test team will compare individual test mappings to provide for consistent and/or compatible interpretations across different company scenarios. Step 2 - Detailed Mapping - Mapping is performed at the attribute level. This step determines if the CDIM model satisfies very detailed attribute data requirements (e.g., types, number, definitions, groupings, dependency) and/or detailed entity data requirements (e.g., cardinality relationships). A common technique used during this step is called "instancing". Instancing consists of inserting test case data into tables, database schemas, or the QDES software tool [11] in order to visually analyze the relationships between data elements. (Note: QDES, or Quick and Dirty Editor for STEP, is described in Section 4.2 of this document.) This technique helps verify that test case data is properly represented in the CDIM data model. This step also concludes with a review of each tester's mapping results by both the testing and modeling teams. Past PDES, Inc. CDIM efforts have shown that the mapping exercise is a significant portion of the overall testing task. This detailed examination of the CDIM documentation typically identifies several issues against the model's ability to represent the test case data. Each tester will also interpret the entity and attribute usages based upon the CDIM document. A comparison of each tester's interpretations will provide the basis for evaluating the clarity of the CDIM documentation. 2) Analyze CDIM Model Coverage. After completion of the mapping task, an analysis of the CDIM data model coverage is performed. This activity determines the extent of the model which is covered by available test data. This analysis assists in determining which entities will be populated into a testing database for further test execution. Entities in the CDIM which are not covered by test data need not be populated. In these cases, a test issue report will be written and a determination of the entity's status (i.e., remove from model, leave in model, search for additional test data, etc.) will be made based on input from the testing team, modeling team, and sheet metal die design application experts. 3) Implement the Database. This activity consists of creating a database schema from the CDIM model using STEP testing software tools (Note: These software tools are described in section 4.2 of this document). This operation alone is quite valuable in that it locates syntax and some logic errors within the CDIM model. 4) Populate Schema with Data. This activity consists of loading test case data into the database schema that was created in the previous "Implement the Database" activity. Test data can be loaded into the database through a number of methods. The most direct method of database population is by writing Structured Query Language (SQL) "Insert" commands to place data elements into the database tables. Alternatively, a STEP file to Oracle database load program (i.e., STEP Working Form to SQL, or STEPWF-SQL) [11], which is discussed in Section 4.2 of this document, can also be used to populate the database. The specific method used to perform this activity is typically dependent upon the format of the given test case data. If the data is in the form of a STEP physical file, the most obvious choice would be to use the STEPWF-SQL software tool to populate the database. 5) Write and Execute Database Queries. This activity consists of writing and executing SQL database queries to retrieve desired information from the database schema populated with test case data. The queries ask sample real-world questions which an enterprise must be able to answer to implement internal systems. Successful execution of these queries proves that an access path to those data elements and the necessary relationships between data elements are, in fact, supported by the CDIM model. Query execution will also be used to perform interoperability and "shared data" testing. Based upon the CDIM SM1 focus of a shared database environment, test case data from all testers will be populated into a single database instance. Queries written by one tester for specific populations of data will be executed against populations of data developed by other testers. This testing activity will verify consistency, usability, and data sharing aspects of the CDIM data model. An additional technique that uses SQL queries to assess the usability of the data model will also be performed. This technique uses the SQL "Update" facility to modify information in the database based upon the content of previously retrieved data. This "Update" exercise demonstrates that the data is usable from reasonable business perspectives. 6) Evaluate and Report Results. As each test result is obtained, the tester must make a comparison of this actual result with the expected result. The test result will be recorded, regardless of the outcome of the test. If the actual result does not match the expected result, a test issue report will also be written. This test result documentation is reported to the CDIM Model Team and is provided as one of the final deliverables from the CDIM Test Team. Current plans include three separate test/fix/test cycles to validate the CDIM SM1 data model. This means that three versions of the CDIM model will be tested by the test team. After each test iteration, all test results and test issue reports will be provided to the model team to incorporate changes into the next release of the CDIM model. 4.2 Express-Based Testing Software Tools The software tools available to facilitate the testing process as outlined above are based upon the Express language representation of a CDIM model. These software tools, which were jointly developed by NIST and PDES, Inc., have been used in past PDES, Inc. CDIM projects. An overview of the software tools to be used in CDIM SM1 testing is provided below [5,11]. 1) A set of software tools exists to perform the function of implementing the database. The Fed-X tool performs an initial syntax check of the CDIM Express model. The Fed-X-SQL tool generates SQL command statements (i.e., "Create" statements) from the CDIM Express model to form the database schema. Database load utilities are then used to initialize the database from these "Create" statements. 2) Another set of software tools exists to load test case data into the database schema. This test data must be in STEP physical file format. In most cases, test data must be pieced together because few STEP translators currently exist that can provide all information necessary for testing. A common route is to obtain geometry data from existing Computer Aided Design (CAD) systems with Initial Graphics Exchange Specification (IGES) translators and to then use the IGES to PDES (ITOP) translator test tool to form STEP physical files. Once in STEP physical file format, the test data can be loaded into the database schema using a STEP Working Form to SQL database loader software tool (i.e., STEPWF-SQL). 3) Due to the limitations of IGES, existing IGES translators, and the IGES to PDES translator, all test case data cannot be obtained from IGES files. A set of test tools addresses the need to interactively create or modify test case data. The Quick and Dirty Editor for STEP (QDES) tool is used to create, modify, or otherwise manipulate test case data. Items such as tolerance, material, or product structure are typically added using QDES to form a complete test case. The Fed-X-QDES tool is used to establish an image of the Express model necessary to execute QDES. Also, the STEPPARSE-QDES tool is used to load existing STEP physical file data into the QDES editor. The output of QDES is once again a STEP physical file to be loaded into the database using the STEPWF-SQL tool. 4) The software tools available to aid execution of SQL queries consist strictly of database utilities. No tools currently exist to aid evaluation of the query results, which may consist of an "answer" to the query and/or database software error/warning messages. 5) Other STEP validation software tools exist that may also prove valuable during the testing process. The STEPPARSE-STEP software tool is used to verify the syntax of a STEP physical file against the structure of an Express model. The VISUALIZER software is used to graphically display a STEP physical file and to test the validity of geometry entities. OTON (i.e., old to new) is a tool that updates a geometry schema from the PDES, Inc. "GEOM1B" representation to the ISO "pre-St. Louis" geometry version. RENUMBER is a tool that renumbers the entity identifiers within a STEP physical file so that test cases can be formed by appending two STEP files together. GETSTEP locates the "parent" or "child" entities of a specified entity in a STEP physical file. (Note: This paragraph provides a sampling of the miscellaneous software tools available to the tester and is not intended to be all- inclusive.) 4.3 B-Rep and Form Feature Testing Strategy Two separate methods will be used to test both the Boundary Representation (B-Rep) solid geometry and Form Feature portions of the CDIM model. Each of these two methods validates a different aspect of the data content. Validation of the B-Rep and Form Feature constructs within the CDIM model (and also to some extent, the geometry and topology constructs) must use somewhat different techniques due to the basic focus of the underlying STEP resource models. Testing of other portions of the CDIM is done from what is known as the "external" view, or user view, of the data. The CDIM SM1 Test Team believes that testing of the B-Rep and Form Feature constructs within the CDIM will be more meaningful if done from an "internal" view, or application system view. For this reason, the following test strategy is proposed. The first test method consists of comparing the internal schemas of existing B-Rep/Form Feature system implementations with the STEP representation. As the STEP resource models should support any reasonable mapping from any system implementation, this technique is used to validate the correctness of the STEP representation. In effect, this activity performs a rough "requirements analysis" phase of a STEP translator development and also evaluates the potential success of an exchange of B-Rep/Form Feature STEP data between modeling systems. It is recognized, however, that this method has a high probability of failure due to inability to obtain the internal schemas of existing modeling system implementations. In most cases, this information is considered to be of a proprietary nature. If these schemas cannot be obtained for at least two existing systems, this test method may not be performed. The second test method will validate the relationship of the B-Rep and Form Feature entities to other parts of the CDIM. This method is performed using a similar test methodology as used for the rest of the CDIM (i.e., obtain test case data, perform mapping of data to the CDIM model, load data into database, execute SQL queries, etc.). As described below, however, this testing will require use of an additional software tool to generate B-Rep STEP test data. Unfortunately, automated tools are not available to generate Form Feature STEP test data. Thus, Form Feature testing by this method will most likely require creation of the data using QDES or hand- population of the database using the SQL Insert utility. Initial geometry for testing B-Rep will be obtained from solid models of appropriate test objects (i.e., sheet metal parts and/or dies) from solid modeling systems available within GM/EDS and Boeing. In order to generate the desired B-Rep STEP test data, this data must then be transferred to a Parasolid modeling system. (Note: A Parasolid system at NIST may be used during this phase if one is not available at GM/EDS.) This transfer may consist of creating solid models of appropriate test objects within Parasolid or importing the data to Parasolid from the other modeling systems. The feasibility of importing data to Parasolid from internal member company systems where sheet metal part and die designs are actually located will be evaluated by the CDIM SM1 Test Team. Once the appropriate Parasolid solid model is created, a software tool developed at NIST called the Parasolid to STEP translator [4] will be used to create STEP physical files which contain the B-Rep data. This STEP physical file can then be used in the testing process, just as any other STEP file test case data. This Parasolid-based tool was selected as the only current practical source of B-Rep test data because, although other modeling systems are available at member companies, this process provides test data in a format that is readily usable in the existing test methodology. The Parasolid to STEP translator has some limitations that must be addressed by the test team. These limitations are outlined below. The only surface types currently supported by this tool are the plane, sphere, cone, cylinder, and torus (i.e., no sculptured surfaces). All surface edges must be represented by a line, circle, or ellipse (or portions of these entity types). The STEP physical file format output by this tool was written to comply with a previous version of the STEP standard (i.e., the "Tokyo version"). Software support may be required from PDES, Inc. to update this STEP physical file to comply with the latest version of the STEP standard. Validation of the B-Rep aspects of the CDIM model through this method will also require that the tester be familiar with the concepts of solid geometry product representation and be able to compare and critique the mapping between the test case product model and the resulting STEP file and/or database schema. 4.4 IDEF-Based Testing An original objective of the Sheet Metal Project was to evaluate portions of the CDIM SM1 model using a testing methodology based upon the IDEF1X representation. This test methodology had not been used in previous PDES, Inc. CDIM projects and, as such, software tools did not exist to facilitate this testing. General software tool requirements were developed to illustrate the feasibility of this testing approach. These requirements were recorded in the Sheet Metal Project software tools requirements document [13]. The focus of CDIM SM1 testing has been modified somewhat since this initial plan. Primary CDIM testing activities will be done using the Express-based approach, as previously described. Some "investigative" IDEF-based testing of CDIM SM1 is still planned, however, during the later stages of testing, most likely during the third testing iteration. The purpose for this testing will be to verify that test results obtained previously from the Express model can be duplicated when analyzing the IDEF1X model. Results of this testing would provide tangible evidence (which has previously been nonexistent) to validate the CDIM model conversion process from IDEF1X to Express. This evidence would also reflect upon the conversion process of the STEP resource models. The IDEF-based testing concept is basically the same as that used in past CDIM testing, except that the database schema is derived from the IDEF1X representation, rather than Express representation. The concepts of creating the database schema, obtaining or editing test case data, loading the test case data into the database schema, and performing database queries to extract desired information are all still valid. As defined in the tools requirements document, new or modified software tools to support these activities would be beneficial. 4.5 Logistics of CDIM SM1 Testing Five options have been identified concerning the logistics of CDIM SM1 testing. Each option describes a method for accessing the NIST National PDES Testbed (NPT) in Gaithersburg, MD, and/or the SCRA PDES Development Lab (PDL) in Charleston, SC. The goal of the CDIM SM1 Test Team is to minimize travel and to maximize use of remote computer access to perform testing activities. The five options are listed below, with specific considerations listed in parenthesis after each item. 1) X-Windows terminal (X-terminal) remote access (9600 baud connection available through modems supplied by SCRA, QDES and Visualizer software not yet available through this connection, X-terminal hardware must be obtained, some travel required) 2) Internet network access (available at NIST and SCRA, need T1-1.6Mb capability, some tester sites do not have Internet) 3) Personal Computer (PC) remote access (9600 baud connection available through modems supplied by SCRA, some window emulation available with PC-EMACS software tool, QDES and Visualizer software not available, some travel required) 4) Local access of QDES and Visualizer software, PC remote access for other tools (limited travel, need software installed at tester companies) 5) All testing done locally at NIST or SCRA Testbeds (extensive travel) The CDIM SM1 Test Team decision is to pursue Option 4 above as the top priority, with Option 3, and then Option 5 as fall-back positions. Options 3 and 5 are actually status quo and are being used by other PDES, Inc. CDIM projects. The X-terminal option did not provide enough functionality increase to warrant selection over a PC. The CDIM SM1 Test Team has documented and distributed its evaluation of the X-terminal connection capability. The CDIM SM1 Team also plans to continue to monitor progress on establishing X-terminal connection to the testbeds. The Internet option could not be selected due to the lack of Internet connection capability at member company sites. The CDIM SM1 Test Team will gather information to assess the feasibility and impact of installing the QDES and Visualizer tools locally at each tester's member company site. Hardware and software requirements/availability and required implementation schedule will also be determined. If the local access option proves feasible, one suggestion that has been proposed is that during the testing process, the QDES image of the CDIM schema could potentially be generated at one facility and shared among the other sites. This practice would save many hours for each tester not involved in generating this image. 4.6 Revision Control System (RCS) The CDIM SM1 Test Team will investigate use of the Revision Control System (RCS) to manage version control of test data. RCS is a configuration management tool that is available at both the NIST and SCRA testbeds and has previously been used to manage version control of documentation [3]. Past PDES, Inc. CDIM testing efforts have found version control of test data to be one of the largest problems to overcome. Proper versions of STEP physical file test cases, Express files, database schemas, SQL queries, etc. must be used together to ensure that any given test yields meaningful results. The CDIM SM1 Test Team will identify the test data types to be controlled, procedures to be established, and interaction between potentially multiple testing sites after gaining more experience in the testing process and in using the RCS utility. 5.0 Test Constraints Several items have been identified as constraints to the CDIM testing activity. These constraints serve to limit both the quantity and quality of CDIM test results. All efforts will be made to overcome obstacles in the testing process, however, and to meet all test objectives. These constraints are listed below primarily to provide explanation for decisions made in this test plan regarding test methodology, CDIM model coverage, etc. Complexity of Test Data - As test data becomes more complex, the task of selecting appropriate test data to form specific test cases becomes more difficult. Some of the available die design-related test data is anticipated to be quite complex, especially in the area of geometry representation (i.e., surfaces and B-rep solids). Although a thorough understanding of the mathematics behind these representations is not actually necessary, the tester must understand the test data well enough to compose meaningful tests and to interpret test results. Similar complex test data situations may arise due to differences in terminology between companies and in unique data combinations, such as in internally-developed forms and documents. Availability of Test Data - Test data must be obtained from the sheet metal die design application experts in order to test the accuracy and completeness of the CDIM model. If appropriate test data is not available to test specific aspects of the model, these portions of the CDIM will not be validated. This constraint is particularly relevant when testing specific "to-be" aspects of the die design process that are included in the CDIM scope (as opposed to the "as-is" die design process). Software Tools Limitations - The testing process will be constrained by software tools in terms of: Availability - Does a tool exist to perform a desired function and can the tester access it? Capability - How well or complete does a given tool perform a desired function? Applicability - Does a given tool perform the actual function needed for a specific test activity? These software tool limitations largely determine the time required to perform a testing activity (and indirectly the practicality of that testing activity). If appropriate software tools are not available when needed to perform specific phases of the testing process, CDIM SM1 testing will not obtain the desired results in the allotted time. Project Schedule - CDIM SM1 testing will also be constrained by the amount of time provided in the project schedule. Test results must be obtained, documented, and presented in a timely manner in order for project management to continue support and funding for this and subsequent CDIM activities. As the PDES, Inc. CDIM methodology is quite schedule-driven, testing may have to be halted before total completion in order to meet deliverable due dates as specified in the project schedule. The CDIM SM1 project schedule is provided for reference in Appendix A. Available Resources - CDIM SM1 testing will be performed using available resources, specifically those human resources assigned to the CDIM Test Team. Any increase in human resources (within reason) assigned to the testing activity would increase the amount of CDIM model test coverage and decrease the time required to accomplish a set of testing tasks. A decrease in human resources would have the opposite effects. 6.0 Test Requirements This section documents the expected deliverables from the CDIM Model Team and the methods by which these items will be tested. 6.1 Model Team Deliverables The CDIM SM1 Test Team expects to receive the following deliverables from the CDIM SM1 Model Team. Each deliverable will be tested using the methods outlined in Section 6.2, CDIM Deliverable Evaluation Methods. 1) Introduction/Overview Document 2) CDIM Express model - Copies of this model should be provided in both paper and electronic form. The electronic copy should compile error-free using the Fed-X software tool. 3) Glossary/Definitions - Definitions of the data model entities, attributes, terms, relationships, etc. should be provided as either an appended glossary or imbedded in appropriate locations throughout the body of the CDIM document. 4) Data Population Examples - The purpose for these examples is to demonstrate entity usage. These population examples can be based upon either the Express or IDEF1X models and can consist of the population of tables or STEP physical files. 5) IDEF1X model - One IDEF1X wall chart should be provided for each tester site. The IDEF1X model should be fully-attributed and normalized to remove many-to-many relationships. Definitions are not required in the documentation of the IDEF1X model, except to explain items created during the normalization process. (Note: definitions of the other items will already exist in the Express model documentation.) In addition, an electronic copy of the IDEF1X Structural Modeling Language (SML) format should be provided for all versions of the CDIM model except for V0.1. The SML is used as input to a software tool called Leverage and will be used during the IDEF-based testing phase. In addition, the CDIM SM1 Model Team will also provide a walk- through of the CDIM model delivered to the test team. This walk- through will at a minimum include presentations on the CDIM Express model and IDEF1X wall charts. Additional model walk-throughs may be required during later stages of the testing process as deemed necessary by the test team. Other agreements between the model team and test team regarding delivery of the CDIM model are as follows: The model team will select specific baseline versions of the STEP resource models and Express Specification. The test team will use this information to determine the impact on the testing software tools. The initial release of the CDIM (V0.1) will not contain entities not already found in the STEP resource models. The model team will instead provide a tentative issue to the test team for verification using the test data. The test team will validate that a gap actually exists in the CDIM data model. 6.2 CDIM Deliverable Evaluation Methods The following are methods or techniques which will be used to test the CDIM deliverables. Build Schema - Build a database schema from the Express model. Compare - Compare the IDEF1X model to the Express model. Compile - Compile the Express model using the Fed-X software tool. Critique - Read, analyze, and evaluate the test item. Execute - Perform the tasks in the data population example. Map - Find places in the model for the test data. Populate - Load test data into the database. Query - Retrieve data from the database using SQL statements. Query Path Analysis - Analyze queries that retrieve the same or similar data. Update - Modify data in the database using SQL statements. Each part of the CDIM data model documentation packet will be tested using the techniques outlined below. 1) Introduction/Overview Document - Critique 2) CDIM Express model - Compile, Build Schema Characteristics of the Express model to be evaluated: a) Entities - Critique, Map, Populate b) Relationships - Critique, Map, Populate Characteristics of relationships to be evaluated: Existence Semantics Cardinality Dependence c) Attributes - Critique, Map, Populate Characteristics of attributes to be evaluated: Existence - missing - extraneous - duplicate/redundant Semantics - wrong type - poorly named Key vs. NonKey - dependency - uniquely identifying - adequate - minimally sufficient d) Keys - Critique, Map, Populate (i.e., Express constraints) Characteristics of keys to be evaluated: Dependency Uniquely identifying - adequate - minimally sufficient e) Cardinalities - Critique, Map, Populate, Compare f) Categorizations - Critique g) Nomenclature - Critique h) Functions and "derived" information - Critique i) Presentation - Critique j) Redundancy - Query Path Analysis k) Usability - Query, Update l) Rules - Critique, Map, Populate 3) Glossary/Definitions - Critique 4) Data Population Examples - Critique, Execute 5) IDEF1X model - Critique Characteristics of the IDEF1X model to be evaluated: a) Entities - Critique, Map, Populate b) Relationships - Critique, Map, Populate Characteristics of relationships to be evaluated: Existence Semantics Cardinality Dependence c) Attributes - Critique, Map, Populate Characteristics of attributes to be evaluated: Existence - missing - extraneous - duplicate/redundant Semantics - wrong type - poorly named Key vs. NonKey - dependency - uniquely identifying - adequate - minimally sufficient d) Keys - Critique, Map, Populate (i.e., IDEF1X attributes) Characteristics of keys to be evaluated: Dependency Uniquely identifying - adequate - minimally sufficient e) Cardinalities - Critique, Map, Populate, Compare f) Categorizations - Critique g) Nomenclature - Critique h) Functions and "derived" information - Critique i) Presentation - Critique j) Redundancy - Query Path Analysis k) Usability - Query, Update 7.0 Test Procedure This section describes the terminology of the testing process, details of the testing procedure, and the relationships between each component of a specific test. 7.1 Test Suites Information to be tested is formally documented in test suites. A test suite is a collection of context diagrams, with corresponding test scenarios and test cases. Test data is also required to create the specific test cases. The test suite perspective is company-specific and defines that company's view of a particular area of interest. The areas of interest are selected from the lowest level activity boxes of the CDIM SM1 Activity Model. One test suite is created for each area of interest/company combination. It is anticipated that each tester will develop the initial structure for multiple test suites through interaction with the sheet metal die design Application Experts (AEs). Each tester will actually focus on selected test suites, however, with the subsequent test activities of remaining test suites to be performed as project resources and schedule allow. Appendix C, Test Suite Example, further illustrates the appearance of a test suite and the relationships between test suites, test scenarios, test cases, and test data. 7.1.1 Test Suite Context Diagrams Test suites contain context diagrams to define detailed areas of interest within a particular subject (i.e., the subject domain for a set of testing activities). Context diagrams are equivalent to an IDEF0 activity model with only one activity. Just as in IDEF0 activity modeling, the name of the activity is located within a box and all Inputs, Controls, Outputs, and Mechanisms (ICOMs) are shown around the outside of the box. An example context diagram is also shown in Appendix C. The context diagrams are developed from a detailed decomposition of selected portions of the overall CDIM SM1 Activity Model. This decomposition of the CDIM SM1 Activity Model occurs during workshops held with sheet metal die designers from various companies. Since a goal of CDIM SM1 testing is to provide 100% test coverage across the CDIM model, high-level context diagram subject areas should optimally be selected previous to these AE workshops to provide maximum test coverage. The context diagrams are generic in that they may apply to any company involved in developing the diagrams. Each context diagram may appear only once in a given test suite. The diagrams may be used in multiple test suites, however, if multiple companies perform the same activities. 7.1.2 Test Scenarios Test scenarios provide a textual description of specific aspects to be tested within a given test suite subject area. Each test scenario consists of a single context diagram with company-specific definitions for the activity and ICOMs. These definitions provide the company's interpretation of the context diagram. If an ICOM on the context diagram is not used by a particular company, that ICOM is not included in the test scenario. Each test suite must contain one or more test scenarios. Within a given test suite, however, a context diagram can only be used in one test scenario. 7.1.3 Test Data Test data consists of all real-world data available to a tester to use during CDIM SM1 testing. This test data will be provided by Sheet Metal Project member companies and other companies whose activities are modeled in the test suite. The test data will be the actual representation of the data used within the particular company (i.e., paper drawings, CAD system IGES files, process plans, change order information, specifications and other documents, etc.). The CDIM SM1 Test Team is responsible for collecting appropriate and sufficient test data to support testing of the desired test scenarios. To achieve this result, the test scenarios should be used as a guideline when collecting test data from the Application Experts. 7.1.4 Test Cases Test cases consist of the portions of the test data required to substantiate the data elements of a specific test scenario. In other words, test cases map the test data to the ICOMs in the test scenario context diagram. The test scenarios provide the context from which to perform testing and the test data provides the information to use during the mapping activity and to load into the database. The test data used within the test cases must come from the same company for which the test suite was developed. Also, each test scenario must contain at least one test case. Each test case also includes the "query strategy" to be used for testing the data elements within the test case. Analysis of the SQL queries and their results is used to evaluate the data dependencies within the CDIM data model. The query strategy used for each test case is selected from one of the three options below: Write pseudo-queries based on the context boxes and test data; then check if they are allowed by the CDIM model. Write queries based on the structure of the CDIM model; then check if they match the context boxes and test data. Do not write queries for this test case and state the reasons for this decision. Possible reasons for selecting this option include a significant number of common data elements among several test cases (i.e., to minimize redundant testing) or a lack of sufficient test data to analyze the path of a meaningful query. 7.2 Test Execution Execution of any given test requires initial development of test suites, test scenarios, and test cases. The test data must be obtained before test cases can be formed. While executing the tests, the Test Issue Log format must be available to record incorrect or unexpected test results. The actual steps in the execution of a test vary depending on the test method. If a high-level test is performed without use of software tools, or "on paper", the concept of test execution is difficult to document. When actual database queries are performed, however, test execution steps can be more easily defined. These execution steps were outlined in Section 4.0 of this document. When problems or deficiencies are found during CDIM SM1 testing, a test issue report is generated and a decision must be made regarding subsequent test execution. The default response to this situation is to fully document the issue and continue testing. This action should be sufficient for the majority of test issues. Other times, however, testing cannot continue without first correcting the problem. Two actions can occur at this point: 1) the tester can make a quick fix on-the-fly if the solution is quite trivial, or 2) testing can be suspended until the CDIM Model Team can provide the proper fix. Both of these actions have serious consequences. The first option should be avoided if at all possible, and if done, the fix should be documented extremely well and reverified in later releases of the CDIM model. From a project schedule standpoint, the second option should also be avoided and will be chosen only after independent verification of the problem and test team consensus has been reached. 7.3 Test Evaluation Criteria Test evaluation criteria are largely determined on a case-by-case basis, depending on the characteristics of the particular test. Evaluation of test results is also quite subjective based upon the tester's expected results and interpretation of actual test results. The test results obtained by this process do not actually guarantee that perceived correct results are indeed satisfactory, but only that specific deficiencies or issues have been identified. The primary evaluation criteria used in the testing process is therefore the tester's opinion on whether the actual test result matches the expected test result. The tester is expected to generate a test issue report for any mismatch or unexpected result. As all test issue reports are reviewed by the entire test team before further distribution, some assurance is provided that the issue is indeed warranted. 8.0 Resources The resources required for testing CDIM SM1 are quite significant. This section provides the requirements and/or current availability for the following types of resources necessary for the testing process: people, facilities, software tools, and training. 8.1 People The CDIM SM1 test team consists of the following personnel, listed below with their level of effort and their company affiliation: Kevin Jurrens, Team Leader 100% NIST Gloria Roth Robinson 100% EDS Shyhdong (Dong) Tsuei 100% EDS Bill Shaffer 100% Boeing Jack Skeels 30% P.D.I.T. Frank Demasek 25% EDS The primary test team consists of Kevin, Gloria, Dong, and Bill. This core team will participate in all aspects of the testing process. It is recognized that the size of this core test team is smaller than actually desired to perform thorough testing of CDIM SM1. Jack will be primarily involved in project management activities, application expert workshops, test team training, and testing guidance. Frank will participate with the test team in validating the Solid Geometry Boundary Representation (B-rep) and Form Feature aspects of the CDIM model. In addition, it is expected that as the CDIM model becomes more mature (i.e., after about the second iteration), less actual modeling work will be needed and some members of the CDIM modeling team will then participate in testing activities. As documented in the CDIM SM1 Project Plan, the following assignments have been made within the test team to distribute task responsibilities: Schedules, Agendas, Kevin Jurrens Distribution, and Announcements Scribe Gloria Roth Robinson Scribe Backup Kevin Jurrens Activity Model Gloria Roth Robinson Administrator Issue Log Custodian Bill Shaffer Issue Log Backup Dong Tsuei Reference Library Gloria Roth Robinson Document Editor Kevin Jurrens Schema Compiler Dong Tsuei SQL Language Expert Dong Tsuei Application Expert Liaison Gloria Roth Robinson Bill Shaffer Other human resources will be provided by SCRA and the Sheet Metal Project support team (i.e., selected members of the PDES, Inc. Core Program) to support program management activities and project reviews, respectively. In addition, NIST and SCRA PDES Testbed personnel will provide hardware/software support and training as necessary. 8.2 Facilities Testing of CDIM SM1 will require use of both the NIST National PDES Testbed (NPT) and the SCRA PDES Development Lab (PDL). Currently, the NIST Testbed has been designated as the primary testing facility for the Sheet Metal Project. This decision was based upon the relative use of each of the testbeds. Other factors, such as testbed readiness and software availability, are quite similar at both testbeds and do not affect this decision. Distribution of PDES, Inc. CDIM testing activities is currently under review, however, and this selection may change in the future. As previously mentioned in Section 4.5, Logistics of CDIM SM1 Testing, remote access to the testbeds through dial-in modems will facilitate a large portion of the testing. The majority of the CDIM testing software tools are not graphics-based and will function through a simple PC connection. As remote access capability is supported at both testbeds, test team member companies will use local hardware and software as required to remotely access the testbed computers. When a particular testing activity cannot be performed locally at a tester's site or through the testbed remote connection, that activity will occur at either NIST or SCRA as chosen by the test team. The selected testbed must be reserved in advance for these activities in order to avoid conflict with other PDES, Inc. or CDIM teams. Additional facility requirements include member company and other PDES, Inc. conference rooms for team meetings. Optimal facility requirements for conducting CDIM development and testing meetings are identified in Appendix H of the Testing Criteria Requirements Analysis Project (TC-RAP) Final Report [7]. Although some items are more a matter of convenience rather than necessity, this appendix includes items such as an overhead projector, paper flip chart, FAX machine, copy machine, secretarial support, outside telephone line, basic office supplies, vending machines, PC with printer, hotel/restaurant/hospital information, evening building access, etc. 8.3 Software Tool Requirements Software tool requirements for CDIM SM1 were originally identified in the document, Tools Requirements for CDIM SM1 Development and Test [13]. These requirements have not changed in their basic content since this original document. Some of the tools, such as electronic mail or ModelPro, have already been provided and are being used for Sheet Metal Project activities. Other tools, such as the activity modeling tool, were not available when needed at specific stages of the project and team members had to use less efficient methods to complete the task. The tool requirements in this document should still be considered current and any tool not yet provided would further enhance project productivity when available. Due to the initial project focus on IDEF-based testing, the original tools requirement document did not address the need for the Express-based NIST and PDES, Inc. STEP testing tools. With the change in CDIM SM1 testing methodology, however, these software tools are now of primary importance to the test team. As discussed in Section 4.2 of this document, this set of software tools will be used in its entirety and should be available at both NIST and SCRA Testbeds. The following software tool requirements are the highest priority new capabilities needed by the CDIM SM1 Test Team. These requirements may or may not have been identified in the original tools requirements document and are provided below in priority order. These software tool requirements have been communicated to the PDES, Inc. software tool development group. In accordance with the test team decision discussed in Section 4.5 of this document to pursue local access of QDES and the Visualizer software at each tester's member company site, assistance and support will be expected from the PDES, Inc. tools developers to establish these environments. CDIM Test Team members are currently gathering information to assess the feasibility and impact of this test environment. Enhancements are required to the IGES to PDES (ITOP) translator software tool. This tool currently supports a limited set of IGES entities. This set must be expanded to accommodate test data obtained from member company CAD/CAM systems used for sheet metal die design. The CDIM SM1 Test Team has asked the PDES, Inc. tools development team for support of the following additional IGES entities: Primary Importance: 142 - Curve on a Parametric Surface 144 - Trimmed (Parametric) Surface Secondary Importance: 120 - Surface of Revolution 140 - Offset Surface As discussed in Section 4.3, the Parasolid to STEP physical file translator developed at NIST will be used to generate STEP files containing B-Rep data. The limitation that this translator generates STEP physical files that comply with the "Tokyo" version of the STEP standard may require that software support be provided by PDES, Inc. to update this file to comply with the latest version of the STEP standard (or that specified by the proposed B-rep Application Protocol AP204, if appropriate). Software tool requirements for the IDEF1X-based portion of CDIM SM1 testing have not been further defined since the initial Tools Requirements Document. As mentioned in Section 4.4 of this document, the CDIM Test Team still plans to perform some investigative IDEF-based testing during the later stages of CDIM SM1 testing. For this reason, software tool support from PDES, Inc. may be requested to support this work. The CDIM SM1 Test Team and the PDES, Inc. tools team would jointly develop more detailed requirements before actual software development could begin on these IDEF-based test tools. 8.4 Training Requirements CDIM SM1 training requirements were also initially addressed in the document, Tools Requirements for CDIM SM1 Development and Test [13]. This document basically identified that training was required for each specific software tool and provided the expected method of training (i.e., self-taught, structured class, etc.) and necessary timeframe when known. These training requirements still apply when new software tools are introduced into the program. Other training already received by the test team includes: IDEF and Express classes provided by D. Appleton Company (DACOM) Overview education in the application areas of sheet metal part production and sheet metal die design (mainly in the form of reference material review, production facility tours, and application expert interviews) Overview training in the Express-based STEP test tools, the UNIX operating system, X-Windows environment, EMACS text editor, email, Revision Control System (RCS), etc. at NIST (Feb. 12-13) and SCRA (Feb. 14) CDIM test methodology training provided by Jack Skeels Conceptual Modeling training provided by P.D.I.T. for both the CDIM SM1 Model and Test Teams (March 12-13) Additional CDIM SM1 Test Team training requirements include: Structured Query Language (SQL) - A one-day class oriented toward PDES, Inc. activities is planned. Training in CDIM testing methodology/philosophy and the STEP testing tools will be ongoing throughout CDIM SM1 testing. Most of this training is very informal or self-taught. It is expected that experienced testers from past PDES, Inc. CDIM projects will provide guidance and informal training when necessary. 9.0 Test Deliverables This section describes the planned deliverables from the CDIM SM1 Test Team. These deliverables include the CDIM SM1 Test Plan, test procedure documentation, test result documentation, test issue reports, a test demonstration capability, and the CDIM SM1 Testing Final Report. 9.1 CDIM SM1 Test Plan The CDIM SM1 Test Plan (i.e., this document) constitutes the first deliverable of the CDIM SM1 test team. This document outlines the test objectives, constraints, procedures, activities, resources, and deliverables of the test team throughout CDIM SM1 testing. 9.2 Test Procedure Documentation Test Procedure Documentation will include all supporting information that defines the test suites, test scenarios, and test cases used in CDIM SM1 testing. Test suite documentation will consist of a textual overview which defines the company perspective and subject area domain. Test scenario documentation will primarily consist of context diagrams with company-specific definitions for the activity and ICOMs. Test case documentation may consist of several items, including portions of STEP physical files, design drawings, or process specifications, and will include textual information to describe how the test data elements are used for that particular test. This test procedure information will be documented to clearly identify the CDIM areas that are tested. This test team deliverable is primarily support documentation to confirm results claimed in the CDIM SM1 Testing Final Report. It is important to note that raw test data, such as design drawings, process specifications, etc., are not distributed past individual testers in accordance with company concerns about potentially sensitive or proprietary data. 9.3 Test Result Documentation Test Result Documentation is also primarily support documentation to confirm results claimed in the CDIM SM1 Testing Final Report. Test results and other test information will be recorded by each tester while performing test activities. When necessary, test results will be documented on CDIM SM1 Test Issue Logs (see Appendix B). Test result summary information will be provided to the CDIM Model Team through test result meetings at the conclusion of each test/fix/test iteration. This test result documentation is recorded to identify the actual extent of CDIM testing coverage. This test team deliverable is primarily a mechanism for sharing information between the testing and modeling teams and is typically not appropriate for extensive outside distribution. The information contained in this documentation is also summarized in the CDIM SM1 Testing Final Report. 9.4 Test Issue Reports Test Issue Reports are a test team deliverable that are provided primarily to the CDIM Model Team. These issue reports will be presented to the model team in a uniform manner through the CDIM SM1 Test Issue Log (see Appendix B) and test result meetings at the conclusion of each test/fix/test iteration. Further information on Test Issue Reports is provided in Section 10.0 of this document. 9.5 Test Demonstration Capability As identified in the CDIM SM1 Project Plan [16], a final deliverable from the testing process will be a test demonstration capability. This demonstration will be given on an as-requested basis and will consist of the execution of several meaningful SQL queries and updates performed on a database populated with test data. This demonstration is intended to provide Sheet Metal Project member companies with tangible results from the CDIM SM1 effort. 9.6 CDIM SM1 Testing Final Report The final and primary test team deliverable will be the CDIM SM1 Testing Final Report. This report will include the following information: An overview of the actual test activities, procedures, facilities, software tools, etc. used in CDIM SM1 testing. A summary of the test results, including all issues (both resolved and outstanding). An evaluation of the actual CDIM testing coverage, including support data to show the correlation between specific test suites and the CDIM model. An overall assessment of the CDIM model in terms of model thoroughness, rigor, completeness, and integrity. An overall assessment of the CDIM testing activity in terms of the test objectives stated in Section 3.2 of this test plan. Documentation on any "lessons learned" and suggestions for future CDIM testing activities. Final recommendations regarding the CDIM data model and potential Sheet Metal Project follow-on activities. 10.0 Issue Procedures This section describes the procedures for reporting, tracking, and resolving issues discovered during CDIM SM1 testing. 10.1 Issue Reporting Issues that are generated during any test/fix/test iteration will be documented using the CDIM SM1 Test Issue Log as shown in Appendix B. The Test Issue Logs will be a recording and reporting mechanism for the test team, and responsibility for maintenance of these logs will reside with the test team. Issues can be generated at any point during CDIM SM1 testing and can be written against any aspect of the testing process, including the model team deliverables, software tools, or even the testing process itself. This mechanism allows the tester to document any problem encountered. All issues will first be addressed at the test team level and will be forwarded to the appropriate body only after team approval. Attempts should be made to resolve software or testbed problems using other existing mechanisms, such as the NIST PDES Hotline, if appropriate, before generating a Test Issue Log. When recording an issue, all available supporting data should be provided, along with the information requested in the Test Issue Log. This information may include items such as the testbed facility or software tool used, the activity being performed when the problem was encountered, software warning or error messages, or other items that seem relevant to the issue. All test issues against the CDIM model should include identifiers to the specific test suite, test scenario, test case, and appropriate Activity/ICOM from the CDIM SM1 Activity Model. Test issue reports can also optionally include test team recommendations for resolving the issue. Recommendations will include, where appropriate, an explanation and sufficient support data to justify the recommendation. 10.2 Issue Tracking Tracking of the CDIM SM1 Test Issue Logs will be accomplished through a Test Team Issue Log Custodian. All Test Issue Logs will be submitted to the Issue Log Custodian in either electronic (WordPerfect) or paper form, as chosen by the tester. The Issue Log Custodian will maintain an Issue Log Notebook which includes paper copies of all test issues. Test issues may have a status of either OPEN, CLOSED, or INACTIVE. A status of OPEN means that the issue has been approved as an issue by the test team, but has not yet been resolved. A status of CLOSED indicates that the issue has been resolved to the satisfaction of the test team. A status of INACTIVE means that the issue has been recognized, but a fix is currently not available or will not be addressed by current work items. Test issues that have been reported to the Issue Log Custodian, but not approved for release by the test team, should not have any of the status fields marked on the Test Issue Log. In addition to the Issue Log Notebook, the Issue Log Custodian will create and maintain an issue log listing to summarize test issue activity. This listing will be a matrix showing the creation date, issue number, CDIM version, issue status, and resolution date. The resolution date will be an optional field used only for those issues that have been closed. 10.3 Issue Resolution Method Upon validation of a Test Issue Log by the test team, a copy of the issue log will be placed in the Issue Log Notebook and the issue log listing will be updated. At this point, an assessment will also be made by the test team whether to forward the issue immediately to the appropriate body, or to wait until a significant number of issues have been accumulated. Although the answer to this question will generally depend on the severity level of the problem, the typical approach for handling most test issues will be to deliver all new issues together at specified times during the testing process (e.g., weekly, on selected dates established with the model team, at the conclusion of each test/fix/test iteration, etc.). Resolved issues will be "closed" through either a consensus agreement by the test team and/or by verification during subsequent test/fix/test iterations. When properly resolved, the issue status will be changed to CLOSED and the issue log listing will be updated. Both resolved and outstanding Test Issue Logs, however, will be retained for later inclusion in the CDIM SM1 Testing Final Report. 11.0 References 1) Danner, William F., and Yang, Yuhwei, Generic Product Data Model (GPDM), Version 0.9, Draft, NISTIR forthcoming, National Institute of Standards and Technology, Gaithersburg, MD, January 5, 1991. 2) Danner, William F., and Yang, Yuhwei, Generic Enterprise Data Model (GEDM), Version 0.2, Draft, NISTIR forthcoming, National Institute of Standards and Technology, Gaithersburg, MD, January 5, 1991. 3) Katz, Susan, Configuration Management Concepts Document, NISTIR 4538, National Institute of Standards and Technology, Gaithersburg, MD, April 26, 1991. 4) Kramer, Thomas R., Extracting STEP Geometry and Topology from a Solid Modeler: Parasolid to STEP, NISTIR 4577, National Institute of Standards and Technology, Gaithersburg, MD, May 2, 1991. 5) Mitchell, Mary, Validation Testing System Development Plan, NISTIR 4417, National Institute of Standards and Technology, Gaithersburg, MD, September, 1990. 6) Palmer, Mark, Gilbert, Mitch, and Anderson, Jody, Guidelines for the Development and Approval of STEP Application Protocols, Version 0.7, Working Draft, ISO TC184/SC4/WG4, International Standards Organization, January 25, 1991. 7) PDES, Inc., Testing Criteria Requirements Analysis Project (TC-RAP), Final Report, PMG008.00.00, June 28, 1989. 8) PDES, Inc., Data Model and Test Criteria Development for CDIM A1, Final Report, PMG011.01.00, October 26, 1989. 9) PDES, Inc., Test Report for Context Driven Integrated Model (CDIM) Application A1, PTV012.01.00, February 1990. 10) PDES, Inc., CDIM A2 Project Plan for PDES, Inc., PMG013.01.00, August 9, 1990. 11) PDES, Inc., PDES Inc. Block Point Release 2.1 Systems Manual, PTI018.01.00, February 1, 1991. 12) PDES, Inc., Context Model Document for the PDES, Inc. Context Driven Integrated Model B4, Exchange of Product Design Data To Process Planning For NC Systems, Draft, March 12, 1991. 13) PDES, Inc. Sheet Metal Project, Tools Requirements for CDIM SM1 Development and Test, October 16, 1990. 14) PDES, Inc. Sheet Metal Project, Technical Development Plan, SM003.01.00, Draft, January 2, 1991. 15) PDES, Inc. Sheet Metal Project, CDIM SM1 Context Document for PDES, Inc., SM002.01.00, March 27, 1991. 16) PDES, Inc. Sheet Metal Project, CDIM SM1 Project Plan For PDES, Inc., SM001.01.00, March 13, 1991. 17) Smith, B., and Rinaudot, G., eds., Product Data Exchange Specification: First Working Draft, NISTIR 88-4004, National Institute of Standards and Technology, Gaithersburg, MD, December 1988. Appendix A: Testing Project Schedule Note: The following pages contain the entire CDIM SM1 project schedule. The activities on this schedule which pertain specifically to the testing effort are numbered 35 through 78. Appendix B: CDIM SM1 Test Issue Log CDIM SM1 TEST ISSUE LOG ISSUE # CDIM VERSION DATE ORIGINATOR NAME ACTIVITY ICOM TEST SUITE TEST SCENARIO TEST CASE TEST RESULT ISSUE DESCRIPTION: IMPACT: CRITICAL SEVERE MINOR SUGGESTED SOLUTION (OPTIONAL): RESOLUTION DATE: RESOLVED BY: RESOLUTION DESCRIPTION: COMMENTS: ISSUE STATUS: OPEN CLOSED INACTIVE Appendix C: Test Suite Example The following diagram is an IDEF1X representation of the relationships between context diagrams, test suites, test scenarios, test data, and test cases. This diagram is provided for further illustration of the test procedure as outlined in Section 7.0 of this document. Following this diagram is a sample test suite using information obtained through AE workshops held with sheet metal die design experts from the Boeing company. CDIM SM1 - Boeing Test Suite #1 (Example) Test Suite Overview This test suite describes the Boeing company perspective for testing the Manage Die activity (A2.1) from the CDIM SM1 Activity Model. See attachments for the CDIM SM1 node tree and IDEF0 Activity Model. The following context diagrams were developed during AE workshops with Boeing sheet metal experts and are used in this test suite: B2.1.1 - Commit to Schedule B2.1.2 - Conduct Coordination Meetings Test Scenario #1 B2.1.1 - Commit to Schedule. This activity consists of development of a binding contract where the Tool Design group agrees to complete a released die design on a given schedule. The schedule is determined by reviewing preliminary scheduling information, manpower, and workload. This review establishes a schedule approval and/or modifies a die design schedule. Manpower | | Workload | | V V --------------------------- | | Modified Die | Commit to Schedule |--> Design Schedule Die Design ---->| | Schedule | B2.1.1 |--> Schedule | | Approval --------------------------- ICOM Definitions ICOM - Die Design schedule DEFINITION Scheduling information for a die design (GREEN ROOM). COMPANY SPECIFIC DATA SOURCES: Indentured part list (IPL) from planning, based on OLS data, and information from production managers' flow charts (PMFC). (IPL and PMFC may be different, depending if tool design issues are present). ICOM - Manpower DEFINITION Identifies available designers, linked to workload. COMPANY SPECIFIC DATA SOURCES: Designer schedules. ICOM - Workload DEFINITION Defines die design committed schedule of work, linked to manpower. COMPANY SPECIFIC DATA SOURCES: Die Design work schedule information. ICOM - Modified die design schedule DEFINITION Modified schedule which reflects inability to meet the production part schedule, but which reflects best effort date, based on current work load. (GREEN ROOM) COMPANY SPECIFIC DATA SOURCES: Revised production manager flow chart information, or input to cause the revised chart. ICOM - Schedule approval DEFINITION Approval by die design group, which reflects acceptance of planned schedule. (GREEN ROOM) COMPANY SPECIFIC DATA SOURCES: Signature data from production manager flow charts. Test Case #1 This test case addresses the Boeing business case as outlined below. Specific values for the data elements are taken from the Boeing data source listed. This data will be used to validate the CDIM data model using the testing methodology provided in Section 4.0 of the CDIM SM1 Test Plan. ICOM - Workload Die Design Work Schedule line 1: work schedule date = 4/19/91 line 5: die number = ABC321 line 12: designer name = Larry Jones line 13: workload % = 25 line 20: completion date = 6/19/91 line 25: % complete = 0 ICOM - Manpower Designer Schedules line 2: designer name = Larry Jones line 3: department = D/878 line 4: activity name = DESIGN DIE FACE FOR ABC321 line 5: % time = 25 line 6: start date = 5/1/91 line 7: end date = 6/19/91 line 12: activity name = CORRECT PROBLEM IN DIE DD4 line 13: % time = 75 line 14: start date = 5/12/91 line 15: end date = ASAP (estimated 5/15/91) ICOM - Die Design Schedule Indentured Part List (from planning) line 14: die number = ABC321 line 16: die design department = D/878 line 23: required completion data = 6/25/91 line 26: part number = AZ-76543-01 line 28: part design version = D line 33: part designer = John Reynolds OLS Data line 3: die number = ABC321 line 4: part reference number = AZ-76543-01 line 6: last modification date = 5/18/91 line 9: fiscal year = 1991 line 14: hours logged = 42 line 18: % complete = 35 line 19: status of die = working prototype Production Manager's Flow Charts line 1: flow chart ID number = XXY547 line 4: date of flow chart = 5/22/91 line 6: activity name = DESIGN DIE FACE FOR ABC321 line 7: activity start date = 5/01/91 line 8: activity end date = June 19, 1991 line 9: person responsible = Larry Jones ICOM - Modified Die Design Schedule Revised Production Manager's Flow Chart line 1: flow chart ID number = XXY547 line 3: name of production manager = Mary R. Smith line 4: date of flow chart = 5/25/91 line 6: activity name = DESIGN DIE FACE FOR DIE #5 line 7: activity start date = 6/01/91 line 8: activity end date = June 23, 1991 line 9: person responsible = John Jackson ICOM - Schedule Approval Production Manager's Flow Chart signature data line 1: flow chart ID number = XXY547 line 3: name of production manager = Mary R. Smith line 4: date of flow chart = 5/25/91 line 33(a): name(s) of approving officer(s) = Bill Shaffer, T. Crume line 33(b): approval date(s) = 5/28/91, 5/29/91 Test Case #1 Query Strategy: Write queries based on the structure of the CDIM model; then check if they match the context boxes and test data. Test Case #2 Test Scenario #2 B2.1.2 - Conduct Coordination Meetings. There are two different types of coordination meetings: 1) Meetings to coordinate all parties concerned with producing the given part/tool and initiating the Tool Design Request. This meeting is schedule-oriented (GREEN ROOM); and 2) Meetings to approve the die design concept. A die design schedule review is performed to determine the die production schedule and general die configuration. Die design concept ideas are exchanged to determine production scheduling indices. This review approves or modifies the die design schedule. Once this review is performed, technical meetings are held to determine concept approval or identify required changes to either the die design concept or the part design. | | | | V V --------------------------- | | ------>| Conduct Coordination |--> | Meetings | ------>| B2.1.2 |--> | | --------------------------- ^ ^ | | | | ICOM Definitions Test Case #1 ICOM - Test Case #N Test Scenario #N Appendix D: List of Acronyms List of Acronyms AAM Application Activity Model AE Application Expert AP Application Protocol ARM Application Reference Model B-Rep Boundary Representation CAD Computer Aided Design CDIM Context Driven Integrated Model DACOM D. Appleton Company EDS Electronic Data Systems Corp. GEDM Generic Enterprise Data Model GPDM Generic Product Data Model ICOM Input-Control-Output-Mechanism IDEF0 ICAM Definition Language (for activity modeling) IDEF1X ICAM Definition Language (for data modeling) IGES Initial Graphics Exchange Specification ISO International Organization for Standardization ITOP IGES to PDES (software tool) NIST National Institute of Standards and Technology NPT National PDES Testbed (at NIST) OTON "Old" to "New" (geometry conversion software tool) PC Personal Computer PDES Product Data Exchange Using STEP P.D.I.T. Product Data Integration Technologies, Inc. PDL PDES Development Lab (at SCRA) PSCM Product Structure Configuration Management QDES Quick and Dirty Editor for STEP (software tool) RCS Revision Control System SCRA South Carolina Research Authority SML Structural Modeling Language SQL Structured Query Language STEP Standard for the Exchange of Product Model Data STEPwf-SQL STEP Working Form to SQL (software tool)