Banner National Institute of Standards and Technology
ISD Research Area Banner
ISD home About ISD ISD Research Areas ISD's Products and Services What's New in ISD Search ISD

 

Research and Engineering of Intelligent Systems

Performance Metrics

Measuring the Performance and Intelligence of Systems: Proceedings of the 2002 PerMIS Workshop

The proceedings are available in PDF format. You can view each paper below. If you do not have Adobe Acrobat Reader you can download it for free from Adobe.

NIST Special Publication 990

August 13-15, 2002
Edited by: E.R. Messina and A.M. Meystel

The Proceedings are available in PDF format. You can view each paper or the entire proceedings below. If you do not have Adobe Acrobat Reader you can download it for free from Adobe.

Session 1M1- Performance Metrics (Chairs: S. Agrawal, H. Yanco)
R. Gao, L. Tsoukalas, Performance Metrics for Intelligent Systems: An Engineering Perspective
S. Ananthakrishnan, S. Agrawal, R. Venugopal, M. Demeri, RCS Based Hardware-in-the-loop Intelligent System Design & Performance Measurement
H. Yanco, Evaluating the Performance of Assistive Robotic Systems
J. DeLeo, Measuring Classifier Intelligence
V. Stefanuk, Problems of Performance Measurement in Locally-Organized Systems

Session 1M2 - Performance of Multiple Agents (Chairs: E. Grant, M. Fields)
A. Shirkhodaie, A Control Scheme for Measure of Performance and Efficiency of Tactical Cooperative Robots
A. Nelson, E. Grant, T. Henderson, Competitive Relative Performance Evaluation of Neural Controllers for Competitive Game Playing with Teams of Real Mobile Robots
M. Fields, Representing Ground Robotic Systems in Battlefield Simulations
E. Dawidowicz, NICCI: A Multi-agent Cognitive Formation

Plenary Lecture 1-1
J. Albus, Metrics and Performance Measures for Intelligent Unmanned Ground Vehicles

Session 1P1 - Performance of Mobility Systems (Chairs: E. Messina, Y. Zhang)
S. Balakirsky, E. Messina, A Simulation Framework for Evaluating Mobile Robots
H. Scott, S. Szabo, Evaluating the Performance of a Vehicle Pose Measurement System
D. Dufourd, A. Dalgalarrondo, Performance Evaluation of Road Detection and Tracking Algorithms

Y. Zhang, C. Eldershaw, M. Yim, K. Roufas, D. Duff, A Platform for Studying Locomotion Systems: Modular Reconfigurable Robots

Session 1P2 - Performance of Planning Systems (Chairs: A. Meystel, W. Van Wezel)
A. Meystel, Performance of Planning Systems
I. Kacem, S. Hammadi, P. Borne, Lower Bounds for Evaluating Schedule Performance in Flexible Job Shops
W. Van Wezel, R. Jorna, Performance Characteristics of Planning Actors

J. Maryak, D. Chin, Global Optimization via SPSA

General Discussion Panel 1 - What is the Role of Ontology in Performance Evaluation?
E. Dawidowicz, Ontology in Performance Evaluation
M. Gruninger, Using Ontologies to Evaluate Knowledge-based Systems
L. Pouchard, Toward an Ontology of Performance
C. Schlenoff, What Role Can Ontologies Play to Improve the Performance of Intelligent Systems?
L. Welsch, Role of Ontologies in Performance Evaluation
A. Meystel, Ontologies OF and FOR Intelligent Systems

Reception
M. Asada, RoboCup: Humanoids as a Testbed for Measuring Performance

Plenary Lecture 2-1
D. Fogel, Evolving Solutions that are Competitive with Humans

Session 2M1 - Uncertainty of Representation I (Chairs: C. Landauer, J. Gunderson)
J. Scholtz, Evaluation Methods for Human-System Performance of Intelligent Systems
R. Cottam, W. Ranson, R. Vounckx, Lifelike Robotic Collaboration Requires Lifelike Information Integration
J. Gunderson, L. Gunderson, Integrating Effective Planning Horizons into an Intelligent Systems Architecture
A. Lytle, K. Saidi, W. Stone, M. Shneier, Mobile Robot Pose Tracking for Performance Analysis

A. Clerentin, L. Delahoche, E. Brassart, C. Cauchois, An Uncertainty Propagation Architecture for the Localization Problem

Session 2M2- Performance of Robots in Hazardous Domains (Chairs: A. Jacoff, C. Norman)
T. Frost, C. Norman, S. Pratt, B. Yamauchi, B. McBride, G. Peri, Derived Performance Metrics and Measurements Compared to Field Experience for the PackBot

D. Bruemmer, J. Marble, D. Dudenhoeffer, M. Anderson, M. McKay, Intelligent Robots for Use in Hazardous DOE Environments

Plenary Lecture 2-2
W. Freeman, On Communicating with Semantic Machines

Session 2P1 - Modeling Intelligence (Chairs: L. Arata, S. Wallace)
L. Arata, Modeling Interactive Intelligences
L. Polyakov, Structured Approach to the Intelligent System Design
L. Perlovsky, Semiotic Fundamentals of Information Processing in Human Brain

S. Wallace, J. Laird, Intelligence and Behavioral Boundaries

Session 2P2 - Modeling of Mind (Chairs: S. Adams, R. Cottam)
N. Alvarado, S. Adams, S. Burbeck, C. Latta, Beyond the Turing Test: Performance Metrics for Evaluating a Computer Simulation of the Human Mind
R. Schrag, M. Pool, V. Chaudhri, R. Kahlert, J. Powers, P. Cohen, J. Fitzgerald, S. Mishra, Experimental Evaluation of Subject Matter Expert-oriented Knowledge Base Authoring Tools
N. Cassimatis, G. Trafton, A. Schultz, M. Bugajska, W. Adams, A Task Domain for Combining and Evaluating Robotics and Cognitive Modeling Techniques

R. Sanz, A. Meystel, Modelling, Self and Consciousness: Further Perspectives of AI Research

General Discussion Panel 2 - Informantion Access Technology: Does it Concern Mechanisms of Mind and Intelligence?
M. Herman, Information Access Technology: Does it Require Getting Involved in Mechanisms of Mind and Intelligence?
J. Cugini, Information Access: Do You Mind?
J. Garofolo, Using Speech Technologies for Information Access: Does it Require Getting Involved in Mechanisms of Mind and Intelligence?

Banquet Speech
L. Zadeh, In Quest of Performance Metrics for Intelligent Systems - A Challenge that Cannot be Met with Existing Methods

Plenary Lecture 3-1
J. Blitch, Robot Intelligence for Tunneling and Confined Space Search and Rescue

Session 3M1 - Measuring Intelligence (Chairs: J. Horst, C. Schlenoff)
B. Clough, Metrics, Schmetrics! How the Heck do you Determine a UAV's Autonomy Anyway?
C. Schlenoff, L. Welsch, R. Madhavan, N. Zimmerman, Towards Measuring the Performance of Architectural Components of Autonomous Vehicular Systems
J. Horst, A Native Intelligence Metric for Artificial Systems

G. Berg-Cross, Dimensions of Intelligent Systems

Session 3M2 - Grouping: A Core Procedure of Intelligence (Chairs: S. Ramaswamy, R. Bialczak)
L. Reeker, A. Jones, A New Classification of Information: A Step on the Road to Interpretability
C. Landauer, K. Bellman, Refactored Characteristics in Intelligent Computing Systems
S. Ramaswamy, R. Neelakantan, Software Design and Testing using Petri Nets: A Case Study Using a Distributed Simulation Software System

R. Bialczak, J. Nida, B. Pettitt, M. Kalphat, Comparison Methodology for Robotic Operator Control Units

Plenary Lecture 3-2
X. Hu, B. Zeigler, An Integrated Modeling and Simulation Methodology for Intelligent Systems Design and Testing (Supporting Material)

Session 3P1 - Uncertainty in Representation II (Chairs: D. Chin, R. Maarfi)
D. Chin, A. Biondo, Multiple Neural Network Model Interpolation
R. Maarfi, E.L. Brown, S. Ramaswamy, A Three-Tier Communication and Control Structure for the Distributed Simulation of an Automated Highway System
A. Meystel, A. Bathija, Tessellating and Searching in Uncertain State Spaces

K. Bellman, Autonomy and Socialization

Session 3P2-Towards Universal Planning/Control Systems (Chairs: A. Meystel, P. Davis)
A. Meystel, A Sketch of Multiresolutional Decision Support Systems Theory
P. K, Davis, J. H. Bigelow, Motivated Metamodels
H. Sarjoughian, On the Role of Quality Attributes in Specifying Software/System Architecture for Intelligent Systems

S. Orr, Uncertain Predictions of Flow and Transport in Random Porous Media: The Implications for Process Planning and Control

Contact

Ms. Elena Messina
NIST
Intelligent Systems Division
100 Bureau Drive, Mail Stop 8230
Gaithersburg, MD 20899-8230
Phone: 301-975-3510
Fax: 301-990-9688
E-mail: elena.messina@nist.gov

Date created: 12/10/2002
Last updated: December 12, 2002
Manufacturing Engineering Laboratory Skip navigation