Laura Baber, Administrative Office of the U.S. Courts
Margaret Mowry, U.S. Pretrial Services, District of New Jersey
Timothy P. Cadigan, Administrative Office of the U.S. Courts

I. Outcome Measurement
Components of the Logic Model
Operationalizing Pretrial Services Supervision Outcomes
II . Data Quality Improvement

THE PRETRIAL SERVICES ACT of 1982 instituted pretrial services in the federal criminal justice system, but current management and organizational thinking holds that instituting an outcome measurement system is key to seeing pretrial services mature and fully develop in its second 25 years. The federal pretrial services system has recently begun the process of instituting such a system. This article is a discussion of that plan and the first task it is undertaking: the improvement of data quality. This article is not a policy statement or procedure determination for the federal pretrial services system. Rather, it merely attempts to apply outcome measurement principles and concepts to the federal pretrial services system in an effort to enhance the discussion within that system.

back to top

I. Outcome Measurement

The federal probation and pretrial services system is developing a results-based management framework that will, in the future, allow it to better assess performance—and make programming and resourcing decisions— based on what it accomplishes rather than solely on what it does. The flow chart shows the steps involved in developing the framework, and highlights where we are in the process.

1. Project Background
This focus on results, and the work done to date to define the system’s mission, goals and desired outcomes, stems from a number of complementary influences and projects.

• In 1999, the Administrative Office of the U.S. Courts entered into a contract with a team of independent consultants, led by IBM, to conduct a strategic assessment of the federal probation and pretrial services system. The overarching recommendation from that assessment—presented first to the Administrative Office in 2003—was that the federal probation and pretrial services system become a results-driven organization with a comprehensive performance measurement system.
• In 2000, the AO Director appointed an Ad Hoc Supervision Work Group comprised of supervisors, deputies, and chiefs from seven districts and a representative of the Federal Judicial Center to update the supervision policy monographs. As part of its work, the group reviewed relevant statutes and mission statements to identify the desired outcomes and goals to be served by the pretrial services and postconviction supervision functions. These outcomes and goals were incorporated into revised supervision policy documents approved by the Judicial Conference of the United States in 2003.
• Strategic planning sessions were conducted at the 2000 and 2002 Federal Judicial Center’s National Chiefs Conferences. The 2000 conference produced a “Desired Futures” roadmap, the first element of which was: “Desired Outcomes are clear, measured and results are communicated.” The 2002 conference resulted in a “Charter for Excellence” that sets forth broad system goals and values.
• In September 2003, one of the IBM strategic assessment consultants facilitated a strategic planning session at a meeting of the Chiefs Advisory Group to translate the broad “Charter for Excellence” statements into more specific “Operational Goals.”
• The operational goals developed by the Chiefs Advisory Group were combined with the desired outcomes set forth in the revised supervision monographs to form the basic structure of the results-based management framework. This concluded the initial goal-setting stage of the framework development process. ¹

The current stage of the process is technical: The development of operational definitions and associated measures for each “desired outcome;” and of statistical approaches to analyze the information that will assure “apples-to-apples” comparisons and allow benchmarking with other programs. The product from this technical phase will be a set of recommendations, to be circulated for broad system comment, that address:

• How to measure a variety of outcomes— including defendant compliance, positive change, and crime reduction;
• What data are needed to construct the recommended measures; and
• What analytical methodologies can be used to assess how these results are affected by supervision interventions as well as by a variety of case, defendant and community factors?

The recommendations are to represent “state of the art” measurement and analytical approaches that are being used by other performance-based systems, program evaluations and/or academic research in criminal justice and related areas such as substance abuse. These recommendations will be circulated to system staff and stakeholders for review and comment to prepare the “Framework Design” document that will guide further refinement of the database and the analyses to be performed.

The next section of this paper will use pretrial services supervision outcomes to illustrate the technical concepts to be incorporated in the framework design. It should be noted that this section is an illustration and does not reflect any policies for pretrial services outcomes. It is provided merely to assist the reader in envisioning the future.

2. Pretrial Services Supervision Logic Model
Building on the results of the goal-setting stage of this project, the next step was to develop a logic model for pretrial services supervision that depicts the underlying assumptions about how “what the system does” affects what it is trying to accomplish; and what other factors—e.g., characteristics of the defendants to be supervised, the requirements and restrictions of their bail conditions, and the system resources devoted to carrying out the supervision mission—are expected to influence this relationship.

This logic model has been refined twice since its development following the goal setting stage. It will continue to be a work in progress that evolves to incorporate feedback from system staff and stakeholders, and results from empirical testing of the posited relationships.

back to top

Components of the Logic Model

The pretrial services supervision logic model has six components: inputs, process (activities), process outcomes, intermediate outcomes, ultimate outcomes, and mission. Each component is described below.

Inputs
Inputs are characteristics of the defendant population and the working environment that are hypothesized to affect expected outcomes regardless of system interventions. For example, prior research indicates that defendants with a lengthy prior record are more likely to become re-involved in criminal activity than those with no or a minimal prior record. This leads to a working assumption that, regardless of supervision interventions, districts that have a high percentage of first defendants will have a lower recidivism rate than those with a low percentage of first defendants.

Pretrial Logic Model
Inputs are used in the analytical model as “control” variables to account for the effects of factors that explain differences in outcomes across offices, districts and time that are not related to system interventions. They may also be used as stratification categories to display outcomes based on key groupings, e.g., by offense charge.

The current model includes as inputs those factors identified in the research and program evaluation literature as related to criminal justice goals. These include:

• Defendant characteristics (e.g., prior record, employment, family/community connections, demographics);
• Characteristics of the instant offense (e.g., class and category);
• Release parameters (e.g., supervision imposed, conditions imposed);
• Office/community characteristics (e.g., location, size, socio-economic indicators);
• Officer characteristics (e.g., experience, demographics, education);
• Supervision resources (e.g., supervision staffing, contract budgets, technological support).

The inputs categories will be further defined and the categories and their specific elements assessed for adequacy by system staff and stakeholders as part of the review of the technical framework document.

Process
Process refers to activities undertaken by the system—practices, programs and interventions— that implement the supervision function. As an example: An officer conducts an initial assessment investigation, identifies lack of stable employment as a risk, recommends an employment condition to the judicial officer, and refers the defendant for job counseling or to a job referral agency. In the analytical model, the process variables define “what we do” for purposes of assessing the basic relationship of how “what we do” relates to what we are trying to accomplish.

The current logic model includes only the most general process categories, e.g., investigation, assessment, monitoring, referral, and assistance. Detailed input on the specific processes that should be included in the model will be sought from system staff and stakeholders—the experts in identifying and defining salient system activities—as part of the outcome development process.

Process outcomes describe defendant actions that occur as a result of system activities. For example, in response to an employment referral, the defendant registers with an employment service or completes “x” hours of employment counseling. Process outcomes enter the analytical model as both an outcome of the service delivery process and as an input (control) for assessing ultimate outcomes. For example, “number of hours of employment counseling” is a measure of how successful an officer’s employment referrals are in engaging defendants in employment services.

Ultimate Outcomes and Mission
The ultimate outcomes are set forth in The Supervision of Federal Defendants, Monograph 111, which establishes Judicial Conference policies related to pretrial services supervision. These outcomes are: To address the defendant’s risks of nonappearance and/or dangerousness. As the Monograph states: “The desired outcome in all cases is for the defendant to successfully complete the supervision period by obeying the law, complying with any other conditions of release, and making required court appearances throughout the period of supervision.” ²

3. Relationships among Components
The arrows in the logic model indicate the specific expected relationships between components that the analytical model will be designed to test. Statistical techniques will be applied to test the relationships depicted. The analysis will test a complete thread of the model, starting from left to right. Basic and advanced techniques will be used to test both direct and indirect and unidirectional and bidirectional relationships, while controlling for inputs that are primarily static and outside the control of the officer. The results will move the system beyond a description of the defendant population and individual outcomes to a more complex assessment of the “theory of change” and the interconnectedness of process and outcomes for pretrial services supervision.

back to top

Operationalizing Pretrial Services Supervision Outcomes

This section further defines the process and outcomes in measurable terms. In order to empirically test the hypothesized relationships between pretrial services processes (activities) and outcomes of the defendant population, it is necessary to first identify appropriate measures for each outcome.

A process outcome represents the immediate outcome for the defendant as a result of system activities. An ultimate outcome is the long-term result of the system activities for the defendant. The ultimate outcomes also reflect achievement of the mission of the federal pretrial services system. The three ultimate outcomes that best reflect the mission include: minimize criminal activity during the period of supervision, minimize technical violations, and maximize appearance in court and self-surrender. The analysis of data on these ultimate outcomes will help protect the public and assist system staff and stakeholders to better assess if the missions of the fair administration of justice are being achieved. Each ultimate outcome is discussed below.

• Minimize criminal activity during the period of supervision—The primary measure of criminal activity during the period of supervision is whether a defendant was arrested for a new offense. Technical violations are not counted as a new offense. The analysis could also examine the time to arrest (length of time before the arrest for a new offense). Finally, the results could be presented overall and by offense type (e.g., violent, property, drug, public order, weapon, immigration) and offense level (felony, misdemeanor, petty).
• Minimize technical violations—The primary measures of technical violations during the period of supervision are judicial determinations that a defendant violated one or more conditions of release. The analysis could also examine the length of time before a technical violation.
• Maximize appearance in court and self-surrender— The primary measures of appearance in court and self surrender are judicial determination that the defendant failed to appear for a required court hearing or the Bureau of Prisons determines that the defendant failed to surrender. Technical violations, such as failing to report to a pretrial services officer, might not be counted as failures to appear. The analysis could also examine the time to failures to appear.

Ultimate outcome data enable system staff and stakeholders to test whether the system activities (processes) are leading to the longterm outcomes that the federal probation and pretrial services system is tasked with achieving. Furthermore, these data will allow system staff and stakeholders to assess how well they are doing at meeting their mission to protect the public and fairly administer justice.

back to top

II . Data Quality Improvement

From outcome measurement systems through data warehouses and a host of other big budget projects in government and business, the landscape is strewn with processes and systems that were undone by poor data quality. By beginning this undertaking for the federal pretrial services system with a focus on improving our data quality, the federal system hopes to avoid this quandary. The goal of a data quality program is not data perfection— that would be impossible and is frankly unnecessary. The goal should be consistently achieving acceptable levels of data errors. Experts in the field of data quality generally consider acceptable error to be no more than one or two percent of the total. This is a realistically achievable goal. This article closes by looking at the process developed and implemented to improve data quality in the federal pretrial services system.

1. Data Quality Improvement Working Group
In 2005, the Office of Probation and Pretrial Services formed a committee of chiefs, supervisors, officers, technical personnel and data quality analysts from probation and pretrial services offices in various districts. The mission of this committee was to provide advice and guidance to the Administrative Office of the U.S. Courts on issues related to the development of a formal data quality program.

The data quality working group has been put into place to help establish how data quality should be defined and how to communicate this information to the districts. The working group established a data quality website that has been used to provide standard data quality reports to the districts for correcting data that is necessary to move the Office of Probation and Pretrial Services to a national standard. Additionally, the working group understands that in order to receive long-term data quality improvement, we must provide the districts with standards and policies for everyday processes.

The working group realizes there is a need to provide information to the district chiefs and deputies along with the data quality analysts who are working with the data on a day-to-day basis. To date, the working group has made two presentations to the data quality analysts and one presentation to chiefs and deputies. The working group has provided the data quality analysts with the basic needs and information to equip them for the necessary data quality clean-up process.

2. Data Quality Improvement Program
The Office of Probation and Pretrial Services suggests that each district create their own data quality improvement program. The federal courts are a uniquely decentralized system, with each chief pretrial services officer/ chief probation officer reporting to a chief judge in one of 94 judicial districts. Given that structure, the data quality working group felt that in addition to a national data quality program, each district needed to have its own district data quality program. Therefore, one of the first products to emerge from the data quality working group was the “District Data Quality Program Development Guide.”

The Guide provides a step-by-step process districts can follow to develop a data quality program. The first step in launching a data quality program is strong leadership, direction and support of quality improvement activities by the chief of the district; these are key to performance improvement. The involvement of organizational leadership assures that quality improvement initiatives are consistent with the mission of the data quality working group.

The Guide recommended that each district establish this program and include one of each of the following representatives to create the team:

• Data Quality Manager (appointed by the chief)
• DQA / Lead DQA
• IT Worker
• SUSPSO / Line Officer
• Treatment Specialist / Administrator
• Supervision Point of Contact Representative
• Data Entry Clerk

Once the team has been established, the district should develop a project plan and conduct an audit of the data, policies and procedures that need to be established. This will provide an understanding of the type of program to be developed, provide training to the entire staff, monitor the data, and improve daily processes. The Guide suggests that each district create a guide that will assist the district in this mission.

3. Data Quality Improvement Training
There are two primary issues in the training area for data quality: 1) How to develop training that adequately prepares data entry staff to enter data accurately and 2) How to develop training that adequately prepares data quality staff to identify data entered inaccurately. To accomplish these it is imperative that the districts create and implement training programs for all staff members. Data entry training should be provided following the established procedures for PACTS training established by the PACTS project team. For assistance, districts should work with the San Antonio Training Center. Creating true data quality training is more complex. The Office of Probation and Pretrial Services (OPPS), in coordination with the Data Quality Improvement Working Group and the Chiefs Advisory Group, developed the Regional Data Quality Improvement Conferences, held over the past year to begin to address this need. We hope that this is the first year of regional conferences on data quality improvement to be held. Even with that piece in place, however, more needs to be done to further enhance training opportunities for data quality analysts.

One of the most effective ways staff members can gain an appreciation for the tasks, issues, and problems data quality analysts and data entry staff encounter in the district is to spend time with the persons performing those functions. Staff can learn how they obtain the data they enter or verify in PACTS and what they do to verify the accuracy of the data once entered. Proceeding step-bystep through the process provides a wealth of knowledge about that process and often identifies problems in the process that can be rectified. For example, there is the issue of whether or not forms should be employed in the data entry process. Originally forms were encouraged and in fact shared and promoted by OPPS. However, the Probation and Pretrial Services Data Quality Improvement Group ultimately discovered that forms for the most part only add to the opportunity for data entry error. As a result OPPS now suggests that data entry be performed directly from source documents. Performing that type of process analysis locally can enhance your data entry procedures.

4. Data Quality Improvement Review System
Given the outcome measurement direction of the Office of Probation and Pretrial Services (OPPS), at the behest of the Criminal Law Committee of the Judicial Conference and with the cooperation of the Chiefs Advisory Group, failure to improve the data quality will result in erroneous decisions based on erroneous data. Any system designed to improve the effectiveness and efficiency of the federal pretrial services system by analyzing and reviewing data on existing procedures and outcomes can only succeed if the decisions are based on accurate, detailed, and reliable data. As with financial accounting and other disciplines, one important tool in improving data accuracy and establishing benchmarks for data accuracy are audits or reviews of the work.

The Office of Probation and Pretrial Services attempts to conduct 20 program reviews per year. Program reviews are designed to assist districts in identifying and addressing problems in existing processes and procedures. The reviews primarily focus on probation and pretrial services program and operational issues. Beginning in FY 2008 OPPS hopes to add data quality program reviews to the areas addressed during the program review of the office. To perform the review a variety of processes will be employed, including staff interviews, data analysis and comparison, and process analysis. It will conclude with a section in the program review report on findings and recommendations specifically focused on data entry and data quality.

Each district has specific areas and needs for improving data quality. The Data Quality Working Group was put into place to provide the field with assistance and guidance to develop a program that works for everyone. A self-assessment for pretrial services data quality improvement has been developed and is a step-by-step guide available to help districts to assess their practices and determine what areas need improving. The self-assessment will also prepare the office to meet national standards in the event of a program review. This self-assessment can be used by all districts regardless of staff or caseload size. This manual explains how to complete the assessment, provides forms for recording and tabulating the findings, and offers ideas for follow-up.

The Office of Probation and Pretrial Services recommends that each district work towards developing and implementing a program that maintains a national standard. The self-assessment will help districts achieve this goal and continue their focus on data quality.

back to top

___________________________________________________________________________
* Parts of this article were adapted from a contract report previously submitted to the Administrative Office of the U.S. Courts by Caliber Associates.