Bureau of Justice Statistics. Performance Measures for the Criminal Justice System. Washington,
DC: Article prepared for the U.S. Department of Justice, Bureau of Justice Assistance by John J.
DiIulio, Jr.; 1993.  pp. 143-156.

View entire document

Measuring Performance When There Is No Bottom Line

by John J. DiIulio, Jr.

The great scholar of American government, Wallace Sayre, promulgated the "law" that "public and private management are fundamentally alike in all unimportant respects." Many leading thinkers-Graham T. Allison, Jr. of Harvard University and James Q. Wilson of UCLA, to cite just two-have echoed this view. ¹

As Wilson has noted, to a "much greater extent than private bureaucracies, government agencies (1) cannot lawfully retain and devote to the private benefit of their members the earnings of the organization, (2) cannot allocate the factors of production (land, labor, capital) in accordance with the preferences of administrators, and (3) must serve goals not of the organization's own choosing. Control over revenues, productive factors, and agency goals is all vested to an important degree in entities external to the organization-legislatures, courts, politicians, and interest groups."²

With few exceptions, the human and financial resources of public managers are contingent upon the goodwill of legislators and other policymakers, not the goodwill of customers and other consumers. Relative to private managers, public managers in civil service bureaucracies have little discretion in hiring, firing, and promotion decisions; and, their ability to innovate-to change the way work gets done-is far more constrained.

Unlike most private corporations, most government agencies have no market-test of output. The managers of a smokestack company are out to turn a profit; the managers of the Environmental Protection Agency are out to "protect the environment." The executives of television networks contrive to generate dividends for shareholders; the heads of the Federal Communications Commission contrive to regulate airwaves in "the public interest." At most, public managers have proxies for outputs: for example, increasing weapons stockpiles as a proxy for "enhancing combat readiness" and "strengthening national defense."

Thus, as Allison observes, there is "little if any agreement on the standards and measurement of performance to appraise a government manager, while various tests of performance-financial return, market share, performance measures for executive compensation-are well established in private business.... Governmental managers rarely have a clear bottom line, while that of a private business manager is profit, market performance, and survival."³ Or, in Wilson's words, "whereas business management focuses on the 'bottom line' (that is, profits), government management focuses on the 'top line' (that is, constraints)."⁴

Thus, the essential differences between public and private management are rooted in the nature of their respective goals. Basically, there are two types of goals: operational and nonoperational.⁵ An operational goal is an image of a desired future state of affairs that can be compared unambiguously to an actual or existing state of affairs. A nonoperational goal is an image of a desired future state of affairs that cannot be compared unambiguously to an actual or existing state of affairs.

Some examples are in order. "Keeping America first in space technology" is a nonoperational goal; "Putting an American on Mars by the year 2010" is an operational goal. "Improving the quality of public education in America" is a nonoperational goal; "Increasing the average verbal and math SAT scores of public school students by 20% between the year 1992 and the year 2000" is an operational goal. "Making America's welfare system work" is a nonoperational goal; "Instituting work-based welfare programs that get 50% of all participants off the rolls in their first 7 years of operation" is an operational goal. "Reforming criminals" is a nonoperational goal; "Doubling the rate of inmate participation in prison industry programs" is an operational goal.

Generally speaking, public managers have nonoperational goals, while private managers have operational goals. Moreover, public managers often work in the context of multiple and contradictory nonoperational goals.

Correctional agencies and forestry services are two classic examples. For much of American history, correctional agencies were authorized to "punish, deter, incapacitate, and rehabilitate" convicted criminals Most employees of the Federal Bureau of Prisons, for example, are "correctional officers" responsible for the "care and custody" of prisoners. By the same token, most employees of the U.S. Forest Service are "forest rangers" charged with "multiple-use management" (hunting, fishing, timber cutting, timber sales, fire prevention, flood control). The correctional officers are "social workers, prison guards, and managers" all rolled into one; the forest rangers are "executives, woodsmen, and planners" all rolled into one.⁶ In both cases, the multiple and contradictory nonoperational nature of the agency's goals renders inapplicable the much-misunderstood concept of efficiency.

Efficiency to a relationship between valued inputs and desired outputs: maximizing outputs from a given set of inputs, or minimizing the inputs necessary to achieve a given level of outputs. In many, perhaps most, government agencies, the relationship between valued inputs (people, money) and desired outputs (less crime, better public health) is ambiguous. Where goals are nonoperational, and the technologies necessary to achieve them are either uncertain, or completely unknown, or simply unavailable, the quest for a "bottom line" is a fool's quest. To the extent that goals are vague or inconsistent, the concept of efficiency is irrelevant.⁷ The concept of efficiency is thus irrelevant to many, if not most, public management tasks.

Defining and measuring nonoperational goals: What is to be done?

Because public management and private management are indeed fundamentally alike in all unimportant respects, it makes little sense to lament the fact that "government isn't run like a business," or to assume that government agencies are inherently less "efficient" than private firms.⁸ As Allison concludes, "the notion that there is any significant body of private management practices and skills that can be transferred directly to public management tasks in a way that produces significant improvements is wrong."⁹

It is yet possible, however, for public managers to strengthen the relationship between administration and goals, inputs and outputs, process and performance. The multiple and contradictory nonoperational character of most public management goals-the lack of a market-test of output in much of the governmental sector-is neither an immovable barrier to defining appropriate measures for the goals of public agencies, nor an inexhaustible excuse for the failure to do so. As Wilson notes, if someone set out to measure "the output of a private school, hospital, or security service, he or she would have at least as much trouble as would someone trying to measure the output of a public school, hospital, or police department. Governments are not the only institutions with ambiguous products." ¹⁰ In recent years, in fact, a number of major public policy reforms were launched as efforts to define appropriate measures for the goals of government agencies.¹¹ There is no reason why the field of criminal justice cannot or should not follow suit.

A first step in defining appropriate measures for the goals of the criminal justice system is for practitioners to ask four questions:

• What do we want to achieve?
• What human and other resources do we have (or can we obtain) that might enable us to achieve it?
• How can we know the relationship between the levels and modes of our resource deployments, on the one hand, and the achievement of our goals, on the other?
• How can we measure routinely the extent to which we are achieving what we want to achieve, or moving toward achieving it?

These may seem like an incredibly obvious set of conceptual questions to ask, but they are ones that public managers have not asked with sufficient regularity or seriousness of purpose. This is true even in public management sectors like defense, where there has been no shortage of well-funded experimentation with different managerial approaches and performance-appraisal techniques.¹² It is even truer for a public management sector like criminal justice, where the occasions for such experimentation have been severely limited both by fiscal constraints and frankly, by intellectually hide-bound management traditions.

A second step in this direction is to translate the agency's nonoperational goals into measurable process and performance criteria.

In a recent volume, a group of public management specialists analyzed "impossible jobs in public management."¹³ An "impossible job" is defined by-

• agency goals that are multiple, contradictory, and nonoperational
• strong political cross-pressures by competing external constituencies who favor one or another of the agency's goals to the exclusion of the rest
• a lack of public confidence in the professional expertise of agency workers
• clients whose mental and behavioral characteristics make them difficult to serve. Examples include the jobs of contemporary big-city police chiefs, State mental health commissioners, public health commissioners, human services (welfare, Medicaid) commissioners, and corrections commissioners. Although they varied widely across time, place, political and organizational context, and nature of task, the reputationally successful public managers in these fields were found to employ certain "coping strategies," chief among them a strategy for translating nonoperational goals into measurable process and performance criteria.
  

An example from corrections: The Federal Bureau of Prisons

A good example of this strategy in action is the Federal Bureau of Prisons (BOP). Established in 1930 and situated in the United States Department of Justice, the BOP is a career civil service agency. It has long enjoyed a reputation as one of the finest correctional agencies in the country, and one of the best managed agencies of the Federal Government. Contrary to the "Club Fed" stereotypes of the agency, the BOP has not always enjoyed "a better class of criminals," been less crowded, spent more per inmate, or had a lower inmate-to-staff ratio than State systems of comparable size and complexity. Instead, its success in running safe, clean, programmatic, and cost effective prisons has been the hard-won product of management innovations made under five consecutive directors.

Historically, the core of the BOP management approach has been to translate its multiple, contradictory, and nonoperational goals (punish, deter, incapacitate, rehabilitate) into measurable process and performance criteria. For example, the agency's fourth director, Norman A. Carlson (1970-87), was an award-winning Federal executive. He self-consciously managed the agency around three criteria: safety, humanity, and opportunity. Essentially, safety meant the incidence of things that either threatened the physical and emotional well-being of inmates and staff, or placed the public at direct risk, or both (assault, homicide, escape). Humanity meant the incidence of things that made for decent living and working conditions behind bars (decent food, clean quarters, recreation). Opportunity meant the incidence of things that enabled inmates to better themselves (programs in remedial reading, job training, drug treatment). Thus, in the recent agency publication Bureau of Prisons Goals for 1992 and Beyond, we read: "The Federal Bureau of Prisons protects society by confining offenders in the controlled environments of prisons and community-based facilities that are safe, humane, and ... provide work and other self-improvement opportunities...."

Many correctional agencies have such mission statements. But few have actually managed themselves around measurable process and performance criteria as the BOP has done from 1930 to the present. A good contemporary example is the agency's Key lndicators/Strategic Support System (KI/SSS). Begun in 1983 and carried out under Carlson's successor, J. Michael Quinlan (1987-92), the BOP has elaborated "safety, humanity, and opportunity" into KI/SSS, a personal computer-based management information system that provides managers at all levels with comprehensive, up-to-date, system-wide information on a wide range of specific performance variables. Invariably, such innovations begin with managers who-

• ask themselves basic questions about the relationship between what they want to achieve and what they actually do
• recognize the need to translate their agency's multiple, contradictory, and nonoperational goals into measurable performance criteria.

For example, in 1988, as the KI/SSS was still being developed, participants at the BOP's annual wardens' conference received a perceptive four-page progress report that read in part: "Unlike private agencies and companies, the Bureau of Prisons is expected to respond to a wide variety of overall objectives, often conflicting, often difficult to obtain.... The Bureau has stated that its mission is to provide humane control of inmates (and) ... opportunities to those inmates who choose to use them.... (W)e need to ... translate that broad goal into more specific measures."

Between 1983 and 1992, they did so, and it helped the agency to manage successfully a doubling in its prisoner population and a concomitant growth in the number of its staff and facilities.

The measurement net: How tight a weave?

Putting the example of the BOP to one side, but continuing for the moment with our focus on correctional agencies, the central question is "How do we define a comprehensive set of appropriate measures for the goals of corrections that can be used on a day-to-day basis by the field's practitioners?" The answer is far from obvious, but some progress
has already been made. The Study Group's Professor Charles H. Logan of the University of Connecticut has defined eight distinct "dimensions of the quality of confinement" (security, safety, order, care, activity, justice, conditions, management), each of which can be measured on four or more "subdimensions" via staff and inmate surveys and institutional records.¹⁴ Logan's work dispels one concern about the possibility of defining such measures and raises a second. On the one hand, it dispels the worry that any such measurement scheme is bound to be based exclusively on one or another moral or ideological view of the "ends of criminal justice." His quality of confinement measurements encompass and satisfy every major school of thought about "what prisons are for."

On the other hand, however, Logan mixes process and performance measurements in a way that may or may not represent the final state of the art. For example, his "security" dimension has six subdimensions: security procedures, freedom of inmate movement, community exposure, staffing adequacy, significant incidents, and drug use. Arguably, the first four of these security subdimensions are process measures, while the last two are performance measures. The same sort of indiscriminate mixing of process and performance measures is present in the BOP's aforementioned KI/SSS. Whether it is either intellectually worthwhile or practically useful to distinguish more cleanly between process and performance measures remains an open question.¹⁵

Be careful of what you measure, for you may (or may not) get it

But before one even begins to treat with such questions, at least four flags of caution should be waved over any effort to define appropriate measures for the goals of criminal justice agencies, or, for that matter, any governmental agencies.

First, it is naive to assume that what government agencies actually do on a day-to-day basis can be much affected by how, whether, or to what extent their managers define goals. In the preface to his study of administration in the Federal Bureau of Investigation (FBI) and the Drug Enforcement Agency (DEA), Wilson observes that the tasks of government workers "are supposedly chosen, defined, revised, or discarded as a result of efforts by administrators to achieve organizational goals," but "tasks are defined by forces far more profound, and accordingly more resistant to change...."¹⁶

Second, even if what government agencies actually did was a simple function of how their goals were defined and measured, that would not necessarily be a good thing in all cases. There is always a danger that measurement-driven government workers will, so to speak, "set up the target in order to facilitate shooting," and give rise to perverse and unintended administrative routines. Criminal justice agencies furnish many examples: FBI agents of the Hoover era who padded "the stats" on bank robberies; DEA agents in the 1970's who persisted in street-level "buy-and-bust" operations rather than more demanding investigative chores that might not succeed.¹⁷

Third, it is often the most dedicated and caring government workers who are anywhere from suspicious to downright dismissive of any attempt to define and apply such measures. For example, a hard-working veteran correctional counselor at a maximum-security prison dismissed the need for such measures and was heedless of what he judged to be the pseudo-intellectual justifications for them: "Experience on the ground is the only way to understand the effects of a place like this." An equally dedicated coworker of his stated: "I believe in this way of doing things. I'd do it this way even if it didn't make any difference."¹⁸ Similarly, a committed participant in a major community policing demonstration project was perturbed by the suggestion that the study would help to define appropriate measures for the goals of police agencies: "It's not about measuring anything, it's about doing things differently."¹⁹

Fourth, even under the best of conditions, efforts to act on appropriate measurements of agency goals can be blocked by personnel regulations and administrative behavior within government. Unfortunately, there are many examples. In a recent study of the Federal procurement process, Professor Steven Kelman of Harvard University studied managers who sought to withhold contracts for poor performance and secure the best possible deal for the Nation's taxpayers; paradoxically, they were defeated by procedural regulations intended to promote competition in contracting.²⁰ In another recent study, Gerald J. Garvey of Princeton University had a similar tale to tell about change-oriented management within Federal regulatory bureaucracies.²¹

None of these cautions, however, should be sufficient to discourage one from exploring new and better ways of defining appropriate measures for the goals of criminal justice agencies.