The Problem of Defining Agency Success

Bureau of Justice Statistics. Performance Measures for the Criminal Justice System. Washington,
DC: Article prepared for the U.S. Department of Justice, Bureau of Justice Assistance by James Q. Wilson; 1993. pp.157-167.

View entire document

The Problem of Defining Agency Success

by James Q. Wilson

For the most part, I agree with what DiIulio says in the first essay in this compendium, and I am generally sympathetic to the Study Group's ideas on performance measures for prisons and police departments. But let me reformulate the question this way: How can government agencies with nonoperational goals best develop performance measures?

The best way to approach this issue is to ask how private organizations with nonoperational goals (schools, colleges, hospitals) cope with the problem of defining success. In principle, they need not define it, for in a competitive market, success will be defined for them by survival or failure. The heads of most such organizations are unwilling to wait for this Darwinian verdict; instead, they try to anticipate it by looking at the following measures:

Client satisfaction (student enrollments and tuition payments, hospital accounts receivable, and so forth).
Donor contributions (Federal subventions, alumni donations, foundation grants).
Employee attachment (the ease or difficulty with which the organization can recruit and retain skilled-or at least highly credentialed or well-thought-of-employees).
Personal, up-close, or professional evaluations (impressions gained by executives as they move about the organization or read and hear the judgments of professional peers).
Proxy measures of goal attainment (for example, percentage of high school graduates accepted at elite colleges, SAT scores, Medicare reimbursements, cure rates).

Two things need to be said about this list. First, the proxy measures at the bottom of the list are somewhat crude and quite easily misinterpreted. A very good inner-city high school may send few students to Princeton but may have many graduates who get jobs at Lockheed or the Bank of America. The number who do, however, may be unknown because no routine reporting system captures it and the employment decision may come a year or two after graduation. In a hospital, the percentage of admitted patients who get better, given a certain disease, may say more about the personal habits and self-care of the patients than about what the hospital does. As the government increasingly subjects private agencies to constraints experienced by public agencies, private organizations are forced to place greater weight on crude proxy measurements like the ones mentioned above because the government places great weight on them.

Second, the executives of private organizations have more opportunity to engage in personal exploration of their organizational environment-so-called "soaking and poking" (soaking up information and poking into corners)- than do their public counterparts. This greater opportunity is because of three factors:

On the average, private organizations are smaller than public ones (both in total size and in size of the overhead staff that must be managed).
Private executives need to spend less time satisfying external political demands than do the heads of public agencies.
The compensation and tenure of private executives are more variable than those of public executives, and so the former have a greater incentive to find ways of anticipating (and, if necessary, correcting) adverse market trends. Good executives can be found among public agencies, but they are rarer. For example, the principals of more effective public schools are almost invariably ones who engage in a great deal of soaking and poking and who devote a lot of effort to student and teacher motivation. The literature on this subject makes it clear that this is not routine behavior for principals and often requires for its existence strong, even eccentric, personalities.

The case of big city police agencies

Most of the efforts to improve performance measures for policing have concentrated on finding either real measures of overall effectiveness or plausible proxy measures. Not much has come of these efforts for reasons that should be obvious. There are no "real" measures of overall success; what is measurable about the level of public order, safety, and amenity in a given large city can only partially, if at all, be affected by police behavior. (For example, if the murder or robbery rates go up, one cannot assume that this is the fault of the police; if they go down, one should not necessarily allow the police to take credit for it.) Proxy measures almost always turn out to be process measures- response time, arrest rates, or clearance rates-that may or may not have any relationship to crime rates or levels of public order.

In my view, the search for better measures of police performance is doomed to failure so long as it focuses on city-wide or even precinct-wide statistics. Most police chiefs will agree on this point, I think. No matter how we improve the Uniform Crime Reports (UCR) or the National Crime Victimization Survey (NCVS), they will not tell us very much (and certainly not very much in a timely fashion) about what difference the police make in the lives of the citizens.

Nor can most of the other devices available to private firms be applied to policing. There are no client payments or donor contributions; police appropriations tend to reflect fiscal constraints and past funding levels modified, from time to time, by generalized concerns over crime increases and police misconduct. Professional evaluations are not very reliable; policing is not a profession with clear standards enforced by knowledgeable peers, except on the rare occasions when an outside group is brought in to evaluate a department in trouble. But small-scale soaking and poking may be applicable.

These considerations lead me to suggest that the better approach to defining police goals and performance measures involves thinking small and from the bottom up.

Let me clarify this suggestion by asking what citizens want that is related to police behavior. They want several things, but in the interests of brevity I will focus on just one.¹ People want to live in safe, orderly neighborhoods. Wesley Skogan has shown in his recent book that there is a high degree of consensus across demographic categories as to what a safe, orderly neighborhood is.² It includes streets free of drug dealers, rowdy juveniles, threatening derelicts, soliciting prostitutes, and predatory criminals; buildings without graffiti or other signs of decay; no drive-by shootings, and so forth.

The police ought to make the production of safer, more orderly neighborhoods (not lower crime rates or more drug arrests or more traffic tickets) one of their goals. They ought to design ways of assessing the conditions of neighborhoods before and after various police interventions. They ought to use that assessment to modify their deployments and tactics.

1. I omit fair play, civil behavior, prompt response, and many other important factors. Taking all these into account would complicate my argument but not change it.

2. Wesley G. Skogan, Disorder and Decline: Crime and the Spiral of Decay in Amercian Neighborhoods (Berkeley and Los Angeles: University of California Press, 1990).

Adopting this view implies certain actions, none easy but none impossible:

Categorize all neighborhoods by two criteria: ³

--What is the level of disorder and danger?
--How amenable is that level to improvement by a given, feasible level of police and public action? (Some neighborhoods are almost self-policing and require only police response to calls for service. Some are almost beyond repair, given feasible levels of intervention, because they are highly transient or have other problems contributing to decay. Some are amenable to improvement, provided existing resources are carefully used.)

Specify as concretely as possible the features of a given neighborhood that are strategic and must be improved if the quality of life is to be improved. For example:

    --A landlord who allows his building to be used by drug dealers
    --A deli or spa that allows teenagers to hang out in front of it in ways others find threatening
    --A bar that is the scene of frequent brawls
    --A public housing project with poor maintenance and security
    --A bus or subway stop that is dominated by aggressive panhandlers or homeless people sleeping on public benches
    --A residential burglary gang that is working the neighborhood
    --A park that is the scene of muggings and rapes.

State the methods to be used to remedy the situation (for instance, filing nuisance abatement charges against the landlord, meeting with the teenagers and the deli owner about how the sidewalk is used, arranging to expel disorderly public-housing tenants, using saturation patrol for a limited period, stationing uniformed officers next to suspected drug dealers, and so forth).

Put in place a micro-level measure of success. These might include the following:

    --Tracking calls for service from a specific address or its immediate neighbors
    --Hiring a resident to make regular observations of life on the street by, for example, at stated intervals counting the number of panhandlers, suspected drug dealers, sleeping vagrants, soliciting prostitutes, and so on
    --Conducting a telephone survey of residents (using random-digit dialing to minimize sample selection bias) to assess their perceptions before and after the intervention.

These micro-measures are likely to be among the few valid measures of police performance. They may well lead to conclusions quite at variance with city-wide, aggregate data. For example, the "crime rate" might be getting worse at the same time that the conditions of life in neighborhoods has measurably improved, or vice versa.⁴ For a thoughtful and more detailed proposal along these lines, the reader should consult a recent essay by George Kelling, especially his account of Operation Crossroads in New York City's Times Square.⁵

Just as important as the measures themselves will be the requirement that the police agency define its operational goals as improving the conditions of life in specific neighborhoods. The attentive reader will have noticed that this way of defining police goals is similar to what many advocates of community-oriented policing (COP) propose. This is not an accident. But my argument is not that COP has such mystical, ideological, or historical importance that it ought to govern police decisions as a matter of principle. My argument is that if the police are serious about defining goals and measuring progress toward them, they will inevitably have to do so by identifying problems relevant to citizen concerns at the neighborhood level, specifying possible solutions, and measuring the effect of those strategies. This is what is meant by problem-oriented policing (POP) which, in my view, is the heart of COP and what makes POP (and COP properly defined) different from police-community relations.

Note the parallels between POP and the More Effective Schools (MES) literature. Both focus on small spaces (neighborhoods, individual schools); both require the identification of specific problems (unruly teenagers, gang fights, and so forth); both can be evaluated only by specifying significant changes in the behaviors of particular people (fewer fights, quieter classrooms); both try to improve the preconditions of better lives (safer neighborhoods, more orderly classrooms) without directly attacking ultimate outcomes (crime rates, SAT scores) which have multiple causes; both require that the executive (police captain, school principal) spend a lot of time observing and talking to the affected people. And, a final parallel: only a minority of police chiefs (like the minority of school officials) are likely to want to do these things.

3. I do not here define "neighborhood." This is an important issue that cannot be resolved conceptually. In rough terms, a neighborhood is what people think it is. The size and shape will vary from place to place and be affected by natural boundaries (for instance, parks, street-car tracks), institutions (grammar schools or shopping centers or churches), ethnic composition, and traffic patterns. I wish to stress that the area should be relatively small and its boundaries easily understood by most people living there.

4. Some will object that neighborhood measures ignore spillover effects. Up to a point, that is true. But such evidence as we have on burglary and robbery, drug-dealing, and other offenses suggests that the displacement of crime from one area to the next is usually much less than 100%. A lot will depend on the mobility of offenders, and that will be constrained by ethnic identity, profit opportunities, the desire for local "cover," the threats of rival gangs elsewhere, and the depressing effect of laziness and/or opportunism.

5. George Kelling, "Measuring What Matters," The City Journal (Spring 1992), 21-33.

About the author

Since 1985 James Q. Wilson has been the James Collins Professor of Management at the University of California at Los Angeles (UCLA). Before that he was for 26 years the Shattuck Professor of Government at Harvard University. He is the author or coauthor of 13 books, including The Moral Sense, Thinking About Crime, Varieties of Police Behavior, Political Organizations, Crime and Human Nature (with Richard J. Herrnstein), Bureaucracy, and Essays on Character. He has in addition edited or contributed to books on urban problems, government, regulation of business, and the prevention of delinquency among children, including Crime and Public Policy, From Children to Citizens: Families, Schools, and Delinquency Prevention (with Glenn Loury), Understanding and Controlling Crime (with David Farrington and Lloyd Ohlin), and Drugs and Crime (with Michael Tonry).

He has served on a number of national commissions concerned with public policy. He was chairman of the White House Task Force on Crime in 1966, chairman of the National Advisory Commission on Drug Abuse Prevention (1972-73), a member of the Attorney General's Task Force on Violent Crime in 1981, a member of the President's Foreign Intelligence Advisory Board (198590), and a director of The Police Foundation (1970-93).

He is currently chairman of the board of academic advisers of the American Enterprise Institute.

He has been elected a member of the American Academy of Arts and Sciences and a fellow of the American Philosophical Society. In 1990 he received the James Madison award for distinguished scholarship from the American Political Science Association and in 1991-92 served as that organization's president. Educated at the University of Redlands (A.B., 1952) and the University of Chicago (Ph.D., 1959), he has received honorary degrees from five universities. He grew up in Long Beach, Calif., and attended public schools there. From 1953 to 1955, he served in the U.S. Navy. He is married to the former Roberta Evans; they have two children.