|
Performance indicators shape an
evaluation, and the choice of performance indicators impacts the resources required to
conduct the study and the utility of the study results. For example, measuring changes in
public awareness of organ donation activities may not ultimately provide insight on what
impact the study activity had on donation rates. Because it is impossible to predict who
might become a potential organ donor before the occurrence of a traumatic event, any
activity targeted at the general public must cast a wide net in order to reach those few
people who will become potential organ donors. The complexity of an evaluation increases
because there are a wide variety of reasons that potential donors do not become actual
donors. These include not being identified as potential donors, caregivers not asking the
families for permission to retrieve organs, families denying consent, and organs
incorrectly deemed not transplantable (Gortmaker 1996). It is difficult to measure, with
any statistical significance, the effect of a population-based program on the actual
number of organs retrieved. The evaluation of population-based programs requires a careful
selection of performance indicators based on the goals and resources of the organization
conducting the evaluation. To overcome limitations
in measuring program effectiveness on actual organ retrieval, the organ donation community
has used three related sets of performance indicators, each with varying
Exhibit 3: Sample Performance Indicators
and Proximity to Donation
Source: Lewin, 1998
degrees of separation from the actual intended outcome of
organ donation. The three types of measures are shown in Exhibit 3 as concentric circles,
illustrating their relative proximity to the donation event. In addition, the US
population is used as an example to depict the relative sizes of the populations captured
in pre-event, post-event, and donation measures.
- Pre-event measures (represented by the outermost
circle) are used to gauge effectiveness in increasing organ donation before an actual
donation opportunity arises. The particular measures tend to be specific to the type of
activity being performed. Compared to the other types of performance indicators, pre-event
measures are most varied in nature, and are the farthest removed from the goal of
measuring increased organ donation.
- Post-event measures (the middle circle) are
commonly used in the organ donation community to measure the five crucial steps in organ
retrieval after a potential donor situation has occurred. Chronologically, and with
increasingly close ties to donation itself, these steps measure: 1) donor identification
rate, 2) referral rate, 3) request rate, 4) consent rate, and 5) retrieval rate.
- Donation rates (the innermost circle) represent
the most direct measure of the success of programs to increase the number of organs made
available for donation.
Pre-event measures are most commonly used to
evaluate activities that precede the trauma leading to brain death. Though frequently used
as a proxy for program effectiveness on donation rates, they are process measures that
serve as weak predictors of actual donation rates. A media campaign to increase the number
of people who become organ donors when they renew their driver's licenses could be deemed
successful if significantly more people signed donor cards than in months past. However,
the link between the number of new licenses with organ donation approval and the donation
rate is uncertain at best. A large follow-up study would be required to determine whether
the process of signing a donor card had an impact on a familys decision to donate
organs. The link between pre-event measures and the primary goal of increasing organ
donation is weak.
Many researchers are striving to improve pre-event
measures by using these measures to assess the five stages of behavioral change that may
lead to organ donation, i.e.: precontemplation, contemplation, preparation, action, and
maintenance. These five stages of change were developed by Prochaska et al. (1983),
originally in relation to smoking cessation programs. Studies in the areas of smoking
cessation, mammography use, and weight loss programs have attempted to: 1) design measures
for the stages of change, 2) determine if the stage of change correlates to success in the
designated program, and 3) determine if activities can be implemented to improve cycling
through the stages of change. These activities are evaluated relative to activities that
were not specifically designed to promote cycling through stages of change. In the field
of organ donation, these stages of change can be thought of as follows (Rohr, manuscript).
- Precontemplation: the person has not thought about
donating organs.
- Contemplation: the person has thought about
donating organs.
- Preparation: the person has made phone calls
or requested information about organ donation.
- Action: the person has taken action to express
his/her wishes about organ donation (e.g., signed a donor card, talked to a lawyer,
discussed wishes with family).
- Maintenance: the person may occasionally
remind or reinforce statement of wishes.
Multiple studies have indicated that familial consent for
organ donation is one of the biggest barriers to donation. Other papers have shown that
families are more likely to consent to donation if the deceased individuals have made
their wishes known. Applying the stages of change methodology to measure progress towards
action may be a useful indicator of outcomes. For example, stages of changes measures can
be developed to determine whether a particular intervention moved people to sign donor
cards and discuss organ donation with their families.
Another group of broadly applicable measures relates
to the various steps of the organ retrieval process: identification, referral, request,
consent, and retrieval. Because these steps are more immediate to the act of donation,
they are more meaningful, though still intermediate, indicators of the effectiveness of an
organ donation activity. Also, because they are common to many different kinds of
programs, they allow for more meaningful comparison across programs than pre-event
measures. Each of the rates is described in detail below, with comments about potential
uses.
The first step of the retrieval process is the
recognition of a patient as a potential donor of viable organs. The identification rate is
the number of potential donors identified divided by the total number of actual potential
donors, as determined by medical record review (MRR).
- The identification rate is a potential measure of the
effectiveness of provider education programs that seek to improve the ability of medical
staff to properly recognize the fulfillment of donor criteria in a patient.
The second step of the retrieval process is the
referral of a potential donor to an OPO. The referral rate is the number of potential
donors referred to an OPO divided by either total potential donors (determined by MRR) or
by total identified donors (determined from hospital data).
- The referral rate can be used to evaluate the effectiveness
of Required Referral and Required Request legislation. Although the intent of such
legislation is to increase donation by involving expert OPO staff more frequently and
earlier in the donation process, it is referral rate, rather than donation rate, that best
indicates the impact of the legislation.
The third step of the retrieval process is the
request made by hospital or OPO staff of the potential donors family. The request
rate is the number of potential donors whose families are asked to donate divided either
by total potential donors (determined by MRR) or by total identified or referred potential
donors (determined from OPO or hospital data).
- The request rate can also be used to evaluate how well
doctors and OPO staff respond to new Required Request legislation. While there are few
intervening factors to make consent and donation themselves the primary measure of
interest in an evaluation, the request rate, cross-referenced with these behaviors, is a
useful tool for use in hospital and OPO-level quality review and intervention.
The fourth step of the retrieval process is the
consent granted for donation by the family of the potential donor. The consent rate is the
number of consenting families of potential donors divided either by total potential donors
(determined by MRR) or by total identified or referred potential donors (determined from
OPO or hospital data).
- Requests decoupled from explanations of brain death,
requests made in a private, quiet setting, and requests made of minority families by
minority requestors have been correlated with higher consent and donation rates (e.g.,
Beasley 1997 and Kappel 1993). Because steps in the retrieval process after consent
are affected largely by scientific matters such as tissue preservation and not by factors
influenced by most donation activities (predisposition through request), consent rate is
considered by many donation professionals to be the de facto outcome measure of
programs to increase organ donation.
The final step of the retrieval process is the
retrieval of a viable organ from a donor. While regularly collected by OPOs to determine
the efficiency of their retrieval processes, the primary factor in the difference between
consent rate and donation rate lies in medical procedures to ensure tissue and organ
viability.
- Due to scientific concerns such as tissue preservation and
autopsy requests, the retrieval rate is not regularly used in place of either consent or
donation in the evaluation of programs designed to increase donation.
The organ donation rate is a measure of the eventual
goal of all donation-related activities: an increase in successful donations. Any activity
that can be shown to independently increase donation is clearly successful.
The most precise definition of donation rate uses
the actual number of potential donors as its denominator, whether or not the potential
donors had actually been asked or even identified by procurement staff. The size of the
potential donor pool is best calculated by retrospective medical record review (MRR).
- This measure is particularly useful for hospital and
OPO-level evaluation, yielding measures not only of the donation rate but also of
post-event measures, such as consent rate. However, the cost and time-intensive nature of
the MRR is prohibitive for national-level donation studies.
A less expensive and faster alternative to
estimating the potential donor pool involves using the number of deaths attributed by
death certificate codes to causes that suggest potential donors. These data are relatively
accessible from national mortality databases or state and local offices of vital
statistics.
- The availability of death certificate data on the national
level allows national rates to be determined consistently with state and local rates. Some
researchers have expressed concern that the estimation is compromised by two problems:
inconsistency in death coding patterns in different areas of the country and an
insufficient definition of potential donors, particularly since there is no authoritative
coding that assures potential for donation.
The crudest outcome measure, and the one most widely
used, is donors per million population (DPMP). It requires the assumption that potential
donors are more or less equally distributed over a population, so that the population
itself can be used as a rough proxy of the potential donor pool. The calculation of this
measure is the easiest of the three outcome measures described here, requiring only census
data and the number of donations in a specified geographical area.
- While the simplicity of this approach has made it the most
recognized of measures, including use internationally and by the US DHHS in OPO
certification, the use of unadjusted population size raises significant methodological
problems. For example, any state with higher than average proportions of residents older
than the allowable age for donation (e.g., Florida) or with a lower than average rate of
trauma death will have an artificially high denominator in the DPMP rate, thus deflating
its true procurement efficiency with "potential donors" who cant be
realized. Despite the ease of calculating DPMP, its lack of comparability across
geographic borders compromises its utility for meaningful evaluation.
|