Detailed Information on the
Juvenile Justice Programs Assessment

Making Juvenile Justice Programs' performance results available to the public through program publications and the internet.

Including performance information in budget submissions to better link resources requested to program performance goals.

Developing a comprehensive evaluation plan for the Juvenile Justice Programs to obtain better information on the programs' impacts.

Measure: Percent of program youth who offend or reoffend

Explanation:This measure is for grants which provide funds for direct service delinquency prevention or intervention programs. Offense is defined as "arrest or appearance at juvenile court for a new delinquent offense." Official records (law enforcement, courts) are the preferred data source. Baseline is comprised of a meta-analysis of juvenile justice

Measure: Percent of states and territories that are determined to be in compliance with the four Core Requirements of the Juvenile Justice and Delinquency Prevention Act of 2002.

Explanation:The Core Requirements are: deinstitutionalization of status offenders and nonoffenders; sight and sound separation of juveniles and adults; removal of juveniles from jails and lockups; and disproportionate minority contact with the juvenile justice system.

Measure: Percent of program youth who exhibit a desired change in the targeted behavior.

Explanation:The purpose of this measure is to determine the extent to which OJJDP-funded programs are facilitating changes in youth behaviors or conditions which would make them less likely to commit delinquent acts in the future. Targeted behaviors differ, depending upon the specific grant program's purpose, and reflect risk and protective factors for delinquency. Through grants and sub-grants to states, localities and organizations, OJJDP supports numerous Direct Service Programs, both prevention and intervention. These include school-based, community-based and residential programs, and represent an array of different activities and targeted populations, such as mentoring, family strengthening initiatives, gang intervention projects or job training. The one thing that all OJJDP-funded Direct Service programs have in common is that they are seeking to facilitate a change in a youth's behavior and/or condition in order to reduce that youth's likelihood of committing future delinquent acts and promote his or her chances at a positive future. Because of the wide variety of different program types funded with OJJDP grants this measure tracks whether the behavior being monitored by the program is changing in a desired manner - that is, either it is being reduced or increased (depending upon whether it is a risk or protective factor). Although OJJDP reports the extent of desired change as a single measure, it represents a collection of several smaller measures - each representing a portion of OJJDP direct service grants - that can be analyzed at a specific behavior level. Specific behaviors monitored and reported against the performance measure include: improvement in social competence; improvement in school attendance; improvement in Grade Point Average; attainment of a GED (High School Equivalency); High School completion; improvement in job skills; improvement in employment status; no teen pregnancy; improvement in family relationships; improvement in family functioning; decrease in antisocial behavior; decrease/non-initiation of substance use; decrease/non-initiation of gang related activities. The behavior selected by the grantee must be relevant to the purpose of the activities which are funded by OJJDP grant funds. This is a mandatory performance measure for all OJJDP direct service grantee or subgrantee programs. OJJDP analyzed data using two different methods to assist in establishing a target. Results of both analyses are listed below.

Measure: Percent of grantees implementing one or more evidence-based programs.

Explanation:Title II and Title V evidenced-based programming is promoted in accordance with 42 U.S.C. ?? 5633(a)(21) and 5783 (a)(8), (c)(5). JJDP Act of 2002 requires that State agencies who receive OJJDP Formula Grant funds ". . . give priority in funding to programs and activities that are based on rigorous, systematic, and objective research that is scientifically based." 42 U.S.C. ?? 5633(a)(21). OJJDP has expanded the intent of these requirements to all OJJDP-funded programs. This performance measure tracks the extent to which OJJDP-funded programs are implementing programs which have been shown to prevent and/or reduce delinquency and associated negative outcomes.

Measure: Average number of processing days by program for grant awards.

Explanation:"Processing Days" reflect the time period from submit date to award date. Between 2004 and 2005, the average number of processing days dropped from 110 to 78 (a drop of nearly 30%). This is a considerable improvement over a one year period.

Measure: Percent of funds allocated to grantees implementing one or more evidence-based programs

Explanation:Twenty percent of OJJDP grant funds were allocated to grantees who implemented evidence-based programs, representing $34,981,658 in OJJDP funds allocated to grantees awarded in 2004 and reporting to OJJDP in 2005. This measure only has one year of data, but OJJDP intends to track this measure in future years. As discussed earlier (in Section 4.2) for grants awarded in 2004 and reporting in 2005, grantees were not required to report their use of evidence-based programs to OJJDP (this became a mandatory reporting requirement for those awarded in 2005 and reporting in 2006). This includes all Formula grants which are required to give priority to funding evidence-based practices. As a result, OJJDP's baseline percentage is likely an undercount of the actual use of evidence-based programs.

Is the program purpose clear?

Explanation: In 2002, the Juvenile Justice and Delinquency Prevention (JJDP) Act (42 U.S.C. § 5601 et seq.) was reauthorized (the 21st Century Department of Justice Appropriations Authorization Act, Pub. L. No. 107-273). Section 102 of the Act provides that the purpose of the Juvenile Justice Programs is "1) to support State and local programs that prevent juvenile involvement in delinquent behavior; 2) to assist State and local governments in promoting public safety by encouraging accountability for acts of juvenile delinquency; and 3) to assist State and local governments in addressing juvenile crime through the provision of technical assistance, research, training, evaluation, and the dissemination of information on effective programs for combating juvenile delinquency. In 2004, DOJ's Office of Juvenile Justice and Delinquency Prevention (OJJDP) undertook a strategic planning process for these programs that included the formal adoption of a mission statement and agency strategic goals in line with the programs' authorization. Authorized under the JJDP Act of 2002 (42 U.S.C. § 5601 et seq.), the Title II Formula Grants Program supports state and local delinquency prevention and intervention efforts and juvenile justice system improvements. Through this program, OJJDP provides funds directly to states, territories, and the District of Columbia to help them implement comprehensive state juvenile justice plans based on detailed studies of needs in their jurisdictions. Also authorized under the JJDP Act of 2002 (Subchapter V; 42 U.S.C. § 5781 et seq.), the Title V Community Prevention Grants Program is a Federal grants program to fund collaborative, community-based delinquency prevention efforts. The program provides communities with funding and a guiding framework for developing and implementing comprehensive 3-year juvenile delinquency prevention plans designed to reduce risk factors associated with juvenile delinquency and decrease the incidence of juvenile problem behavior. Authorized under the JJDP Act of 2002 (42 U.S.C. §§ 5665-68), the Part E program (also known as the discretionary grants program) provides "grants and contracts to States, units of general government, Indian tribal governments, public and private agencies, organizations, and individuals, or a combination thereof, to support projects for the development, testing, and demonstration of promising initiatives and programs for the prevention, control, or reduction of juvenile delinquency."

Evidence: OJJDP authorizing legislation is the Juvenile Justice and Delinquency Prevention Act, 42 U.S.C. § 5601 et al. and can be accessed at http://ojjdp.ncjrs.org/about/jjdpa2002titlev.pdf. Widely disseminated to the general public and others through OJJDP's website, OJJDP's mission statement can be accessed at http://ojjdp.ncjrs.org/about/missionstatement.html. Title II and Title V evidenced-based programming is promoted in accordance with 42 U.S.C. §§ 5633(a)(21) and 5783 (a)(8), (c)(5). US Department of Justice's Strategic Plan for Fiscal Years (2003-2008) can be accessed at http://www.usdoj.gov/jmd/mps/strategic2003-2008. OJJDP strategic planning activities are ongoing and are reflected in the following "working" drafts: Outline OJJDP Strategic Plan; OJJDP Strategic Operational Plan; Draft Framework for Three - Five Year Plan. Additional Information on major OJJDP program can be accessed at http://ojjdp.ncjrs.org/programs/ProgSummary.asp?pi=16&ti=&si=&kw=&PreviousPage=ProgResults (Title II); http://ojjdp.ncjrs.gov/titlev/index.html (Title V).

Does the program address a specific and existing problem, interest, or need?

Explanation: While juvenile crime and violence peaked in the mid-1990s and has since dropped, delinquency and youth violence continue to be issues of concern at the national, state and local levels. The Juvenile Justice Programs were established to help states and communities prevent and control delinquency and improve their juvenile justice systems. While these programs (formula, earmarks, and competitive) may differ in the specific issues they target (e.g., mentoring, underage drinking, gang prevention and suppression, treatment services for juveniles in correctional programs, tribal youth concerns) their overall goal is the same??to prevent and respond to juvenile delinquency and victimization. Many states and localities rely on these programs for training and technical assistance and other guidance to address their juvenile crime issues. OJJDP has developed a number of key resources for states and others to promote the development and implementation of best and effective practices and program models. The Administration also has identified juvenile crime (e.g., gang violence) and related issues as a significant national concern. In October 2003, the White House Task Force on Disadvantaged Youth issued its Final Report detailing a plan for a "comprehensive Federal response to the problems of youth failure, under existing authorities and programs, with a focus on enhanced agency accountability and effectiveness."

Evidence: Juvenile Offenders and Victims: 2006 National Report was released by OJJDP in late March 2006. This publication draws on scientifically sound data, the latest statistics, and relevant research to provide a comprehensive view of juvenile crime across the nation. The report includes seven chapters which summarize information on juvenile population characteristics, juvenile victims, juvenile offenders, the juvenile justice system structure and process, law enforcement and juvenile crime, juvenile offenders in court, and juvenile offenders in correctional facilities. Citing FBI and other data sources, the report demonstrates that the rate of juvenile violent crime arrests has consistently decreased since 1994, falling to a level not seen since at least the 1970s. However, during this period of overall decline in juvenile violence, the female proportion of juvenile violent crime arrests has increased, marking an important change in the types of youth entering the juvenile justice system and in their programming needs. The report also points out that although juvenile crime has dropped nationwide over the past several years, in some localities juvenile violence has remained high or even increased. OJJDP has developed a special web site for the Report, www.ojjdp.ncjrs.gov/ojstatbb/nr2006/index.html. As a companion to the National Report, OJJDP maintains an online Juvenile Justice Statistical Briefing Book at http://ojjdp.ncjrs.org/ojstatbb/index.html which provides basic statistical information on the same topic areas covered in the National Report. There remains a high demand for OJJDP training and technical assistance, its publications, and other resources, reflecting the overall nation's concern for addressing the needs of this nation's children, youth and families, especially those at risk for or involved in delinquent behaviors. The number of individuals who visit the OJJDP website monthly ranges into the millions. Most recently, in January and February 2006 (combined), over eight million individuals visited the OJJDP website. See "Monthly Hits to OJJDP Web Sites." Information on juvenile justice issues is also disseminated to the field through the OJJDP JUVJUST listserv and through links on the Coordinating Council web site (http://www.juvenilecouncil.gov/)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The primary responsibility for the administration of juvenile justice and the prevention of juvenile delinquency remain with state and local governments. However, with the assistance of schools, communities, social service agencies and families, the Juvenile Justice Programs represent "seed" money that help to implement innovative, evidenced-based programs, as well as foster state compliance with national objectives. These objectives include deinstitutionalization of status offenders, separation of juveniles and adults in secure institutions, removal of juveniles from adult jails and lockups, and reduction of disproportionate minority contact, where it exists. Meeting these core objectives is essential to creating a fair, consistent, and effective juvenile justice system. To diminish potential redundancy within the federal system, OJJDP provides leadership within the Coordinating Council on Juvenile Justice and Delinquency Prevention. The function of the Council is to "coordinate all Federal juvenile delinquency programs (in cooperation with State and local juvenile justice programs), all Federal programs and activities that detain or care for unaccompanied juveniles, and all Federal programs relating to missing and exploited children." Federal efforts to coordinate also are reflected at the state level, as state and local authorities' input on juvenile justice-related concerns is received through a Federal Advisory Committee. The Federal Advisory Committee on Juvenile Justice is a consultative body composed of appointed representatives of the nation's State Advisory Groups. The committee advises on matters related to juvenile justice, evaluates the progress and accomplishments of juvenile justice activities and projects, and provides input on the Juvenile Justice programs.

Evidence: OJJDP authorizing legislation is the Juvenile Justice and Delinquency Prevention Act, 42 U.S.C. § 5601 et al. and can be accessed at http://ojjdp.ncjrs.org/about/jjdpa2002titlev.pdf. Legal citations specific to the OJJDP's federal concentration of efforts and Title II Core requirements are: Coordination and consultation with other agencies: 42 U.S.C. §5614(a), (b), (c), & (f). Deinstitutionalization of Status Offenders (DSO): 42 U.S.C. §5633(a)(11) Sight and Sound Separation: 42 U.S.C. §5633(a)(12) Jail Removal: 42 U.S.C. §5633(a)(13) Disproportionate Minority Contact (DMC): 42 U.S.C. §5633(a)(22). With the passage of the JJDP Act in 1974 and its subsequent reauthorizations, no other federal agency has been charged with the statutory responsibility of assisting states in achieving compliance with the core requirements of the Act. One important element in the 1974 Act was to protect juveniles in the juvenile justice system from inappropriate placements and from the harm, both physical and psychological, that can occur as a result of exposure to adult criminal offenders. Another important element of the JJDP Act emphasized the need for community-based treatment for juvenile offenders. In passing the JJDP Act, Congress recognized that keeping children in the community is critical to their successful treatment. The JJDP Act, as amended in 2002, continues four core requirements with which participating States and Territories must comply to receive grants under the JJDP Act. Additional information on OJJDP's Coordinating Council on Juvenile Justice and Delinquency Prevention can be obtained at http://www.juvenilecouncil.gov. Additional information on the Federal Advisory Committee on Juvenile Justice can be obtained at http://ojjdp.ncjrs.org/facjj/index.html. The Federal Advisory Committee on Juvenile Justice: Annual Report 2005 to the President and Congress of the United States represents the consensus thinking of the represented states and jurisdictions on juvenile justice. The report highlights serious concerns and issues and recommends legislative and administrative solutions. See http://www.ojjdp.ncjrs.org/facjj/annualreports/212757.pdf. The OJJDP Annual Report 2003-2004 provides background on state compliance with core requirements of JJDP Act (p. 36) http://www.ncjrs.gov/pdffiles1/ojjdp/206630.pdf

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Juvenile Justice Programs are generally well-designed programs with components that ensure effectiveness and efficiency. Formula grant recipients (the bulk of the Juvenile Justice Programs funding) submit state plans that detail the need for juvenile justice programming in their states and outline plans for distribution of the grant funds. OJJDP uses a competitive process to award the discretionary funds for demonstration programs, research activities, and training and technical assistance. OJJDP holds all grantees accountable for measuring performance. All funding solicitations, including those involving both discretionary and non-discretionary programs, now require that applicants (and grantees/subgrantees) identify performance measures in their applications and report on these measures in their periodic progress reports to the agency. Each solicitation includes guidelines and requirements for applicants regarding performance measures reporting. By 2006, funding solicitations clearly identified the required mandatory performance measures for each program category. Also, in an effort to understand more about the impact of congressionally earmarked programs, and enhance the accountability of these funds, OJJDP initiated an effort with OJP's National Institute of Justice to conduct rigorous evaluations on a sample of these programs.

Evidence: The JJDP Act allows for an effective, efficient, and varied approach to the administration of OJJDP program funding. It provides for agency oversight in administration of Title II formula and block grant programs to the States that ensures compliance with the JJDP Act's core requirements and at the same time allows States to work in partnership with OJJDP and have flexibility in addressing unique community needs. 42 U.S.C. §§5631-33. It provides OJJDP leadership with discretion in the administration of discretionary grants programs designed to enhance the field's knowledge of effective programming based on OJJDP's research, statistics, evaluation, and demonstration programs and through training and technical assistance to juvenile justice and related constituencies. 42 U.S.C. §§5661-68. Most importantly, the 2002 JJDP Act reauthorization gives OJJDP additional authority to promote state's and others' use of evidenced-based programming under Title II and Title V. 42 U.S.C. §§ 5633(a)(21) and 5783 (a)(8), (c)(5). The website link to the OJJDP earmark solicitation is http://ojjdp.ncjrs.gov/grants/solicitations/2006CongEarmark.pdf. OJJDP authorizing legislation can be accessed at http://ojjdp.ncjrs.org/about/jjdpa2002titlev.pdf.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Almost one third (31 percent) of Juvenile Justice programs funding (2006 appropriation) is allocated to congressional earmarks under the Title II Part E Demonstration Grants program. The earmarking process inhibits the effective targeting of resources to States and localities based on need.

Evidence: See the Conference Report for the 2006 State, Justice, and Commerce, and related agencies appropriation for percentages of Juvenile Justice program funding allocated to major programs, as well as detail on congressional earmarking for the Title II Part E Demonstration Grants, at: http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi? dbname=109_cong_reports&docid=f:hr272.109.pdf

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Juvenile Justice Programs use two long-term measures that tie directly to the goals and purposes of the programs to gauge performance: Long Term Measure #1: Percent of program youth who offend or reoffend. Offending/reoffending, defined as "arrest or appearance at juvenile court for a new delinquent offense," is a universal mandatory measure for all of the Juvenile Justice-funded direct service grants. To comply with performance measure requirements, the grant-funded programs track the levels of new offending among all youth program participants, and report them to OJJDP as part of their periodic progress reports. Prevention programs report on levels of offending by program youth and intervention programs report on levels of youth re-offending. Official records (police, juvenile court) are the preferred data source. Long Term Measure #2: Percent of states and territories that are determined to be in compliance with the four Core Requirements of the JJDP Act. The Core Requirements are: deinstitutionalization of status offenders and nonoffenders, sight and sound separation of incarcerated juveniles and adults, removal of juveniles from adult jails and lockups, and disproportionate minority contact with the juvenile justice system. State and territorial compliance with the core requirements is used as a fundamental benchmark to determine the impact of the Juvenile Justice Programs' effectiveness in "support[ing] states and communities in their efforts to . . . improve the juvenile justice system so that it protects public safety, holds offenders accountable, and provides treatment and rehabilitative services tailored to the needs of juveniles and their families." If states can demonstrate full compliance with minimal de minimus exceptions, they have will have met core goals of the Juvenile Justice Programs.

Evidence: The first long term measure directly addresses the purpose of the Juvenile Justice Programs' authorizing legislation, stated under Sec. 102 of 42 U.S.C. 5602: (1) To support State and local programs that prevent juvenile involvement in delinquent behavior; (2) To assist State and local governments in promoting public safety by encouraging accountability for acts of juvenile delinquency; and (3) To assist State and local governments in addressing juvenile crime through the provision of technical assistance, research, training, evaluation and the dissemination of information on effective programs for combating juvenile delinquency. The second long term measure also is directly tied to the programs' purpose. When it passed the JJDP Act, Congress outlined conditions for state participation in the Formula Grants Program, requiring (1) deinstitutionalization of status offenders and nonoffenders, and (2) the separation of juvenile delinquents from adult offenders. The "disproportionate minority confinement" (DMC) requirement (added in 1988) originally focused on the extent to which minority youth were confined in proportions greater than their representation in the population. The 2002 reauthorization of the JJDP Act broadened the DMC concept to encompass disproportionate minority CONTACT with the juvenile justice system. The Juvenile Justice Programs' authorizing legislation can be accessed at http://www.ojjdp.ncjrs.org/compliance/appendixa.pdf. Legal citations specific to the Juvenile Justice Title II Core requirements are: Deinstitutionalization of Status Offenders (DSO): 42 U.S.C. § 5633(a)(11); Sight and Sound Separation: 42 U.S.C. § 5633(a)(12); Jail Removal: 42 U.S.C. § 5633(a)(13); and Disproportionate Minority Contact (DMC): 42 U.S.C. § 5633(a)(22).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Ambitious targets and timeframes have been established for both of the Juvenile Justice Programs' long-term measures. The targets for the long-term measures are described below, and are also outlined in the Performance Measures section of this assessment. Long Term Measure #1: Percent of OJJDP program youth who offend or reoffend five year target: By 2011, no more than 33% of program youth will offend or reoffend. Baseline data, which reflect both reoffending and first time offending, show rates of 40%, or well above this level. The increase over baseline is about 18%, which represents an ambitious improvement goal. Long Term Measure #2: Percent of U.S. States and territories that are determined to be in compliance with the four Core Requirements of the JJDP Act five year target: By 2010, 100% of states and territories will meet minimal compliance for the four core requirements of the JJDP Act. This target was established based on an expectation of gradual improvement in core compliance. Each year, states/territories may fall out of compliance with one of the requirements and put eligibility for program funding in jeopardy. Therefore, the challenge for these programs is not increasing the level of overall compliance, but also maintaining compliance among states and territories that have already achieved relatively high compliance in the past.

Evidence: These targets were formulated using an analysis of performance measure data collected from Juvenile Justice Program grants and an extensive review of research literature on delinquency programs that have demonstrated effectiveness (through rigorous evaluation methods) in preventing or reducing juvenile offending and associated risk factors. Specific resources: Updated PART Analysis Memo (dated 4/11/06); Center for Evaluation Research Methodology at Vanderbilt Institute for Public Policy (see http://www.vanderbilt.edu/cerm/), which conducted a meta-analysis of over 500 controlled studies of the effectiveness of delinquency programs, concluding that recidivism rates for youth in programs with the highest level of evidence could anticipate a rate of 21% (compared to comparable youth not in a program (rate of about 40%). Related publications: Howell, J.C., & Lipsey, M.W. (2004). A practical approach to improving juvenile justice system programs. Juvenile and Family Court Journal. Vol. 55 (No. 1), Pgs. 35-48. Howell, J.C. and Lipsey, M.W. (2004) A Practical Approach to Linking Graduated Sanctions with a Continuum of Effective Programs. Juvenile Sanctions Center Training and Technical Assistance Center Bulletin, Vol. 2, (No. 1). (available at http://www.ncjfcj.org/content/view/515/331/). OJJDP's Model Programs Guide (MPG) (http://www.dsgonline.com/mpg2.5/mpg_index.htm) designed to assist communities in implementing evidence-based prevention and intervention programs for youth and families. It includes summaries of program evaluations, and includes information about rates of recidivism and behavior improvement; Center for the Study and Prevention of Violence's Blueprints for Violence Prevention (http://www.colorado.edu/cspv/blueprints/). The Blueprints for Violence Prevention project provides detailed information about Model and Promising programs, including rates of recidivism and levels of reduction or enhancement in targeted behaviors, such as substance use and academic improvements; The Helping America's Youth (HAY) Program Tool (http://www.helpingamericasyouth.gov/programtool.cfm), which contains information about risk factors, protective factors, and programs that have been evaluated and found to work. Programs must demonstrate results in accordance with widely accepted scientific criteria for program effectiveness; List of key references utilized in establishing targets (Research_References_for_LongTermTarget1.doc). Information used to establish targets for Long Term Measure #2: The Juvenile Justice Programs' authorizing legislation is the Juvenile Justice and Delinquency Prevention Act, 42 U.S.C. § 5601 et. al. and can be accessed at http://www.ojjdp.ncjrs.org/compliance/appendixa.pdf. Legal citations specific to the programs' Title II Core requirements are: Deinstitutionalization of Status Offenders (DSO): 42 U.S.C. §5633(a)(11); Sight and Sound Separation: 42 U.S.C. §5633(a)(12); Jail Removal: 42 U.S.C. §5633(a)(13); and Disproportionate Minority Contact (DMC): 42 U.S.C. §5633(a)(22). Since the inception of the Juvenile Justice Programs, an important goal has been full state and territorial compliance with these four requirements. The long-term target of 100% is thereby statutorily mandated. Regulations, Standards for Compliance and other supporting materials related to the Core Requirements are available at the Compliance Monitoring website at http://www.ojjdp.ncjrs.org/compliance/index.html. The following two reports were also used to establish targets for Long-Term measure #2: FY 2005 Determination of State Compliance with the Juvenile Justice and Delinquency Prevention Act of 2002; and FY 2004 Determination of State Compliance with the Juvenile Justice and Delinquency Prevention Act of 1974.*

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Three major annual performance measures have been developed for the Juvenile Justice Programs, which are directly linked to the long term measures. Annual measure #1 is the Percent of program youth who exhibit a desired change in the targeted behavior. This measure determines the extent to which funded programs are facilitating changes in youth behaviors or conditions, which would make them less likely to commit delinquent acts in the future. Targeted behaviors differ, depending upon the specific grant program's purpose, and reflect risk and protective factors for delinquency. Through grants and sub-grants to states, localities and other organizations, the Juvenile Justice Programs supports numerous direct service programs. What these programs have in common is that they attempt to help modify a youth's behavior and/or condition to reduce that youth's likelihood of committing future delinquent acts and promote his or her chances of a positive future. Because of the wide variety of different program types funded with the Juvenile Justice Programs' grants, this measure tracks whether the behavior being monitored by the program is changing in a desired manner - that is, either it is being reduced or increased. Although the extent of desired change is reported as a single measure, it represents a collection of several individualized measures - each representing a portion of the Juvenile Justice Programs' direct service grants, which can be analyzed at a specific behavior level. The behavior selected by the grantee must be relevant to the purpose of the activities funded by the grants. Annual Measure #2 is Percent of states and territories that are determined to be in compliance with the four Core Requirements of the JJDP Act. The Core Requirements are: deinstitutionalization of status offenders and nonoffenders, sight and sound separation of juveniles and adults, removal of juveniles from jails and lockups, and disproportionate minority contact (DMC) with the juvenile justice system. State and territorial compliance with the core requirements is a useful benchmark to determine the impact of the Juvenile Justice Programs' effectiveness in "support[ing] states and communities in their efforts to . . . improve the juvenile justice system so that it protects public safety, holds offenders accountable, and provides treatment and rehabilitative services tailored to the needs of juveniles and their families." If states can demonstrate compliance, they have will have met core goals of the Juvenile Justice Programs. Annual Measure #3 is Percent of grant implementing one or more evidence-based programs. The Juvenile Justice and Delinquency Prevention Act of 2002 requires that state agencies receiving Juvenile Justice Formula Grant funds "... give priority in funding to programs and activities that are based on rigorous, systematic, and objective research that is scientifically based." OJJDP has expanded the intent of these requirements to all of the Juvenile Justice Programs. This performance measure tracks the extent to which funded programs are implementing programs that have been shown to prevent and/or reduce delinquency and associated negative outcomes.

Evidence: Annual Measure #1 is an important measure for all direct service juvenile justice programs, especially those that are prevention programs. Direct service intervention programs typically serve youth who have already recorded an offense, either through an arrest and/or court referral. Prevention programs, however, may target the general population of youth or youth who are at-risk of becoming involved in delinquent behavior, but who do not already have a record of offending. Prevention programs also often target youth who are very young (under 12) and, as a result, would typically not be formally referred to the juvenile justice system, even if they did commit a delinquent act. Therefore, this measure is a good method to determine whether the Juvenile Justice Programs are having an impact in reducing the likelihood of future offending among these youth. Annual Measure #1 also is a key indicator of progress toward Long Term Measure #1 (see Question 2.1) for youth enrolled in direct service intervention programs. To determine whether progress is being made toward the long-term goal of reduced offending, it is necessary to track whether intervention program youth are demonstrating desired changes in behavior (an indicator that the likelihood of future reoffending is reduced). OJJDP plans to track each of the indicated behaviors separately, but will report this measure ("desired change in targeted behaviors") as a single, aggregate measure. Annual Measure#2 is also Long Term Measure #2. OJJDP tracks the percent of states and territories that are in compliance with the four Core Requirements of the JJDP Act annually. Compliance with each requirement is tracked separately, but the annual and long-term goal is full compliance with all requirements among all states and territories. Annual tracking of compliance is a key factor in achieving long-term performance improvements. OJJDP determines states' compliance status based on their annual compliance monitoring reports on the first three core requirements and their accomplishments in reducing disproportionate minority contact (DMC) as documented in their annual DMC Compliance Plans. Any annual reduction in compliance (either a compliance reduction at the state level, or a reduction in compliance with a single requirement among multiple states) is an important alert that additional focus must be given to ensure that juvenile offenders are being dealt with appropriately and fairly. OJJDP has a powerful mechanism - the ability to reduce a state's Formula Grant allocation - to ensure compliance. OJJDP intends to use Annual Measure #3 to track progress in measuring the extent to which evidence-based programs are being incorporated into delinquency prevention programming and juvenile justice systems supported by OJJDP funding. This measure is a mandatory measure for OJJDP grant-funded direct service and system improvement programs. While direct service activities account for the majority of funds awarded, system improvement activities play an integral role in facilitating improvements in behavior and reductions in offending among youth, by taking lessons learned from the research regarding effective programs and strategies, and translating them into policies and practices across juvenile justice and delinquency prevention systems nationwide.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and targets have been established for the Juvenile Justice Program's annual measures: Annual Measure #1 is Percent of program youth who exhibit a desired change in the targeted behavior. The baseline for 2004 is 35%. OJJDP has set a target for improvement of two percentage points per year. For example, 37% is the target for 2005 (and reported actual), and 39% is the target for 2006. Annual Measure #2 is Percent of states and territories that are determined to be in compliance with the four Core Requirements of the JJDP Act. Baseline: 87.5% for 2004; Annual targets: 91% for 2005 and 2% increase each year afterward. Annual Measure#3 is Percent of grantees implementing one or more evidence-based programs or practices. Baseline: 26% of the Juvenile Justice Programs' grantees reported implementing one or more evidence-based programs or practices for 2005; Target: 10% increase per year.

Evidence: The targets identified above were established using the following resources: First, an analysis of performance measure data collected from grantees that provides a baseline of performance on these measures. (The results of this analysis are summarized in Section 4 of the PART.) Second, an extensive review of research literature and resources on delinquency prevention and intervention programs that have demonstrated effectiveness (through rigorous evaluation methods) in preventing or reducing juvenile offending and associated risk factors. OJJDP relied on the following resources to establish ambitious, yet appropriate, annual performance targets: Updated PART Analysis Memo, dated 4/11/06 (submitted in hardcopy); OJJDP's Model Programs Guide (MPG) (http://www.dsgonline.com/mpg2.5/mpg_index.htm), designed to assist communities in implementing evidence-based prevention and intervention programs for youth and families. It includes summaries of program evaluations, and includes information about rates of recidivism and behavior improvement; Center for the Study and Prevention of Violence's Blueprints for Violence Prevention (http://www.colorado.edu/cspv/blueprints/). The Blueprints for Violence Prevention project provides detailed information about Model and Promising programs, including rates of recidivism and levels of reduction or enhancement in targeted behaviors, such as substance use and academic improvements. The Helping America's Youth (HAY) Program Tool (http://www.helpingamericasyouth.gov/programtool.cfm), which contains information about risk factors, protective factors, and programs that have been evaluated and found to work. Programs must demonstrate results in accordance with widely accepted scientific criteria for program effectiveness. Programs effectiveness ratings range from 1 to 3 (1 being the highest rating), and depend upon four factors: the conceptual framework of the program, program fidelity, strength of the evaluation design, and the empirical evidence demonstrating the prevention or reduction of problem behaviors. Also used: List of key references to research utilized in establishing targets (Research_References_for_AnnualTarget1.doc). OJJDP's authorizing legislation is the Juvenile Justice and Delinquency Prevention Act, 42 U.S.C. § 5601 et. al. and can be accessed at http://www.ojjdp.ncjrs.org/compliance/appendixa.pdf. Legal citations specific to OJJDP's Title II Core requirements are: Deinstitutionalization of Status Offenders (DSO): 42 U.S.C. §5633(a)(11); Sight and Sound Separation: 42 U.S.C. §5633(a)(12); Jail Removal: 42 U.S.C. §5633(a)(13); and Disproportionate Minority Contact (DMC): 42 U.S.C. §5633(a)(22). Since the inception of OJJDP, the agency's goal has been full State and Territorial compliance with these four requirements. The long-term target of 100% is thereby statutorily mandated. Regulations, Standards for Compliance, and other supporting materials related to the Core Requirements are available at the Compliance Monitoring website at http://www.ojjdp.ncjrs.org/compliance/index.html. The following two reports were also used to establish targets for Annual measure #2: FY 2005 Determination of State Compliance with the Juvenile Justice and Delinquency Prevention Act of 2002; and FY 2004 Determination of State Compliance with the Juvenile Justice and Delinquency Prevention Act of 1974.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All grantees are required to commit to the Juvenile Justice Programs' performance goals and report data in support of the programs' mandatory, universal performance measures. All grantees who are implementing direct service programs, regardless of whether they are recipients of Formula Grant, Title V, or Part E funding, are required to report data in support of the following mandatory measures: ?? Percent of program youth who offend or reoffend. ?? Percent of program youth exhibiting a desired change in the targeted behavior ?? Percent of programs implementing one or more evidence-based programs or practices Evidence that illustrates grantee commitment to these performance goals is in all Juvenile Justice Program grant solicitations, which specify the required performance measures and the data that grantees will be expected to report. Additional evidence is provided through a Special Condition included in the documentation signed by grantees which reads as follows: "The recipient agrees to report data on the grantee's OJJDP-approved performance measures as part of the semi-annual categorical progress report. This data will be submitted on line at OJJDP's Performance Measures website (http://ojjdp.ncjrs.gov/grantees/pm/index.html) by July 31 and January 31 each year for the duration of the award."

Evidence: All OJJDP funding solicitations require that applicants (and grantees/subgrantees) identify performance measures in their applications to OJJDP and report on these measures in their periodic progress reports to the agency. Each solicitation includes guidelines and requirements for applicants regarding performance measures reporting to OJJDP. By 2006, OJJDP's performance measures system had evolved so that all agency funding solicitations clearly identified the required mandatory performance measures for each program category. A list of the 2006 solicitations is below (those not active are archived at the indicated location): Field-Initiated Demonstration Program, 5/1/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=59; FY 2006 Congressional Earmark Program, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=49; Training and Technical Assistance Program for Mentoring Initiative for System Involved Youth, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=54; Enforcing the Underage Drinking Laws Discretionary Program: Initiative To Reduce Underage Drinking, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=55; Mentoring Initiative for System Involved Youth, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=56; Evaluation of Mentoring Initiative for System Involved Youth, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=57; Title V Incentive Grants for Local Delinquency Prevention Programs, 4/7/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=58; FY 2006 Tribal Juvenile Accountability Discretionary Grant Program, 4/4/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=53; Formula Grants Program, 3/30/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=41; FY 2006 Tribal Youth Program, 3/15/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=44; Enforcing the Underage Drinking Laws Block Grant Program, 1/31/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=40. To provide additional assistance to funding applicants and grantees, OJJDP also created a number of online resources related to performance measures. These resources are available through OJJDP's Performance Measures Website http://www.ojjdp.ncjrs.gov/grantees/pm/index.html.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Juvenile Justice Programs would benefit from full-scope evaluations to evaluate their effectiveness and continued relevance. OJJDP has not undertaken such reviews, partly because the diverse nature of the activities funded makes such evaluations difficult and expensive to undertake. OJJDP has, however: ?? Sponsored a variety of research on issues in delinquency prevention; ?? Contracted through the National Institute of Justice for evaluability assessments of congressional earmark programs; ?? Sponsored the Blueprints for Violence Prevention initiative at the University of Colorado-Boulder in its efforts to rate juvenile justice programs for evidence-based effectiveness and their ability to be replicated; ?? Compiled the Model Programs Guide, an online database of juvenile justice programs commonly funded by states and grantees under the Juvenile Justice Programs that are rated for their evidence-based effectiveness; ?? Supported launch of the Helping America's Youth (HAY) website, which also provides an online database of juvenile assistance programs rated for their evidence-based effectiveness; and ?? Set targets for grantees to achieve in implementing evidence-based programming. Together, these steps comprise a body of evaluation-related activity that provides good evidence of effectiveness and relevance of many of the programs funded by grantees under the Juvenile Justice Programs. The Model Programs Guide is offered to grantees as tool for improving the extent of evidence-based programming, which OJJDP currently estimates is only 26 percent (percent of grantees implementing effective programming). As do a number of other agency efforts to rate programs for known effectiveness, this online guide rates programs as "Exemplary," "Effective," and "Promising" based on the nature of the evaluation conducted (e.g., randomized controlled trial, quasi-experiment, etc.) and other factors. (The HAY tool uses a similar methodology.) Unfortunately, the various rating schemes employed by OJJDP and other agencies are not well synchronized and may lead to confusion about what programs are truly known to be effective. The rating scheme employed by the Model Programs Guide needs improvement to ensure that programs rated "Effective" have had a positive impact as measured by a recent, high quality, quasi-experimental evaluation. Further, there must be some evidence of the capability for replication of the programs to other sites. OMB plans to work with OJJDP and other agencies to better synchronize these evidence-based program rating schemes.

Evidence: Some current OJJDP evaluation projects that are implementing experimental or quasi-experimental research designs include (see Section 4.5 for more information): Evaluation of Enforcing Underage Drinking Laws (Community Trial Initiative); Evaluation of OJJDP's Truancy Reduction Demonstration Program; and Promising Programs for Substance Abuse Prevention Replication and Evaluation Initiative. The following recent solicitations issued by OJJDP (and NIJ, through an IAA with OJJDP) outline the research and evaluation standards supported by OJJDP: Evaluation of Mentoring Initiative for System Involved Youth (2006) (http://ojjdp.ncjrs.gov/grants/solicitations/06mentoringeval.pdf); Outcome Evaluations of Violence Prevention Programs (http://www.ncjrs.gov/pdffiles1/nij/sl000732.pdf). OJJDP also plans to release a solicitation for Field-Initiated Research and Evaluation projects shortly. Aside from directly promoting good research practices through its funding solicitations, OJJDP also works to educate the field regarding strong evaluation design and methodology. Resources such as the Blueprints initiative, OJJDP's Model Programs Guide, and the Helping America's Youth website provide guidance to those designing and carrying out strong evaluation designs. More information can be found at: OJJDP's Model Programs Guide (http://www.dsgonline.com/mpg2.5/mpg_index.htm); Blueprints for Violence Prevention (http://www.colorado.edu/cspv/blueprints/index.html); and Helping America's Youth Tool (http://helpingamericasyouth.gov/programtool.cfm). In addition, through the support of the Juvenile Justice Evaluation Center (JJEC) Online (http://www.jrsa.org/jjec/), OJJDP provides juvenile justice practitioners, policymakers, and state agency administrators with tools to assist in the assessment and evaluation of programs and initiatives. JJEC Online also produced a report in 2005 entitled, "Status of the States Regarding Evaluation," summarizing the status of juvenile justice evaluation efforts in the states and territories as of 2005, including information on current evaluation-related policies and practices and changes that occurred since 1999. The report indicates that during that period, a total of 16 states demonstrated positive changes in evaluation policies, practices, evaluation change or evaluation focus. Only three states showed little to no positive change during that period. NIJ (with funds transferred from OJJDP) issued 2 solicitations to the field for the evaluation of OJJDP congressional earmark programs. Copies are located online at http://www.ncjrs.gov/pdffiles1/nij/sl000620.pdf and http://www.ncjrs.gov/pdffiles1/nij/sl000694.pdf. NIJ made 8 awards to research institutions to carry out evaluations of the following earmark programs: I-Safe (Caliber Associates); Teens, Crime and Community (University of Missouri-St. Louis); Youth Crime Watch (Abt Associates); Boys and Girls Town (Developmental Services Group, Inc.); LA's Best Afterschool Enrichment Program (University of California, Los Angeles); Evaluation of the Chicago Project for Violence Prevention (Northwestern University); Youth and Families with Promise (Justice Research and Statistics Association); Evaluation of LIFESKILLS and Early Intervention Prostitution Program (Developmental Services Group, Inc.).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Currently, the Juvenile Justice Programs' internal DOJ budget requests are linked to only one of the identified long term performance goals in Section 2.1??the level of state and territorial compliance with the four Core Requirements of the JJDP Act. This measure is reported as an annual performance measure in the Office of Justice Programs' congressional budget submission for 2006. Budget requests to OMB and Congress, however, typically have not included information about how the resources requested will assist OJJDP in achieving the long-term or annual performance goals. While the budget submissions are generally well justified, there is no explicit linkage given in the OMB or the congressional budget submissions to the achievement of performance targets. In fact, performance targets for major juvenile justice programs have not been included in recent OMB or congressional budget submissions. Making these linkages explicit in budget submissions is an area identified for improvement. OJJDP intends that future budget requests to DOJ, OMB and Congress will include explicit information regarding how the funds will be used to carry out activities that support the Juvenile Justice Programs' long term and annual performance goals, as described in this PART, and that these requests will also include updates regarding progress in achieving long term and annual goals.

Evidence: U.S. Department of Justice, FY 2007 Performance Budget, Office of Justice Programs, January 2006 (Congressional Budget Submission).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: OJJDP intends that future budget requests to DOJ, OMB and Congress for the Juvenile Justice Programs will include explicit information regarding how the funds will be used to carry out activities that support OJJDP's long term and annual performance goals, as described in this PART, and that these requests will also include updates regarding progress in achieving long term and annual goals. The agency has worked to streamline its performance measures activities, ensuring that all grants and program functions are in concert with the agency's annual and long term performance goals.

Evidence: Discussions with OMB and DOJ staff have confirmed that future budget requests will include all key performance measures for the Juvenile Justice Programs, and a better linkage to the resources requested.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Performance data are being collected from all Juvenile Justice Programs grantees on an annual or semi-annual basis. OJJDP grant managers also conduct compliance system audits to Formula Grant state grantees every five years to ensure that each state has an adequate compliance monitoring system and to verify data accuracy. They conduct program monitoring visits to the state grantees annually and other grantees as needed for compliance with the approved program plan and budget. In addition, OJJDP maintains frequent telephone and e-mail contacts with its grantees, convenes regularly scheduled conference calls (bi-monthly conference calls for state juvenile justice specialists, state compliance monitors, and Disproportionate Minority Contact coordinators), and conducts conference calls with grantees to monitor performance and other issues. Through such frequent contact with and assistance to grantees, OJJDP is able to make mid-year adjustments or take appropriate actions to improve grantee program performance.

Evidence: A list of the 2006 Juvenile Justice Programs solicitations, which incorporate performance reporting requirements for grantees, is below (those not active are archived at the indicated location): Field-Initiated Demonstration Program, 5/1/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=59; FY 2006 Congressional Earmark Program, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=49; Training and Technical Assistance Program for Mentoring Initiative for System Involved Youth, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=54; Enforcing the Underage Drinking Laws Discretionary Program: Initiative To Reduce Underage Drinking, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=55; Mentoring Initiative for System Involved Youth, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=56; Evaluation of Mentoring Initiative for System Involved Youth, 4/17/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=57; Title V Incentive Grants for Local Delinquency Prevention Programs, 4/7/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=58; FY 2006 Tribal Juvenile Accountability Discretionary Grant Program, 4/4/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=53; Formula Grants Program, 3/30/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=41; FY 2006 Tribal Youth Program, 3/15/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=44; Enforcing the Underage Drinking Laws Block Grant Program, 1/31/2006, http://ojjdp.ncjrs.gov/funding/FundingDetail.asp?fi=40. See also: OJJDP's Performance Measurement Website at http://ojjdp.ncjrs.org/grantees/pm/index.html (OJJDP's Performance Measurement Website leads visitors to program-specific performance measures and Data Collection Technical Assistance Tool (DCTAT)); OJJDP's Title II and Title V Performance Measurement System Website at http://dsgonline.com/performance_measures.htm; screen shot of the OJJDP Data Collection Technical Assistance Tool (DCTAT) at http://www.jabg-dctat.org/ojjdpbeta/; sample program monitoring visit reports; and sample grantee conference call minutes.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: When ensuring that grantees comply with their award's administrative and programmatic requirements and the fulfillment of program goals, OJJDP implements the requirements of the OJP Office of the Comptroller's Financial Guide 2005, which incorporates provisions of OMB circulars and government-wide common rules applicable to grants and cooperative agreements. To hold grantees accountable for cost, schedule and performance results, OJJDP attaches a number of special conditions that grantees must satisfy. Special conditions include the requirement that grantees submit quarterly financial reports and 6-month progress reports (or annual performance reports). These reports provide information about program implementation status and performance results at regular intervals. For grantees with delinquent reports, OJJDP adds a special condition to the subsequent year award to restrict drawdown of funds. Moreover, when there are questions regarding costs, such as the appropriateness of grant expenditures or the fulfillment of a match requirement, OJJDP requests that the Office of the Comptroller (OC) conduct audits of the involved grantees and works with OC and the grantees to reach satisfactory resolutions. A unique example of holding grantees accountable is the enforcement of the four core requirements of the JJDP Act of 2002 to ensure that juvenile justice systems of the Formula Grants Program's state grantees treat juvenile offenders appropriately and fairly. (The four core requirements are: (1) deinstitutionalization of status offenders and nonoffenders; (2) separation of juvenile delinquents from adult offenders; (3) removal of juveniles from adult jail and lockup facilities; and (4) reduction of disproportionate minority contact.) A state's Formula Grant allocation is reduced by 20 percent for each of the core requirements for which the state was found out of compliance in the prior year. OJJDP holds its grant/program managers accountable through performance work plans and supervisor review for quality and professional standards of practice. Standards address timeliness for posting, processing, and awarding grant information and grant funds, and if applicable, improvements or innovations in program administration. Individual performance plans for grant/program managers include these standards and are linked to the organizational goals set forth in OJJDP's Strategic Plan. In addition, on June 30, 2004, the Attorney General transmitted decisions on the Senior Executive Service (SES) Performance-based Pay System to OPM and OMB with the Human Capital report, including a generic work plan for all departmental SES members with a performance "contract" that must relate to the Department's, the President's or the AG's defined goals. On December 10, 2004, DOJ obtained approval for the SES and General Service/Prevailing Rate performance orders. It completed the application package for OPM/OMB certification of the DOJ SES Performance Management and Compensation Plan. DOJ components have implemented five-level performance plans for all SES. New SES and manager work plans include cascading tasks/assignments that are linked to the DOJ Strategic Plan and the President's Management Agenda. By December 30, 2004, all DOJ components certified to the Attorney General that all SES and direct report performance work plans were in place. By the end of December 2004, the Department had performance appraisals for most of the work force that connect to agency mission, goals and outcomes in DOJ's strategic Plan; hold employees accountable for results appropriate to their level of responsibility; effectively differentiate between various levels of performance; and provide consequences based on performance.

Evidence: For requirements of the OJP Office of the Comptroller's Financial Guide 2005, see: http://www.ojp.usdoj.gov/FinGuide/. For evidence of holding grantees accountable for delinquent financial/progress reports, see selected pages from Title II Formula and Title V Application Review Checklists and a sample list of special conditions in a FY 2006 award with a drawdown restriction resulting from delinquent financial/progress reports (submitted in hardcopy). For evidence of OJJDP's determination of Formula Grant state grantees' compliance with the four core requirements of the JJDP Act of 2002, see the FY 2004 and FY 2005 Summaries of States' Compliance. The level of state compliance with the core requirements is reported in OJJDP Annual Reports and as an annual performance measure in OJP's Annual Performance Budget and DOJ's Annual Performance Accountability Report. Based on the FY 2004 compliance determination, a total of three states' FY 2005 Title II Formula Grants allocations were reduced by 20%, one state by 40%, and one state by 60%. Based on the FY 2005 compliance determination, a total of two states expect to have their FY 2006 Title II Formula Grants allocations reduced by 20%, two states by 40%, and one state by 60%. See sample OC's financial audit report. All managers are subject to the new DOJ Performance Management System, Attorney General's Memorandum dated August 10, 2004??"Performance Management Plans for SES Members and Managers" at www.usdoj.gov/jmd/ps/memagpma-ses1.htm; OJP Financial Guide 2005 (www.ojp.usdoj.gov/FinGuide/); OJJDP Grants Manager's Manual, Chapter 8: Performance Monitoring; OJJDP Special Conditions for Grants; and Examples of FY 2006 OJJDP Grant Solicitations (submitted by OJJDP).

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: More than 95% of the Juvenile Justice Programs' Formula, Title V, and congressional earmark program funds were obligated and committed within the fiscal year of the appropriation in 2004. In 2005, 100% of funds were obligated within the year of appropriation. The Office of Justice Program's Web-based Grants Management System (GMS), first instituted in FY 2002, was designed to make the application process easier and application review, award-making, and acceptance more efficient. As a result, OJJDP is better able to obligate all funds in a timely manner. The length of the obligation process varies, with block/formula and earmark grants typically taking less time to process than competitive awards (e.g., Enforcing Underage Drinking Law Program and Tribal Youth Program) due to the latter's requirement of peer review. However, Formula Grant applications also require a relatively long time to process because of the need for extensive reviews of states' juvenile justice and delinquency prevention plans and the OJJDP's Administrator's signed compliance determination letters to individual states before the awards are made. One key task in reviewing all applications is to make change requests of the applicants, as needed, to assure that the proposed activities and budgets meet all guidelines and purposes of the respective grants programs. Applicants must respond to these requests with satisfactory modifications of their applications before further award processing. In FY 2004, OJJDP made 389 awards in an average of 110 days; and 465 awards in an average of 78 days in FY 2005, achieving a 32-day reduction (-29%) in the award processing time during one year. After the awards are made, all requests for programmatic and/or administrative budget changes must be submitted in a timely manner by the recipient. According to Chapter 5, "Adjustments to Awards," of the OJP Financial Guide 2005, recipients must request and receive approval from OJJDP for changes in approved budget categories in excess of 10 percent of the total award amount. All requests for changes to the approved award are carefully reviewed by OJJDP grant managers and supervisors for consistency with the Financial Guide and their contribution to project goals and objectives. Grant recipients must receive OJJDP approval through a grant adjustment notice before they can implement the desired changes. This is an important mechanism to ensure that Federal funds are spent for the intended purposes. OJP's Office of the Comptroller (OC), Contracts Office, and OJJDP staff monitor draw down and expenditure of awarded funds through financial status reports and programmatic progress reports to determine if recipients obligate Federal funds on a timely basis and are in compliance with Federal cash management regulations and OMB Circular A-133, Audits of States, Local Governments, and Non-Profit Organizations, as appropriate. Using telephone, e-mail, onsite program monitoring visits, and other technical assistance, OJJDP staff helps grantees address issues involving fund expenditures. Office of the Comptroller onsite reviews determine whether grantees are properly accounting for the receipt and expenditure of Federal funds, and that expenditures are in compliance with Federal requirements and award special conditions. OJJDP management assesses requests for no-cost extensions, which would lead to a lengthening of the period of grant award.

Evidence: See OJJDP FY 2004 and FY 2005 Appropriation Analyses (submitted in hardcopy); OJJDP FY 2004 and FY 2005 Timeliness in Obligating Federal Funds Analysis (submitted in hardcopy); A sample Grant Adjustment Notice to describe and approve budget modification (submitted in hardcopy); Chapter 5, Adjustments to Awards, in the OJP Financial Guide 2005 (http://www.ojp.usdoj.gov/FinGuide/part3chap5.htm).

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Office of Justice Programs has taken a number of steps designed to cut costs and increase efficiency. In concert with OJJDP leadership and staff, OJP has been undertaking competitive sourcing activities intended to deliver government services that are market-based, results oriented, and citizen-centered in accordance with OMB Circular A-76 and the President's Management Agenda. Although OJP pursued the implementation of a Most Efficient Organization (MEO) for grants support services and received House approval, the Senate has disapproved the request. OJP still intends to capture many of the cost savings and efficiencies the MEO process would have enabled, including capturing savings of $5 million associated with a consolidation of the peer review process and implementing personnel savings initiatives. OJP will make what cost and efficiency changes are possible under the current organizational structure. OJP also has instituted a Business Process Improvements (BPI) initiative. The purpose of the BPI initiative is to streamline and simplify business flows throughout OJP. The BPI team has developed and implemented new policies on out-processing, time and attendance, and Grant Adjustment Notice (GAN). The BPI team is in the process of redesigning Solicitation, Application, Peer Review, Redbook, and Closeout processes in OJP. In FY 2006 OJP developed and deployed an automated GANs module in the OJP Grant Management System (GMS). GANs is an initiative that automates, standardizes and consolidates disparate manual paper-based processes across all OJP program offices. Twenty-seven separate GAN processes were consolidated to 13. A WEB interface has been extended to the grantees and Grant Managers allowing them to self generate a variety of GAN changes to their Grants (i.e., name change, point of contact change, period of performance changes). The GANs module provides automatic workflow routing and notification of a GAN activity, provides audit records of each transaction and automates the information update between GMS and the OJP financial management system, and provides mail enabled notification to the grantee and the grant manager as needed during the workflow. Improvements in accuracy also include pre-validation of change requests against current data to ensure the request can be processed before it can be submitted, as well as eliminating the human interface between the GMS and financial systems. Although the GANS module has only been operative since the beginning of April, 2006, preliminary results indicate large savings with GANs processing cycle time for "quick" turn items being reduced from at least two weeks to approximately a day and a half." A number of other IT initiatives also demonstrate the potential for improvements in efficiency, reliability, and accuracy for grants management. These include implementation of Grants.Gov and Grant Monitoring Office of the Comptroller (GMOC) module that will provide an audit capability for OC to monitor grantee programmatic progress reports and programmatic grant monitoring efforts by grant managers.

Evidence: The OJP MEO proposal was submitted to House/Senate Appropriations committees as a reprogramming request. Under Sec. 605 of the Title VI General Provisions of the 2006 Departments of State, Justice, and Commerce, and related agencies appropriation (Public Law 109-108), Justice must submit a reprogramming notice prior to implementing reorganization. Public Law 109-108 can be accessed here: http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=109_cong_bills&docid=f:h2862enr.txt.pdf. For additional information on OJP's Business Process Improvement (BPI), see BPI Steering Committee Charter; Grant Adjustments: Business Process Improvement Recommendations (October 7, 2004); Grant Closeouts: BPI Recommendations (Final Version, September 15, 2005); Functional Requirements Document: Grant Adjustments (Version 1.2, March 15, 2006). (Material submitted in hardcopy). Regarding Grants.Gov. implementation, in FY 2005, OJP initiated a grant application using the Grants.Gov system. It was piloted with one office and 4 program opportunities. In FY 2006, OJP extended this capability to all program offices that conduct competitive discretionary grant making. Applications will be solicited for 96 programs with an anticipated 5200 applications. OJP participation in the Grants.Gov initiative provides the public with one-stop shopping for grant opportunities across the entire base of 27 grant making agencies in the Federal Government using a standard form/format making the process more efficient and easier for applicants. In addition, Grant Monitoring Office of the Comptroller (GMOC) module is under design for GMS. It will provide an audit capability for the Office of the Comptroller (OC) to monitor grantee Programmatic Progress Reports and Programmatic grant monitoring efforts by the grant managers beyond what OC currently has with the SF-269 Financial Progress Reporting system. This automates a paper-based process and improves the reliability and accuracy of the OC monitoring efforts. This enhancement is currently in design and is expected to be deployed in late FY 2006. GMOC is a precursor capability required to implement the GMS Grants Closeout Module. The Office of Justice Programs (OJP) Grants Manager's Manual (GMM) serves as the basic reference source for the administration of OJP grants (submitted in hardcopy). See the Grant Manager's Manual for a description of the GANs process.

Does the program collaborate and coordinate effectively with related programs?

Explanation: OJJDP effectively collaborates and coordinates with all levels of government (federal, state, local) and with a variety of private programs, cultivating partnerships that further the mission of the Juvenile Justice Programs. OJJDP regularly exchanges information, initiates joint efforts that leverage federal funds, monitors the progress of joint efforts, and adjusts objectives, as needed. OJJDP participates and works through the Coordinating Council on Juvenile Justice and Delinquency Prevention, the Federal Advisory Committee on Juvenile Justice (FACJJ), various inter-agency task forces and working groups, inter-agency agreements (IAAs), joint training conferences, joint grant programs, and joint publications. At the federal level, OJJDP assumes a vital role in directing the Coordinating Council on Juvenile Justice and Delinquency Prevention by working together with federal partners to draw on their different areas of expertise and resources. The Coordinating Council has the responsibility to "coordinate all Federal juvenile delinquency programs (in cooperation with State and local juvenile justice programs), all Federal programs and activities that detain or care for unaccompanied juveniles, and all Federal programs relating to missing and exploited children." Recently, the Coordinating Council established or advanced a number of interagency initiatives designed to prevent delinquency and improve services to juveniles and missing and exploited youth. These include the Labor-led Youth Vision, the National Conference of the Coordinating Council, and the development of an Action Plan for Juvenile Justice. The Federal Advisory Council on Juvenile Justice (FACJJ) also is a key means of coordination to address the problem of delinquency. The FACJJ comprises representatives from 56 jurisdictions whose purpose is to advise the executive and legislative branches on juvenile justice. The FACJJ enhances federal/state collaboration by developing two reports annually - one to the President and Congress, and the other to the OJJDP Administrator - both of which highlight key policy and program concerns and recommend appropriate action. OJJDP also has been working with several agencies of the Coordinating Council on Juvenile Justice and Delinquency Prevention and other partners to develop a Geographic Information System (GIS) with the capacity to display crime problems, community and governmental assets, and socioeconomic factors that can prevent and control crime and delinquency. The use of this system, which draws on census data and program and resource information from several federal agencies, will help agency decision-makers to identify gaps and overlaps in their own and other federal programs, as well as other governmental and private sector resources.

Evidence: Additional information about the Coordinating Council on Juvenile Justice and Delinquency Prevention can be found at http://www.juvenilecouncil.gov. Additional information about the Federal Advisory Committee on Juvenile Justice can be found at http://ojjdp.ncjrs.org/facjj/index.html. The January 2006 National Conference of the Coordinating Council was developed by an interagency planning team with OJJDP support. Representatives collectively developed an agenda that reflected each member agency's response to the problem of youth failure. The conference agenda represented a follow up to the final report of the White House Task Force on Disadvantaged Youth. President Bush created the Task Force in December 2002 to develop a comprehensive federal response to the problems of youth failure, with a focus on enhanced agency accountability and effectiveness. (See http://www.ncjrs.gov/html/ojjdp/news_at_glance/212789/topstory.html). Through the Coordinating Council, OJJDP funds provided to the Department of Labor (DOL) also have advanced a joint effort-- "Creating New Opportunities for Youth: A Shared Vision"--among the Departments of Labor, Education, Human Services, and Justice to organize state-based teams of administrators and staff of local and state public and private youth-serving agencies and programs, federal agencies, and youth. This partnership seeks to ensure that states and localities are able to offer youth who have become disconnected from mainstream institutions and systems opportunities to successfully transition to adult roles and responsibilities. Also see: http://www.doleta.gov/ryf/. Similarly, in 2005, OJJDP provided the Departments of Housing and Urban Development, DOL, HHS, and the Corporation for National and Community Service (CNCS) with $100,000 each. CNCS is using the funds to expand an informal intra-agency working group on mentoring into a formal interagency council that includes private sector, faith and community-based representation. This mentoring council, unlike its predecessor working group, will be staffed and is funded to leverage and coordinate federal mentoring resources with the goal of increasing the number of mentors available to youth in accordance with the President's goals. HHS is using the funds to better align with OJJDP regarding the planning, development and implementation of policies, programs and practice as these apply to clinical treatment and recovery support for justice-involved youth. Efforts are aimed at improving treatment services for justice-involved youth and are being coordinated with federal, state and local stakeholders, including the National Governors' Association and the National Association of County Officials. OJJDP has nearly two dozen current inter-agency agreements (IAAs) with other federal agencies. See Listing of FY 2005 Inter-Agency Reimbursable Agreements with Brief Description of IAAs (submitted in hardcopy). OJJDP participates in several inter-agency working groups, task forces and committees, including: The Federal Inter Agency Working Group on Child Abuse and Neglect, the Fetal Alcohol Spectrum Disorder Juvenile Justice Working Group, the Federal/National Partnership for Children's Mental Health, and the inter agency working groups of the Council that address gangs, juvenile information systems, and state and local planning.

Does the program use strong financial management practices?

Explanation: OJJDP is a component of the Office of Justice Programs (OJP). According to the Department of Justice Office of Inspector General's March 2006 report on OJP's 2005 Financial Statements, "In its FY 2005 Report on Internal Control over Financial Reporting, KPMG identified four reportable conditions, three of which were considered to be material weaknesses. All four reportable conditions are repeated from the restated FY 2004 report." Reported issues included: ?? Data validity, completeness, and calculation errors in grant data underlying the advance and payable amounts; ?? OJP still needs to improve its financial statement preparation process, specifically its posting of final adjusting journal entries; ?? The need for significant improvements to the de-obligation/closeout process for grant and non-grant undelivered orders; and ?? OJP's financial management systems were not in compliance with the Federal Financial Management Improvement Act of 1996 with regard to federal financial management systems requirements and applicable federal accounting standards. While OJP reports progress in resolving auditor concerns, the report on the 2006 financial statements will confirm whether sufficient progress has been made to close out these concerns. OJP has made progress in other areas. For example, in October 2005, OJP received an unqualified opinion for FYs 2005, 2004, and was able to reverse the disclaimer opinion previously given for FY 2004.

Evidence: See the Office of Justice Programs Annual Financial Statement, Fiscal Year 2005, Audit Report 06-17, March 2006, Office of the Inspector General. (http://www.usdoj.gov/oig/reports/OJP/a0617.htm). Also see the DOJ 2005 Performance and Accountability Report and the Independent Accountants' Report on Financial Statements (see http://www.usdoj.gov/ag/annualreports/pr2005/TableofContents.htm).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: OJJDP managers and staff continuously monitor and assess controls using various sources, including knowledge gained from the daily operation of agency programs and systems, Inspector General (IG) and Government Accountability Office (GAO) reports, program evaluations, audits and monitoring reports of funding recipients and financial activities, financial reports generated by the Office of the Comptroller (OC) and Office of Budget and Management Services, and annual performance plans. OJJDP reports that its managers and employees directly identify deficiencies in management controls from the sources of information described above. Employees and managers report deficiencies to the next supervisory level, allowing the chain of command to determine the relative importance of and appropriate response to each deficiency. OJP embraces a number of wide-ranging business process improvements (BPI). The purpose of the BPI initiative is to streamline and simplify business flows throughout OJP. The BPI team has developed and implemented new policies on out processing, time and attendance, and Grant Adjustment Notice (GAN). The BPI team is in the process of redesigning Solicitation, Application, Peer Review, Redbook, and Closeout processes in OJP.

Evidence: For additional information on OJP's Business Process Improvement (BPI), see BPI Steering Committee Charter; Grant Adjustments: Business Process Improvement Recommendations (October 7, 2004); Grant Closeouts: BPI Recommendations (Final Version, September 15, 2005); Functional Requirements Document: Grant Adjustments (Version 1.2, March 15, 2006). The Office of Justice Programs (OJP) Grants Manager's Manual (GMM) serves as the basic reference source for the administration of OJP grants. (Materials submitted in hardcopy).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: OJJDP gains knowledge of grantees' activities through the following oversight practices: ?? requiring submission of regular performance measurement reports; ?? ensuring frequent staff monitoring and assistance contact with its grantees; and ?? conducting compliance system audits and program monitoring visits. To improve the quality and usefulness of progress/performance reports required of all grantees, OJJDP developed a performance measurement system in 2004 for the Formula Grants (Title II) and Title V Incentive Grants for Delinquency Prevention programs and instituted annual performance reports in FY 2005 with the use of the Data Collection Technical Assistance Tool (DCTAT). Building on this experience, OJJDP's FY 2006 Part E (earmarks and discretionary awards) solicitations include a requirement for all recipients to collect data and report on mandatory performance measures in their 6-month progress reports. OJJDP grant managers conduct compliance system audits of Formula Grant state grantees every five years to ensure that each state has an adequate compliance monitoring system and to verify data accuracy. They conduct program monitoring visits to the state grantees annually and other grantees periodically for compliance with approved program plans and budgets. In exit interviews??reinforced in follow-up letters??OJJDP grant managers discuss with grantees their findings of the programs' strengths and weaknesses, related recommendations, and requirements for corrective actions. In addition, OJJDP maintains frequent telephone and e-mail contacts with its grantees and convenes regular conference calls (bi-monthly conference calls for state juvenile justice specialists, state compliance monitors, and Disproportionate Minority Contact coordinators). OJJDP also sponsors annual regional trainings on topics designed to support the grantees in the implementation of their programs. Through frequent contact with and assistance to grantees, OJJDP gains sufficient knowledge of grantee activity to make mid-year adjustments or take appropriate actions to improve grantee program performance.

Evidence: OJJDP's Performance Measurement Website at http://ojjdp.ncjrs.org/grantees/pm/index.html. OJJDP's Performance Measurement Website leads visitors to program-specific performance measures and Data Collection Technical Assistance Tool (DCTAT). OJJDP's Title II and Title V Performance Measurement System Website at http://dsgonline.com/performance_measures.htm; Screen shot of the OJJDP Data Collection Technical Assistance Tool (DCTAT) at http://www.jabg-dctat.org/ojjdpbeta/; Sample program monitoring visit reports submitted in hardcopy. Sample grantee conference call minutes submitted in hardcopy.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Grantee performance data are not yet made available to the public for the Juvenile Justice Programs, although OJJDP has plans to do so. To improve the quality and usefulness of progress/performance reports required of all grantees, OJJDP developed a performance measurement system in 2004 for the Formula Grants (Title II) and Title V Incentive Grants for Delinquency Prevention programs and instituted annual performance reports in FY 2005 with the use of the Data Collection Technical Assistance Tool (DCTAT). Building upon the above experience, all OJJDP's FY 2006 solicitations require award recipients to report performance measurement data. For formula grants programs, local subgrantees can enter performance measurement data themselves. Most state grantees, however, choose to collect data from their subgrantees and enter this data under each subgrant within a specific formula grants program on the Web-based DCTAT. The DCTAT then aggregates the data at the state level at the time of submission. State grantees can analyze subgrants within a specific program area or across program areas to aid management decisions and actions. Once state grantees submit their performance reports with state-level aggregated data to OJJDP, the DCTAT aggregates the data nationally and generates reports that contrast programs by identifying programs with the most positive outcomes, those needing technical assistance, and other information beneficial to overall program improvement. OJJDP's earmark and discretionary awards are required to collect and report such data every 6 months. Nationally aggregated performance data and reports are subsequently presented to OJJDP staff and grantees (at regional conferences) to promote grantees' use of performance measurement data to better manage their programs. OJJDP plans to begin making grantee performance data available to the public with the 2005 Title V Report to Congress. OJJDP also plans to, for the first time, present nationally aggregated performance data and reports on the II Formula Grants, and Title V Incentive Grants for Delinquency Prevention Grants Programs at the fall 2006 OJJDP State Relations and Assistance Division Regional Training Conferences. Data and reports presented at the Regional Conferences will be subsequently posted at OJJDP's Website, making them accessible to the public.

Evidence: Discussions with OJJDP staff. Also, see selected PowerPoint slides for March 2006 Performance Measurement Training for OJJDP staff; selected pages from the draft 2005 Title V Report to Congress. (Material submitted in hardcopy.)

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The vast majority of appropriated funds are designated as formula grants, block grants, or noncompetitive programs earmarked for award to specific organizations. In Fiscal Years 2006, 2005, and 2004, the proportion of appropriated funds available for competitive award was only 14%, 13%, and 14%, respectively. Of the discretionary funds available for competitive award in FY 2006, most (83%) were awarded competitively to new projects, utilizing a standardized process for peer review, or to continuation programs that had originally been awarded competitively with a prescribed project period. The percentage of funds competitively awarded could be improved. (However, for FY 2004 and 2005, the percentages of competitive discretionary awards were lower??76% and 72%, respectively.) All OJJDP competitively awarded programs are announced through multiple web sites, including http://www.grants.gov and the funding pages of the OJJDP and OJP Web sites. OJJDP alerts the juvenile justice community about new funding opportunities through its "News @ a Glance" bimonthly newsletter with more than 16,000 subscribers, and its JUVJUST listserv of more than 18,000 members.

Evidence: OJJDP has submitted data in hardcopy, which summarizes the appropriations for FY 2006, 2005, and 2004, and categorizes awards by fund type and competitive status. The OJJDP Peer Review Guideline http://ojjdp.ncjrs.org/funding/peerreview.html establishes procedures for organizing and conducting peer reviews of applications submitted for discretionary funding to OJJDP. The peer review process is managed by the Juvenile Justice Resource Center (JJRC) which maintains a consultant pool of more than 700 experts from a wide range of disciplines to rate and evaluate applications. JJRC maintains complete records of all aspects of the review, from receipt of applications and initial determination of applicant eligibility through to an in depth summary of the peer review meetings and conference calls. Each prospective peer reviewer is required to identify any potential conflicts of interest (for recusal or possible removal from the review panel) and certify that there is none for any application they have been assigned. They are provided with in-depth guidance on how to consider various rating criteria to ensure consistency in application of the evaluation criteria. A listing of all peer review panels convened and related costs between 2004 and 2006 (planned), along with a sample peer review packet for peer reviewers have been submitted in hardcopy.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: OJJDP gains knowledge of grantees' activities through the following oversight practices: ?? requiring submission of regular performance measurement reports; ?? ensuring frequent staff monitoring and assistance contact with its grantees; and ?? conducting compliance system audits and program monitoring visits. To improve the quality and usefulness of progress/performance reports required of all grantees, OJJDP developed a performance measurement system in 2004 for the Formula Grants (Title II) and Title V Incentive Grants for Delinquency Prevention programs and instituted annual performance reports in FY 2005 with the use of the Data Collection Technical Assistance Tool (DCTAT). Building on this experience, OJJDP's FY 2006 Part E (earmarks and discretionary awards) solicitations include a requirement for all recipients to collect data and report on mandatory performance measures in their 6-month progress reports. OJJDP grant managers conduct compliance system audits of Formula Grant state grantees every five years to ensure that each state has an adequate compliance monitoring system and to verify data accuracy. They conduct program monitoring visits to the state grantees annually and other grantees periodically for compliance with approved program plans and budgets. In exit interviews??reinforced in follow-up letters??OJJDP grant managers discuss with grantees their findings of the programs' strengths and weaknesses, related recommendations, and requirements for corrective actions. In addition, OJJDP maintains frequent telephone and e-mail contacts with its grantees and convenes regular conference calls (bi-monthly conference calls for state juvenile justice specialists, state compliance monitors, and Disproportionate Minority Contact coordinators). OJJDP also sponsors annual regional trainings on topics designed to support the grantees in the implementation of their programs. Through frequent contact with and assistance to grantees, OJJDP gains sufficient knowledge of grantee activity to make mid-year adjustments or take appropriate actions to improve grantee program performance.

Evidence: OJJDP's Performance Measurement Website at http://ojjdp.ncjrs.org/grantees/pm/index.html. OJJDP's Performance Measurement Website leads visitors to program-specific performance measures and Data Collection Technical Assistance Tool (DCTAT). OJJDP's Title II and Title V Performance Measurement System Website at http://dsgonline.com/performance_measures.htm; Screen shot of the OJJDP Data Collection Technical Assistance Tool (DCTAT) at http://www.jabg-dctat.org/ojjdpbeta/; Sample program monitoring visit reports submitted in hardcopy. Sample grantee conference call minutes submitted in hardcopy.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: To improve the quality and usefulness of progress/performance reports required of all grantees, OJJDP developed a performance measurement system in 2005 for the Discretionary/Competitive and Earmark Grants. Grantees are required to collect and report such data every 6 months and will begin online data reporting in January 2007. OJJDP's Performance Measures Website provides grantees and the public with guidelines and tools to assist in identifying and reporting on their performance measures. Because the system is so new, OJJDP has not yet begun reporting data systematically to the public from the Discretionary/Competitive programs, but plans to do so in the near future.

Evidence: OJJDP's Performance Measures Website http://www.ojjdp.ncjrs.org/grantees/pm/index.html

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Juvenile Justice Programs have reported mixed performance for their long term performance goals for 2005, the first year for which performance data was available for comparison against targets. The 2005 target for the percent of youth who offend or reoffend was 39%--and OJJDP beat the target with reported performance of 11%. However, the figure is based on partial data and may not be representative. OJJDP set a 2005 target of 91% for the percent of states/territories in compliance with the core requirements of the JJDP Act. While the rate obtained of 89% is an improvement over the 2004 benchmark figure, it still does not meet the target.

Evidence: Long Term Goal #1 see Updated PART Data Analysis memo (dated April 11, 2006). Long Term Goal #2 see: FY 2004 Determination of State Compliance with the JJDP Act of 1974; FY 2005 Determination of State Compliance with the JJDP Act of 2002. (Materials submitted in hardcopy.)

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Juvenile Justice Programs have reported mixed performance for their annual performance goals for 2005, the first year for which performance data was available for comparison against targets for most measures. The 2005 target was 91% for the percent of states/territories in compliance with the core requirements of the JJDP Act. While the rate obtained of 89% is an improvement over the 2004 benchmark figure, it still does not meet the target. The Juvenile Justice Programs did meet the 2005 performance target of 37% for the percent of program youth who exhibit a desired change in the targeted behavior. OJJDP obtained benchmark data for its remaining annual performance measure--the percent of grantees implementing one or more evidence-based programs or practices. However, a comparison of target to actual performance will only become available when 2006 data are collected.

Evidence: Annual Goals #1 and #3: see Updated PART Data Analysis memo (dated April 7, 2006); Annual Goal #2 see: FY 2004 Determination of State Compliance with the JJDP Act of 1974 and FY 2005 Determination of State Compliance with the JJDP Act of 2002. (Materials provided in hardcopy.)

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Juvenile Justice Programs have been able to demonstrate increased efficiency with a drop in the number of days to process grant awards from 110 in 2004 to 78 in 2005, a drop of nearly 30%. Further OJP's Business Process Improvement initiative has already begun to yield increased effeciencies. For example, in FY 2006 OJP developed and deployed an automated Grant Adjustment Notices (GANs) module in the OJP Grant Management System (GMS). GANs is an initiative that automates, standardizes and consolidates disparate manual paper-based processes across all OJP program offices. Twenty-seven separate GANs processes were consolidated to 13. A WEB interface has been extended to the grantees and Grant Managers allowing them to self generate a variety of GANs changes to their Grants (i.e., name change, point of contact change, period of performance changes). The GANs module provides automatic workflow routing and notification of a GAN activity, provides audit records of each transaction and automates the information update between GMS and the OJP financial management system, and provides mail enabled notification to the grantee and the grant manager as needed during the workflow. Improvements in accuracy also include pre-validation of change requests against current data to ensure the request can be processed before it can be submitted, as well as eliminating the human interface between the GMS and financial systems. OJJDP also has begun collecting data for a new performance measure designed to gauge the effeciency and cost-effectiveness of the Juvenile Justice Programs: "Percentage of Funds Allocated to Grantees Implementing Evidence-based Programs or Practices." Twenty percent of OJJDP grant funds were reported allocated to grantees who implemented evidence-based programs or practices in 2005. With aggressive targeting to substantially improve this percentage, the Juvenile Justice Programs should be able to demonstrate dramatically increased cost-effectiveness over time.

Evidence: Grant Processing Time Excel Spreadsheet (OJJDPEfficiencyGrantproc.xls); Updated PART Data Analysis memo (dated April 11, 2006); USDOJ's Solicitation Reference Number 2005R_003. (Materials submitted in hardcopy.) For additional information on OJP's Business Process Improvement (BPI), see BPI Steering Committee Charter; Grant Adjustments: Business Process Improvement Recommendations (October 7, 2004); Grant Closeouts: BPI Recommendations (Final Version, September 15, 2005); Functional Requirements Document: Grant Adjustments (Version 1.2, March 15, 2006). (Material submitted in hardcopy).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Outcome findings for the Juvenile Justice Programs regarding offending rates, and levels of behavior changes among program youth are consistent with (and in some cases better than) findings of many research-based programs. Many of the resources that OJJDP has relied on to establish targets provide information about the performance of other delinquency prevention efforts. These include both prevention and intervention programs, as well as those which are considered "model" in terms of the level of evidence which exists to indicate their effectiveness in preventing and/or reducing delinquency, and other negative outcomes and behaviors among youth. Programs reviewed and ranked as "model" or "promising" by the Center for the Study and Prevention of Violence's Blueprints for Violence Prevention (at University of Colorado, Boulder) reports recidivism rates and behavior changes among participating youth. For example, evidence-based programs, such as Functional Family Therapy and Quantum Opportunities showed decreases in offending between 16% and 4%. There was a reported 66% reduction in aggressive behavior for the Incredible Years Program and a 25% reduction in bullying behavior for the Bullying Prevention Program. These findings and other similar rates among prevention programs can also be found through a review of programs in OJJDP's Model Programs Guide, which was designed to assist communities in implementing evidence-based prevention and intervention programs for youth and families. It includes summaries of program evaluations, and includes information about rates of recidivism and behavior improvement. The Center for Evaluation Research and Methodology's meta-analysis of over 500 controlled studies of the effects of intervention programs for juvenile offenders on recidivism and other outcomes concluded that recidivism rates for youth in programs with the highest level of evidence could anticipate a recidivism rate of 21% (compared to comparable youth not in a program who could anticipate a recidivism rate of 40%). In addition to these programs, OJJDP's rates also compare favorably to rates tracked by some State-level criminal and juvenile justice agencies, most notably: Washington State Institute for Public Policy (WSIPP), created by the Washington State Legislature in 1983 to carry out practical, non-partisan research??at legislative direction??on issues of importance to Washington State. A primary activity of WSIPP has been to investigate the cost-effectiveness of different programs and policies. As part of this work, WSIPP has conducted a series of studies on programs targeting youth, including juvenile justice youth. Some results for state programs include: Functional Family Therapy (recidivism rates of 18% among program youth in the most competent programs); Aggression Replacement Therapy (recidivism rate of 45% among program youth); and Coordination of Services (recidivism rate of 17% among program youth). Results also compare favorably to those of the Florida Department of Juvenile Justice (DJJ), Office of Research and Planning, which conducts annual Outcome Evaluation Reports and Program Accountability Measures (PAM) Reports of its programs, including reported recidivism rates. Overall, the recidivism rate of the youth who completed Florida DJJ prevention programs was 12%. Rates for residential programs were higher than those for non-residential programs (22% compared to 10%). Recidivism rates of the two diversion programs evaluated (Intensive Delinquency Diversion Services and Early Delinquency Intervention Program) showed recidivism rates of 16% and 24%. The recidivism rate for youth completing residential commitment programs was 40% statewide.

Evidence: See the Center for the Study and Prevention of Violence's Blueprints for Violence Prevention (http://www.colorado.edu/cspv/blueprints/) for information on specific program models, and also see the following documents: Blueprints for Violence Prevention, July 2004 (Report, 180 pages) Mihalic, S., Irwin, K., Elliott, D., Fagan, A. and Hansen, D. http://www.ncjrs.gov/pdffiles1/ojjdp/204274.pdf; Successful Program Implementation: Lessons from Blueprints, July 2004 (Bulletin, 12 pages) Mihalic, S., Irwin, K., Fagan, A., Ballard, D., Elliott, D http://www.ncjrs.gov/pdffiles1/ojjdp/204273.pdf; Mihalic, S., Irwin, K., Elliott, D., Fagan, A., & Hansen, D. (2001); Blueprints for Violence Prevention. OJJDP Bulletin NCJ 187079 available at http://www.ncjrs.gov/html/ojjdp/jjbul2001_7_3/contents.html; OJJDP's Model Programs Guide (MPG) (http://www.dsgonline.com/mpg2.5/mpg_index.htm); Center for Evaluation Research and Methodology (CERM) at the Vanderbilt Institute for Public Policy Studies (http://www.vanderbilt.edu/cerm/) for these documents: Howell, J.C., & Lipsey, M.W. (2004). A practical approach to improving juvenile justice system programs. Juvenile and Family Court Journal. Vol. 55 (No. 1), Pgs. 35-48; and Howell, J.C. and Lipsey, M.W. (2004); A Practical Approach to Linking Graduated Sanctions with a Continuum of Effective Programs. Juvenile Sanctions Center Training and Technical Assistance Center Bulletin, Vol. 2, (No. 1). (available at http://www.ncjfcj.org/content/view/515/331/). See Washington State Institute for Public Policy (http://www.wsipp.wa.gov/) for the following documents: Benefits and Costs of Prevention and Early Intervention Programs for Youth 2004 July. Steve Aos, Roxanne Lieb, Jim Mayfield, Marna Miller, Annie Pennucci. #04-07-3901. (http://www.wsipp.wa.gov/pub.asp?docid=04-07-3901); Washington State's Experience with Research-Based Juvenile Justice Programs: Presented at the Los Angles County Juvenile Justice Conference May 19, 2005 Robert Barnoski. #05-19-2005. (http://www.wsipp.wa.gov/pub.asp?docid=05-19-2005; Outcome Evaluation of Washington State's Research-Based Programs for Juvenile Offenders 2004 January. Robert Barnoski. #04-01-1201 (http://www.wsipp.wa.gov/pub.asp?docid=04-01-1201). See Florida Department of Juvenile Justice (http://www.djj.state.fl.us/Research/index.html) for these reports: Outcome Evaluation Reports (http://www.djj.state.fl.us/Research/OE/index.html); Program Accountability Measures Reports (http://www.djj.state.fl.us/Research/PAM/index.html).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: OJJDP has not undertaken such full-scope evaluations to assess the effectiveness and relevance of its major programs partly because the diverse nature of the activities funded makes such evaluations difficult and expensive to undertake. As noted under Question 2.6, however, OJJDP has: ?? Sponsored a variety of research on issues in delinquency prevention; ?? Contracted through the National Institute of Justice for evaluability assessments of congressional earmark programs; ?? Sponsored the Blueprints for Violence Prevention initiative at the University of Colorado-Boulder in its efforts to rate juvenile justice programs for evidence-based effectiveness and their ability to be replicated; ?? Compiled the Model Programs Guide, an online database of juvenile justice programs commonly funded by states and grantees under the Juvenile Justice Programs that are rated for their evidence-based effectiveness; ?? Supported launch of the Helping America's Youth (HAY) website, which also provides an online database of juvenile assistance programs rated for their evidence-based effectiveness; and ?? Set targets for grantees to achieve in implementing evidence-based programming. Together, these steps comprise a body of evaluation-related activity that provides good evidence of effectiveness and relevance of many of the programs funded by grantees under the Juvenile Justice Programs. Using data from the Model Programs Guide database, OJJDP currently estimates that only 26 percent of its grantees have implemented programs that are effective based on evidence-based evaluations. While small, this is still a significant percentage and demonstrates the potential that the programs are resulting in a positive outcome. OJJDP has set targets for its grantees to increase this percentage in coming years, which should demonstrate increased effectiveness of the programs. The rating scheme employed by the Model Programs Guide needs improvement to ensure that programs rated "Effective" have had a positive impact as measured by a recent, high quality, quasi-experimental evaluation. Further, there must be some evidence of the capability for replication of the programs to other sites. OMB plans to work with OJJDP and other agencies to better synchronize evidence-based program rating schemes.

Evidence: Some of the ongoing program evaluations are as follows: Evaluation of Enforcing Underage Drinking Laws - Community Trial Initiative Wake Forest University School of Medicine; National Evaluation of Free To Grow - Wake Forest University School of Medicine; Evaluation of Safe Schools/Healthy Students Initiative - Research Triangle Institute (RTI); Evaluation of the Truancy Reduction Demonstration Program - Colorado Foundation for Families/National Center for School Engagement; Promising Programs for Substance Abuse: Replication and Evaluation - Pacific Institute for Research and Evaluation (PIRE); National Evaluation of the SafeFutures Initiative - The Urban Institute; Evaluation of Parents Anonymous - National Council on Crime and Delinquency; Evaluation of the Child Development/Community Policing Program Model in New Haven, CT, and Charlotte, NC - Caliber Associates; Gang Free Schools and Communities Program Evaluation - COSMOS Corporation. The following recent solicitations issued by OJJDP (and NIJ, through an IAA with OJJDP) outline the research and evaluation standards supported by OJJDP: Evaluation of Mentoring Initiative for System Involved Youth (2006) (http://ojjdp.ncjrs.gov/grants/solicitations/06mentoringeval.pdf); Outcome Evaluations of Violence Prevention Programs (http://www.ncjrs.gov/pdffiles1/nij/sl000732.pdf). OJJDP also plans to release a solicitation for Field-Initiated Research and Evaluation projects shortly. Aside from directly promoting good research practices through its funding solicitations, OJJDP also works to educate the field regarding strong evaluation design and methodology. Resources such as the Blueprints initiative, OJJDP's Model Programs Guide, and the Helping America's Youth website provide guidance to those designing and carrying out strong evaluation designs. More information can be found at: OJJDP's Model Programs Guide (http://www.dsgonline.com/mpg2.5/mpg_index.htm); Blueprints for Violence Prevention (http://www.colorado.edu/cspv/blueprints/index.html); and Helping America's Youth Tool (http://helpingamericasyouth.gov/programtool.cfm).