Detailed Information on the
Runaway and Homeless Youth Assessment

Focus on site monitoring and technical assistance efforts on performance improvement. Milestone: Incorporate outcome goals into the Associate Commissioner's performance contract, and cascade down to staff.

Complete the design and implementation of an evaluation to assess the long-term outcomes on homeless youth in the transitional living (residential) program. Milestone: Finalize survey instrument (i.e. obtain OMB approval) and publish second (final) Federal Register notice.

Focus on site monitoring and technical assistance efforts on performance improvement. Milestone: File/implement 80% of required corrective action plans from pilot and provide intensive technical assistance to remaining 20% throughout year.

Develop new measure(s) and data collection capacity to track preventive activities by Basic Center Programs (BCPs). Milestone: Review six month data from new BCP element and establish baseline; if information not sufficient, establish baseline after semi-annual data transfer, May 2009.

Identify and address the barriers that led to the program not meeting some targets. Milestone: Analyze FY 06 data and compare with previous performance.

Complete the design and implementation of an evaluation to assess the long-term outcomes on homeless youth in the transitional living (residential) program. Milestone: Define tasks and select research organization.

Focus on site monitoring and technical assistance efforts on performance improvement. Milestone: Incorporate outcome goals into the Associate Commissioner's performance contract, and cascade down to staff.

Complete the design and implementation of an evaluation to assess the long-term outcomes on homeless youth in the transitional living (residential) program. Milestone: Develop survey instrument.

Focus on site monitoring and technical assistance efforts on performance improvement. Milestone: Finalize new on-site monitoring protocols.

Identify and address the barriers that led to the program not meeting some targets. Milestone: Analyze midyear FY 2007 data and compare with previous performance.

Complete the design and implementation of an evaluation to assess the long-term outcomes on homeless youth in the transitional living (residential) program. Milestone: Consult with partners (Oregon coalition, National Alliance to End Homelessness, staff at sites visited, etc.) and finalize survey instrument prior to submission to OMB approval.

Focus on site monitoring and technical assistance efforts on performance improvement. Milestone: Finalize monitoring protocol based on pilot testing at 60 sites by 17 staff.

Develop new measure(s) and data collection capacity to track preventive activities by Basic Center programs (BCP). Milestone: Develop new measurement language and definitions for RHYMIS, obtain OMB collection approval.

Develop new measure(s) and data collection capacity to track preventive activities by Basic Center Programs (BCPs). Milestone: Program new data elements and distribute new software version to RHY grantees; hold webinars to roll out new content and technical elements for RHY grantees, especially new starts (BCP).

Measure: Increase the proportion of youth living in safe and appropriate settings after exiting ACF-funded Transitional Living Program (TLP) services.

Explanation:This measure addresses the core mission of TLP: to provide safe places and developmental experiences that will lead to safe and appropriate living situations when older homeless youth (ages 16-21) complete the program. "Safe and appropriate" exits include entry into independent living and housing situations, residential apprenticeships, higher education, military service and other destinations with positive opportunities; mental health or correctional systems can be appropriate or necessary placements. The following exit situations are regarded as "not safe and appropriate" for TLP: the street, the unknown, and homeless shelters. Computation: number of safe and appropriate exits/total number of youth exiting services. Why the target is ambitious: The "safe tenure rate" for both BCP and TLP programs (the two programs comprising RHY) has almost always been 99 to 100%. While the "safe exit rate" for BCP has remained consistent at approximately 90%, the TLP rate has remained lower because the measurement definition is stricter, given program goals. RHY youth tend to be high risk and often difficult to treat, have a high proportion of mental health and substance abuse problems, and often experiences of abuse and neglect by their families. Exits by youth into shelters from TLP are not counted as "appropriate." Thus TLP's long term target rate is 85%, instead of the current level of 90% as in BCP. (Shelters may be appropriate in BCP when children taken in by BCPs a great distance from their home may be temporarily housed in family or youth shelters during homeward relocation.) Additionally, older TLP youth, as do youth in general, have higher probabilities of risk behavior, including dropping out of programs or being expelled. Because of its structure and goals, TLP may remain below 90%. The TLP level has remained below 80% until it began rising after specific attention from FYSB during FY 2004-2005. Therefore, it is logical to move from a measure of the entire RHY program to a measure of only the TLP program. In 2010, the results will be analyzed to determine if TLP targets should be higher, or maintained at 85%. To seek unlimited increases in target levels could result in programs favoring lower-risk youth for entry into scarce TLP slots. An RHY youth's commitment should always take precedence over the severity of his or her barriers to success. FYSB will closely study if yet higher levels can be achieved without negative effects.

Measure: Increase by 2% annually the proportion of youth who are prevented from running away through BCP in-home/off-site services as a percentage of all youth receiving such services, including those who must be fully admitted to the shelter despite such preventive efforts.

Explanation:The Runaway and Homeless Youth Act provides that BCP programs can deliver services to youth and families on a non-residential, outpatient, in-home or off-site basis to prevent episodes of running away. They also help families in crisis deal constructively with conflicts and other causes of risky or destructive decisions and behavior. Home-based services include: 24-hour service to respond to family crises and counseling, as well as information and support for youth and families (including services relating to basic life skills, interpersonal skills, educational advancement, job attainment skills, mental and physical health care, parenting skills, financial planning, and referral to sources of other needed services). Computation: The proportion of youth receiving various modalities of preventive services that are not entered into full shelter services as a percentage of all youth receiving such preventive services. Why the target is ambitious: FYSB plans to increase the level of prevention at least 8 points by 2010 then assess higher targets. FYSB will proceed carefully to avoid unduly influencing programs into urging youth to stay at home in situations where youth are at risk of victimization in the home or may harm others in the household. FYSB is enriching the RHYMIS data element(s) for this measure during FY 2006 and will have a more complete picture of in-home services either during the second half of FY 2006 or beginning with FY 2007.

Measure: Increase the proportion of Transitional Living Program (TLP) youth who are engaged in community service and service learning activities while in the program.

Explanation:An important principle of positive youth development is creating opportunities where youth can make a difference and have a positive impact in their community. Being engaged in the community can be a powerful stimulant of self-efficacy and pro-social attitudes. This has been recognized in many studies. Youth need not only safe settings, appropriate structure, and supportive relationships; they also thrive upon opportunities to belong, develop positive social norms, express civic engagement, and learn skills. Such youth in RHY programs are more likely to successfully reunite with their families and/or navigate toward independence. The TLP program provides sufficient time over a number of months for such experiences, along with structured opportunities for youth to reflect upon their ability to be an asset to others and their communities. Short BCP tenures make service learning programs less feasible (making it reasonable to focus exclusively on a measure within TLP), but FYSB is exploring service models suitable for BCP. Computation: Youth receiving service learning experiences/total youth exiting services. Why the targets are ambitious: Community service learning is a valuable stimulus for positive youth development (PYD), but it may not be appropriate or feasible for each youth. It has been rising steadily in TLP, due largely to FYSB's emphasis on PYD (for example, in program announcement). Continuous increases may be feasible, but the target level should be set at a point that represents the best allocation of treatment among diverse TLP youth, who vary by risk levels, needs, and fitness or ability to profit from such experiences. Analysis and research will help formulate an optimal targeting approach.

Measure: Increase factors contributing to safe exits by increasing the percent/number of youth who complete the transitional living program (TLP) by graduating or who leave ahead of schedule based on a positive opportunity.

Explanation:A youth's completion of his or her TLP program (even when ahead of schedule) has been shown to increase the likelihood of a safe and appropriate exit, compared to situations where youth drop out with no plans or are expelled. Longer tenure in the program also improves educational and employment progress. This measure primarily focuses on decreasing the percentage of TLP youth who drop out with no plan or opportunity to pursue. Computation: youth completing TLPs/total youth exiting TLPs. Why the targets are ambitious. A two percent increase each year represents a struggle against the problem of very difficult to serve youth. If the long term target is reached, we will study whether aspiring to a further rise is appropriate. We feel it is right only to focus on reducing the national "drop out" rate. Pushing for a reduction in the expulsion rate as well may conflict with issues of endangerment, unacceptable conduct, and other factors that can only be addressed case-by-case by youth professionals at the scene.

Measure: Increase funding efficiency by increasing the percent/number of youth who complete the transitional living program (TLP) by graduating or who leave ahead of schedule based on a positive opportunity.

Explanation:This measure accounts for efficiency in the TLP because youth who complete the programs maximize the benefits, as demonstrated by the positive correlation of length of participation and successful completion with superior educational and employment outcomes. Additionally, a youth's completion of his or her TLP (on schedule or ahead of time, with a revised transitional plan) has been shown to increase the likelihood of a safe and appropriate exit, compared to situations where youth drop out with no plans or are expelled. In particular, RHYMIS data demonstrate that youth who drop out of a TLP are more likely to return to the streets, which is an unsafe exit. Computation: youth completing TLPs/total youth exiting TLPs. Why the target is ambitious. A two percent increase each year represents the difficulty of working with this population of youth at-risk. If the long term target is reached, FYSB will study whether aspiring to a further rise is appropriate. This measure primarily focuses on decreasing the percentage of TLP youth who drop out with no plan or opportunity to pursue. Pushing for a reduction in the expulsion rate as well may conflict with issues of endangerment, unacceptable conduct, and other factors that can only be addressed case-by-case by youth professionals at the scene.

Is the program purpose clear?

Explanation: Although the purpose of the Runaway and Homeless Youth Program is multifaceted, it is clear and consistent: to develop an effective system of care outside the law enforcement, juvenile justice, child welfare and mental health systems for youth who have become homeless or who leave and remain away from home without parental permission. There are several components to the RHY network; some are nation-wide, and some are community-based, and each is designed for different, but related, populations and problems. The system provides: prevention services for youth in high risk situations who may be thinking of running away; a national hotline and street-based programs for youth already on the run; short-term emergency shelters and reunification/placement services for younger runaway youth; and intensive residential programs for older homeless youth unlikely to return home to help them develop the skills and capacities they need to live independently. Some youth are ultimately served by multiple components. 1. The Basic Centers program (BCP) provides in-home prevention services targeting youth at risk of running away, and shelter and services to runaways, reuniting them with their families or placing them in appropriate living situations. 2. The Street Outreach Program (SOP) finds and builds trust with youth living on the street (and at risk for exploitation, exposure and survival-criminality) and provides them with food, information, health/first aid products, referrals and other services. 3. The National Runaway Switchboard (NRS) is a 24 hour, multi-lingual toll-free hotline that provides runaways and youth in crises with counseling and referrals to services and safe places. 4. The Transitional Living Program (TLP) provides services to homeless youth ages 16-21 in a longer-term residential setting, including educational, employment, and life skills training, and other services to help them make healthy transitions to independent adulthood. 5. In addition, FYSB funds a public education/information dissemination institution (the National Clearinghouse on Families and Youth or NCFY), a network of training and technical assistance (T&TA) providers in each region, and a national management information system IT contract to collects and compile information on system youth for analysis. 6. FYSB works with staff in each of the HHS regions in managing grants, participating in on site monitoring , guiding technical assistance, holding conferences, and others.

Evidence: 1.1.1-Authorizing Legislation: The Runaway and Homeless Youth Act, (Title III of the Juvenile Justice and Delinquency Prevention Act of 1974), as Amended by the Runaway, Homeless, and Missing Children Protection Act (P.L. 108-96) mandates street based services, basic centers, transitional living programs, and the national communications system. 1.1.2-BCP, TLP, SOP and NRS fact sheets and logic models.

Does the program address a specific and existing problem, interest, or need?

Explanation: As programs and shelters meet the basic needs of runaway and homeless youth, other services promote their positive development and help them prepare to make healthy choices in adulthood. Clearly, RHY program address and tackle a specific problem that is complex, pervasive, and formidable. The U.S. Department of Justice estimated that nearly 1.7 million young people ran away or were thrown out of their homes in 2002 alone. An estimated 71% of those youth experienced substance or drug abuse problems, had been sexually or physically abused, or had witnessed criminal acts in their homes. The prevalence sexual abuse alone "ranged from 21% to 42%," compounded by the practice of survival sex and criminality by many homeless youth. (RTI, 1999) The RHY program provides runaway and homeless youth with emergency shelter, food, clothing, counseling, referrals for health care, and other critical services. In FY 2005, 514,657 young people living "on the street" received food, counseling, information, first aid, and other services through 146 street outreach programs; approximately 55,067 exited 342 basic centers where they found services and a safe place, and over 3,048 exited intensive residential services in 194 transitional living programs. (3,765 youth entered TLP during the year. Many remain in the program up to 18 months or more.) In FY 2005 the National Runaway Switchboard hotline received 24,877 incoming calls from youth in crises. The NRS handles over 100,000 calls each year, interfacing with youth in crises, coordinating service with local agencies, and arranging travel home (free to over 10,000 since the service began in 1971).

Evidence: 1.2.1-Hammer, H., Finkelhor, D., and Sedlak, A. October, 2002. Runaway/Thrownaway Children: National Estimates and Characteristics. From National Incidence Studies of Missing, Abducted, Runaway, and Thrownaway Children, (NISMART) U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. 1.2.2-Research Triangle Institute (RTI) Child Trends, Pacific Institute for Research and Evaluation, and the Seattle Homeless Adolescent Research Project (1999). "Sexual Abuse among Homeless Adolescents: Prevalence, Correlates, and Sequelae" (Table of Contents and Executive Summary) 1.2.3-Robertson & Toro. (1998) "Homeless Youth: Research, Intervention and Policy." Presented at National Symposium on Homeless Research. (pages 1-13) NOTE-Examples of need for more capacity in some areas already served by RHY programs (not to mention areas where none exit): ?? Transitional Living Program - FY 05-Waitlists/Turnaway* Requests for Assistance- By Phone-2100; In Person-455. ?? Basic Centers Program - FY 05 Turnaway* Requests for Assistance- By Phone-1,848 In Person-230 NOTE-Youth turned away are referred or transported to other services with capacity or more appropriate or where eligibility is a factor. Often there are just not enough beds or the maximum youth-to- staff ratio is reached and State or Federal standards prohibit additional intake. 1.2.6-RHYMIS Instrument (see 3.1) 1.2.7-Examples of RHYMIS BCP data, which contains most of the same elements as TLP. NOTE: RHYMIS data used in reports and caseload analyses are pre-validated at input by edit checks and business rules. Prior to analysis, extensive cleanup and revalidation is conducted. RHYMIS studies by FYSB, NORC, and OPRE. quoted in this PART are based not on samples but on the universe of youth in services during the periods studied. Other studies cited in this PART that used RHYMIS focused on specific regions (i.e., four Midwestern States in Pollio, 2002, see 4.5). 1.2.8-NRS statistics 1.2.9-Positive Youth Development Fact Sheet 1.2.10-National Clearinghouse on Families and Youth, (1996) "Reconnecting Youth & Community," The Exchange, www.ncfy.com (Table of Contents and Introduction) 1.2.11-National Research Council and the Institute of Medicine, Committee on Community-Level Programs for Youth, (November, 2004) issue brief on "Community Programs to Promote Youth Development." (See 2.1) 1.2.12-Family and Youth Services Bureau, Agendas for National Youth Summits, (youth planning committees): FY 2003 (page 2), 2004 (inside cover) and 2005 (inside cover)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Unlike other Federal, state, local or private effort, RHY programs focus specifically on the needs of runaway and homeless youth outside of the child welfare, mental health, and juvenile justice systems. Similar services exist, but for very different populations. For example, the Foster Care Independent Living Program serves child welfare system youth (wards of the State) aging out of Foster Care. Housing and Urban Development (HUD) programs for the chronically homeless serve adults and families. While the TLP resembles HUD-funded transitional programs for adults, it addresses the specific physical and developmental needs of adolescent youth, who often cannot be safely housed either with adults or among families with small children. Older youth, particularly males, are often excluded from Family Violence Prevenetion shelters. BCP, on the other hand, is a family reunification or child protection program, that provides shelter on an interim basis. The various components of the RHY system are complementary, not duplicative, as discussed in 1.1. The RHY program supports community-based organizations that often secure supplemental or additional funding from other federal and state grants programs and private donations. Some organizations provide additional complementary services to youth or take on overflow from centers at full capacity. FYSB funds are awarded based upon need and avoid duplicating sufficient existing services in the local area. The RHY grants are rarely sufficient* to fund the full range of services set forth in the RHYA, but provide organizations with seed money, entree, and credibility (as a competitively-funded federal grantee), and enable them to leverage other public and private funds. (See Juneau, Alaska example, Evidence column, 2.5) * Average annual BCP grant, FY 2003: $128,000 Average annual TLP grant, FY 2003: $192,000 Average annual SOP grant, FY 2003: $94,000 (FY 2002-2003 Report to Congress)

Evidence: 1.3.1-BCP, SOP and TLP Program Announcements 1.3.2-The Foster Care Independent Living program legislation (Section 470 and 477 of the Social Security Act, "Findings"). 1.3.3-HUD statements of Program Goals and Objectives for HUD housing programs (HUD serves adults and families, but not unaccompanied minors) (www.hud.gov/offices/cpd/homeless) 1.3.4-The Domestic Violence Battered Women's Program legislation (Section 310 and 311(g) of the Family Violence Prevention and Services Act).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: According to the prevailing literature, the strategies and developmental frameworks used by the RHY program to serve runaway and homeless youth are consistent with theoretical, research, and effective practice models (Finance Project, 2005). The RHY program awards grants on a competitive basis, requiring a ten percent match. There is no evidence to suggest that an alternative mechanism or different match rate would be more efficient or effective. In fact, grantees tend to leverage many sources of funding. Based on comparable programs, a block grant to States or outsourcing the national system would entail large administrative fees and overhead costs, which would mean an unavoidable reduction in the number of centers that could be funded, even if administrative costs were kept to 5-10%. Overhead costs for private vendors can be as high as 40% or more, on top of salaries, expenses, and other direct costs. In comparison, Statute limits FYSB non-project grant expenses to 10%, including costs for the National Switchboard, training and technical assistance networks, NCFY, travel for on-site monitoring, grant review logistics, RHYMIS, research and evaluation. FYSB and regional staff salaries and travel are paid by ACF S&E administrative funds. A voucher system would be inappropriate for the program, in that the potential beneficiaries are nearly impossible to identify or track. With only a few exceptions, tribes are the only "public agencies" eligible for SOP funding. Almost all providers are community-based, which ensures that programs are tailored and attuned to local circumstances and connect and collaborate with local partners. This is valuable in BCP and TLP, but essential for SOP. Many at-risk youth avoid institutional and social service settings but are comfortable in neighborhood centers where their peers congregate. Unsatisfactory alternatives for runaways have historically included jail, adult homeless shelters, and youth detention centers.

Evidence: NOTE-"??. It is clear that the support provided through RHYA, in combination with support from other federal programs, state and local governments, and private funders, has enabled local youth-serving agencies to develop an array of innovative approaches to reach and serve this vulnerable population." -- The Finance Project (2005), "Promising Strategies to End Youth Homelessness"; draft report with literature review and research assessment.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Again, although there are several components to RHY, each is designed to effectively target its intended population and fulfill the program's purpose. Statute requires that state funding for RHY Basic Centers be determined by a formula based upon each state's population of youth under 18 (according to most recent census data), which is an effective way to predict local need. Since the shelters provided by Basic Centers must be prepared to respond to the immediate, situational needs of youth in their community on demand. While some youth migrate out of state, the majority run away within a 50 mile radius. All grants are awarded competitively, and require that applicants establish the need for assistance in a local area and the characteristics of the targeted population to ensure that targeted beneficiaries are those RHY means to be served and can be served effectively and efficiently. Credible documentation of this is weighted heavily in this section of the grants review process, and plans and principles for serving these youth must also be described. Federal Staff in the regional offices travel to each site to monitor RHY services, improve overall program quality, and ensure the attainment of targeted and measurable results. The ten regional training and technical assistance networks facilitates the coordination of services withing and across states, as large areas often share important commonalities.

Evidence: 1.5.1-Program Announcements (see 1.3) 1.5.2-State Funding Based on Census Population Data. Department of Health and Human Services, Administration for Children and Families (ACF), [Program Announcement No. HHS-2006-ACF-ACYF-CY-0063 (pages 9-12) 1.5.3-List of current RHY grantees by State and type of grant(s) 1.5.4-Regional Office Youth Specialists 1.5.5-Regional T&TA providers

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Source for all measures: Runaway and Homeless Youth Management Information System (RHYMIS) 1. Increase the proportion of youth living in safe and appropriate settings after exiting ACF-funded TLP services (revised). This measure addresses the primary mission of TLP: to provide safe places and developmental experiences that will lead to safe and appropriate living situations after completion. "Safe and appropriate" exits can be entry into independent living and housing situations, residential apprenticeships, higher education, military service, and other positive opportunities. In some cases mental health or correctional facilities may be appropriate safe exits. The following exit situations are "not safe and appropriate" exits following TLP: the street, the unknown, and homeless shelters. Computation: number of safe and appropriate exits/total number of youth exiting services. 2. Increase by 2% annually the proportion of youth prevented from running away after receiving BCP in home/off-site services. (new) The Runaway and Homeless Youth Act authorizes BCP programs to provide services to youth and families on a non-residential, in-home, or off-site basis. They also help families in crisis deal constructively with conflicts and to prevent running away. Home-based services can include: 24-hour crisis services, counseling, training (basic life skills, interpersonal skills, educational advancement, job skills, financial planning, or parenting), mental and physical health care, and referral. Research has shown that these types of services can prevent risky behavior like running away. (2.1.5) Computation: The proportion of youth receiving preventative services who are not entered into full shelter services as a percentage of all youth receiving preventative services. ("Full admission" means moving in to a BCP for more than a drop-in visit or brief "cooling off" period. See evidence 2.1.6. for new data elements.) 3. Increase the proportion of youth who are engaged in community service/ service learning activities while in a TLP program (revised). Engaging in the community can be a powerful stimulant of self-efficacy and pro-social attitudes. (see 1.2) Youth need more than safe settings, appropriate structures, and supportive relationships; they thrive when given opportunities to belong, develop positive social norms, and learn skills. RHYMIS data also shows that participation in service or service learning correlates with successful reunification and/or successful transitions. The TLP program is long enough to provide such experiences, as well as structured opportunities for youth to reflect upon their value to others and the community. Short BCP tenures make service learning programs less feasible, but FYSB is exploring streamlined service models. Computation: Youth participating in service learning experiences/total youth exiting services. While the Runaway and Homeless Youth program aims ultimately to promote the safety and well-being of its target population, it is difficult, if not impossible, to track such a transient population over a long period of time. The program will still attempt to track client outcomes, but because these results are so far removed from the intervention, these measures are most appropriate to evaluate program performance.

Evidence: NOTE 1: In previous performance budgets, ACF defined this goal in terms of the both TLP and BCP, which, combined, has remained at 89-90% for several years. This is because BCP is ten times the size of TLP, with a less stringent definition of "safe and appropriate"*. BCP has demonstrated a 89-90% "safe and appropriate exit" rate for several years. TLP has, in contrast, has historically has rates well below 80%, partially due to a stricter standard for "safe and appropriate" exits. (*Exits to homeless shelters from the TLP are not counted as successful or appropriate). The youth served are almost always 100% safe while under the direct care of either program. The combined RHY "success" rate on safety is probably above 90%, and higher than actuals reported for both BCP and TLP because (1) shelter exits from TLP are safe, but are not counted as successful and (2) exits into the "unknown" in both programs count for nearly half of the unsafe exits. It is possible that many "unknown" exits are in fact safe and appropriate but are not reported back to the shelter/TLP. The status of many "unknown" exits later become known to the program, but only after RHYMIS data has been transferred to the national data base. Data updates are accepted into the national data base well after the deadline, but often not in time to be included in the Performance Budget. FYSB and Regional staff have been discussing with grantees the importance of data accuracy and the need to update records. FYSB and regional staff spent FY 2004 and FY 2005 talking about the safe exit issue at TLP conferences and meetings, providing examples of effective exit services, and enlisting regional office and T&TA network support. FYSB focused on increasing "program completion rates" in TLP, predicting that this would also improve TLP safe exit rates. As anticipated, after the program completion rate (efficiency measure) went up two years in a row, during FY 2005 the TLP safety rate increased by 4 percentage points to 81.8% 05. 2.1.1-2007 Performance Budget 2.1.2-After the original PART (conducted in FY 2003), the re-engineering of RHYMIS has supplied FYSB with more reliable national data than previously was available. Before 2002, only 45%-55% of grantees reported thru RHYMIS as required. Since 2002, response levels have been at 99% and above (even after '05 hurricanes). 2.1.3-The first full year of this data was FY 2002, which was submitted early in FY 2003. As subsequent years of data in RHYMIS have enriched understanding of the RHY programs and the youth served, FYSB believes that a few revisions in RHY performance measures are warranted in order to maximize the effectiveness of funding, oversight and technical assistance efforts. (Further discussion is under 2.2.) 2.1.4-(For actual and target performance data, and discussion of several revisions and additions, see 2.2.) 2.1.5-Among the many assets identified by the National Academies of Science/Institute of Medicine as key for youth's social development were: ?? Sense of social place/integration, being connected and valued by larger social networks; ?? Attachment to prosocial/ conventional institutions, such as school, church, and nonschool youth programs; ?? Ability to navigate in multiple cultural contexts; ?? Commitment to civic engagement."; -- National Research Council and the Institute of Medicine, Committee on Community-Level Programs for Youth, (November, 2004) issue brief on "Community Programs to Promote Youth Development." 2.1.5: "Effective out-of-facility services for at-risk youth." 2.1.6: Data elements added to RHYMIS to capture preventive service information, approved by OMB for addition to RHYMIS, March 2006

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: 1. Increase the proportion of youth living in safe and appropriate settings after exiting ACF-funded TLP services. (See NOTE 1) FY 2010 Long Term Target 85% FY 2006 Target 83% FY 2005 Actual 82% FY 2005 Target 80% FY 2004 Actual 78% FY 2004 Target 80% FY 2003 Actual 78% FY 2003 Target 80% FY 2002-2003 Baseline 79% Why ambitious: While the "safe exit rate" for BCP has remained consistent at approximately 90%, the TLP rate has remained lower because the youth are older and the standard stricter. Exits from TLP into shelters are not deemed "appropriate," according to TLP goals. This measure remained below 80% in TLP until FYSB paid specific attention to it in FY 2004-2005 (See 4.1). RHY youth tend to be high risk and often difficult to treat. Many have mental health and substance abuse problems and have been abused or neglected. Additionally, the older the youth, the higher the probability of risky behavior,. By its very nature, TLP may never reach the 90% level, but the targets push for incremental progress. The targets are ambitious because the rate has hovered around 80% for years and because the 20% of cases that do not make safe exits logically tend to be the most difficult and resource-intensive. Setting targets too high could cause programs to favor lower-risk youth when filling scarce TLP slots (or "cream"). FYSB has committed to studying how high targets can be set without creating perverse incentives. (SEE NOTE 2) 2. Increase by 2% annually the proportion of youth prevented from running away after receiving BCP in-home/off-site services. (See NOTE 3) FY 2010 Long Term Target FY 2006 +8% FY 2009 Target: FY 2008 +2% FY 2008 Target: FY 2007 +2% FY 2007 Target: FY 2006 +2% FY 2006 Baseline: Dec 2006. Why ambitious: FYSB hopes to increase the level of prevention steadily by at least 8% by 2010, but proceed carefully to avoid creating perverse incentives. (NOTE 2) FYSB is expanding RHYMIS to track this measure during FY 2006 (see evidence 2.1.6) and will have a more complete picture of in-home services either in the second half of FY 2006 or in FY 2007. 3. Increase the proportion of youth who are engaged in community service learning (CSL) while in a TLP program. (NOTE 4) FY 2010 Long Term Target 35% FY 2009 Target 35% FY 2008 Target 34% FY 2007 Target 33% FY 2006 Target 32% FY 2005 Actual 31% FY 2005 Target 30% FY 2004 Actual 27% FY 2004 Target 30% FY 2003 Actual 29.8% FY 2002 Actual/Baseline 27.5% Why ambitious: FYSB expects over one-third of TLP youth to participate in CSL by FY 2009. Community service learning is a valuable tool for positive youth development, but it may not be appropriate or feasible for every individual youth. Use has been rising steadily in TLP, due largely to FYSB's emphasis on youth development at conferences, in program announcements, etc. Continual increases may be feasible, but the ultimate target should be set at a level that represents the best allocation of treatment among the diverse population of TLP youth, who vary by risk levels, needs, and potential gain from CSL. Analysis and research will help determine the best target and whether to aim for higher levels after 2010.

Evidence: NOTE 1: As stated above, several revisions in the FY 2007 Performance Plan are in process, involving a few shifts in goals. For example, because BCP safe exit level has been consistently high, the emphasis will be moved to TLP. (The TLP baselines for safe exits and program completion (efficiency measure) were calculated over two years of data, FY 2002-2003, since TLP tenures can last well over twelve months or over two years.) NOTE 2: FYSB does not expect RHY programs to favor easy-to-serve participants, but would not want to make family reunification, for example, of such importance, that youth are convinced to remain at home when safety dictates otherwise. Similarly, it would not want to impose CSL on youth who are not likely to benefit and on whom costly services could be wasted. Annual increases in RHY targets are comparable or greater than those set by HUD's homeless programs(e.g., 2% -v- 1%). NOTE 3. BCP performance will be gauged by the effect of prevention services provided by programs (See 1.6.2. for more detail). NOTE 4. The community service learning measure will focus on TLP, since BCP services are so brief, although preliminary analyses suggest that CSL services correlate with better safe exit rates for both TLP and BCP. For this reason, ACF is developing streamlined, short term models to try in BCP. Data on revised or retired measures (such as BCP safe exits) will continue to be collected and analyzed (and reported in Performance Budget as required). 2.2.1-2007 Performance Budget (see 2.1); 2.2.2 HUD PART http://www.whitehouse.gov/omb/expectmore/detail.10001234.2005.html

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: All long-term goals detailed in 2.2 are tied to annual targets, with baselines for measuring progress, as shown. FYSB added the BCP prevention annual/long-term measure because reaching at risk youth and preventing them from running away is a key priority because after youth leave their homes, they often are exposed to the considerable dangers of life on the street. Dangerous situations and family conflict can often be assessed, addressed, and corrected by home visits and therapy. Preventing youth from running away in the first place clearly ties to the overall condition of program beneficiaries. The annual performance measures, measuring safe exits from programs, runaway prevention, and others, are telling intermediate outcomes that are directly related with bettering the overall outcomes for program participants and for runaway and homeless youth.

Evidence: 2.3.1-2007 Performance Budget (see 2.1)

Does the program have baselines and ambitious targets for its annual measures?

Explanation: As explained in incremental annual targets and baselines for long term measures in 2.2, RHY sets targets for these outcomes with a careful eye for balancing ambition and foresight. Because setting targets at unrealistic levels could discourage grantees from seeking out and treating the most difficult clients, and because the program is intended for runaway and homeless youth regardless of the severity of their situations, FYSB has set targets in consultation with relevant research and grantees, and will continue to adjust them as appropriate. However, as they stand, the targets are quite progressive.

Evidence: 2.4.1-2007 Performance Budget (see 2.1)

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: RHY programs can only be effective if all partners collaborate towards the overall mission, and the program is structured to ensure that this is the case. Grantees must document adherence to the goals and standards of the program in mandatory reports that relate to the accomplishment of program goals. Grantees are required by contract to submit service and performance data through the RHYMIS contractor. Program goals and requirements are included in Program Announcements, Cooperative Agreements with T&TA providers, contracts (RHYMIS, Research and Evaluation), reports to Congress, and presentations at annual meetings of T&TA providers, TLP grantees, RHY grantee community, Regional meetings, etc. By Statute, regional staff monitors one-third of the RHY grantees on site each year. A logistics contract supports on site visits by peers from other organizations and federal regional staff to conduct reviews and audits. The teams inspect operations, interview management, staff, and youth, ensure accountability, and enhance grantees' ability to deliver services appropriately and effectively within established standards, goals, and measures. Corrective action plans and technical assistance remedies are imposed as needed, or grants may be terminated. The National Clearinghouse on Families and Youth supports annual regional meetings, supports workgroups, and publicizes program objectives and new initiatives. NCFY serves as a central information point for professionals and agencies involved in the development and implementation of services to young people and their families. Annual RHY grantee meetings in each region are supported by ten training and technical assistance organizations. THey also participate with FYSB in the annual national meetings of the grantees' association, the National Network for Youth.

Evidence: 2.5.1-Program Announcements (See 1.3) 2.5.2-2007 Performance Budget (see 2.1) 2.5.3-RHY Performance Standards 2.5.4-National Clearinghouse on Families and Youth, (May, 2005), "Thinking Positively: FYSB Grantees Promote Positive Youth Development in Exemplary Ways," The Exchange www.ncfy.com/pubs/may2005.htm 2.5.5-Sample Cooperative Agreement with T&TA providers 2.5.6-Sample T&TA provider E-newsletter 2.5.7-Agendas for National Network for Youth Symposium, National T&TA meeting, Region III meeting. (See 4.1)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: High-quality national, multi-year studies and evaluations are regularly conducted by independent organizations on the range of RHY programs to support program improvements and evaluate effectiveness and relevance to RHY's purpose. Each year comprehensive national data sets are regularly analyzed using both in-house and independent expertise for validation, certainly sufficient in scope in itself. Since the last evaluation of the RHY program, it has solidified its own program-wide reporting and data collection system and provided for its analysis. RHY programs report BCP and TLP data for RHYMIS every six months, and the RHYMIS contractor validates the data. During FY 2005, the National Opinion Research Center analyzed BCP data from 2002, 2003, and 2004 (all BCP youth, N = 171,000) to understand the situations, characteristics, and needs of BCP youth, and to guide program design. (NORC is subcontracted on the RHYMIS contract to study RHY data each year.) The study identified factors associated with unsafe exits and ranked high and poor performing RHY programs by risk levels of youth in their programs. In FY 2005, OPRE (ACF) analyzed FY 2003 BCP data and verified FYSB's findings. During FY2004 and 2005, FYSB conducted quantitative analyses of FY 2002-2003 TLP data (all TLP youth, N= 5,123) to examine safety on exit from RHY programs and what the causal factors involved. In FY 2005, the Finance Project validated strategies used in FYSB programs and elsewhere to combat youth homelessness. In 2003, the Research Triangle Institute reviewed issues in estimating the number of runaway and homeless youth. In 1997, a CSR, Inc. study showed that successfully completing the TLP correlated with self-sufficiency and independence (primary objectives of TLP). Compared to similarly at-risk youth, youth completing 10 TLPs were more likely 6 months after entry to be employed, employed and attending school, in college, or to have savings. In 1997 a Westat found that 88% of youth in BCPs reported that they were "very" or "somewhat satisfied" with services in 6-month follow-up interviews. Overall, six months after their stay, most youth positively rated their situation in 12 critical areas. 77% with physical abuse as a presenting problem were either better or much better at follow-up. 68% with law enforcement problems were better or much better following services. In 2002, "Short-Term Outcomes for Youth Receiving Runaway and Homeless Shelter Services" found that RHY youth who utilized shelter services had better outcomes six weeks after exit and reported fewer days away from home, improved feelings of support from families, and fewer school suspensions, expulsions and detentions; more youth were employed and less were sexually active. Nearly all of the research done on RHY programs has been conducted by entirely independent, unbiased, and respected research organizations. Although there is more that could be done to design research with program design in mind, the agency has developed a detailed workplan for a longitudinal study that would surely have implications for program design. Future: An independent national evaluation of long term outcomes for TLP youth has been funded and is in the design stage (see 2.6.16).

Evidence: RHY program structure, legislative requirements and other characteristics have changed little since the nineties, at the time of the earliest studies in this section. 2.6.1-RHYMIS Instrument (see 3.1) 2.6.2-Pergamit, M., Ernst, M., (2005), "Exploratory Analysis of the Basic Center Program Data in the RHYMIS Database," National Opinion Research Center/ University of Chicago. (Table of Contents, Introduction) 2.6.3-Trivits, L., (2005), "Preliminary Data Analysis of Basic Center RHYMIS Data," ACF Office of Planning, Research and Evaluation. (Introduction) 2.6.4-FYSB and ACF Region 10) (2004), "Post Service Safety in the Transitional Living Program." (See 4.1) 2.6.5-The Finance Project (2005), "Promising Strategies to End Youth Homelessness"; draft report with literature review and research assessment. U.S. Department of Health and Human Services. (See 1.4) 2.6.6-Research Triangle Institute et al, (May, 2003). "Incidence and Prevalence of Homeless and Runaway Youth," U.S. Dept of Health and Human Services. (Table of Contents, Sections 1 and 2) 2.6.7-Pollio, Thompson, Constantine, Reid, Von Nebbit, Washington University. St Louis, MO. (September, 2002), "Short Term Outcomes for Youth Receiving Runaway and Homeless Shelter Services" (long and short term outcomes of youth in youth shelters in four Midwestern States, N = 261), Research on Social Work Practice: 12:5. P. 589-603 (See 4.5) The following studies are nine to ten years old, but RHY program structure, legislative requirements and other characteristics have changed little. 2.6.9-CSR Inc. (1997), Evaluation of the Transitional Living program for Homeless Youth, N = 285; 175 TLP participants and 110 comparison youth, U.S. Dept of Health and Human Services, (binder includes Phase 2 Table of Contents and Executive Summary up through "Findings of the Impact Evaluation" and "Conclusions;" original is approx. 200 pages but has been provided to OMB examiner) (See 4.5) 2.6.10-Westat, Inc., and Urban Institute (1997), National Evaluation of Runaway and Homeless Youth (BCP program). N = 969. (in depth interviews, comparisons calculated against Youth Risk Behavior Survey (CDC, 1995) and purposive samples of street youth in seven cities (Macro international, 1992), U.S. Dept of Health and Human Services, (included are Table of Contents and Executive Summary, pages 1 & 2 and "Status at 6 month follow-up," pages 15-20; original is approx. 200 pages but has been provided to OMB examiner). (See 4.5) 2.6.11-Cauce, A.M., & Morgan, C.J. (1994), "Effectiveness of intensive case management for homeless adolescents: Results of a 3-month follow-up," Journal of Emotional and Behavioral Disorders, 2(4). (See 4.5) 2.6.12-Other recent or ongoing evaluations and studies (copies available on request): 2.6.13-Bartlett, Copeman, Golin, Miller and Needle (Wilson, Tanner, ed.), (2004), "Unlocking the Potential of Homeless Youth Adults," New England Network for Child, Youth and Family Services. 2.6.14-Pollio, Thompson, & North, (June, 2000), "Agency-Based Tracking of Difficult-to-Follow Populations: Runaway and Homeless Youth Programs in St. Louis, Missouri," Community Mental Health Journal, Vol. 36, No.3. NOTE: The HHS Assistant Secretary for Planning and Evaluation published a report concluding: "Meaningful formative evaluation is possible without a followup. Conversely, if the evaluation is formative or exploratory in nature an argument can be made for conducting a study that examines only what happens at the immediate point of discharge." This supports the value of FYSB's use of data ending at the immediate point of program exit, (as is available from RHYMIS) as valid and valuable for caseload analysis and service improvement. 2.6.15-Evaluability Assessment of Discharge Planning and the Prevention of Homelessness (http://aspe.hhs.gov/hsp/05/discharge-planning/execsum.htm). 2.6.16-Timeline for TLP Evaluation

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Currently, budget requests for the RHY program are not explicitly tied to accomplishment of the annual and long-term performance goals in the context of the program's budget. ACF will continue to improve its outcome driven performance budget.

Evidence: RHY Performance Budget

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: RHY has undertaken significant efforts to correct its strategic planning deficiencies since its last review. Over the past four years, reengineering and incremental improvements of RHYMIS has yielded much more accurate national data and new data elements of significant interest, without increases in collection burden. Prior to the reengineering effort, which was completed in 2001, only 45%-55% of grantees reported through RHYMIS as required. Since 2002, response levels have been at 99% and above. This can be attributed to the more user-friendly system and intensive messaging by FYSB, regional staff, and T&TA providers. Energetic follow-up by the RHYMIS contractor significantly contributes to these improved reporting rates. This has enabled FYSB to track a large set of indicators (over a hundred variables for each BCP or TLP youth), not all of which are appropriate for strategic outcome budget reporting, but which guide policy, communications and technical assistance. FYSB has continued to develop and refine long-term and annual performance indicators that demonstrate progress towards long term goals and outcomes, improving upon those assessed in the FY 2003 RHY PART. "Safe and appropriate exits" from RHY has long been a primary strategic goal. Since FY 2004, FYSB has been implementing a national Safe Exit Initiative to promote awareness among service providers of promising practices in effective transitional living and discharge planning, exit care, after care, and connection to mainstream services. The initiative involves ACF's national and regional federal staff, technical assistance networks, information dissemination resources, on-site monitoring visits, and data analysis resources. (See 4.1)

Evidence: 2.8.1-2007 Performance Budget (see 2.1) 2.8.2-NEORHYMIS instrument and data elements (See 3.1) 2.8.3-The Safe Exit Initiative has primarily been directed at improving the "safe exit" rates in the TLP, and was successful in raising these rates by four points from FY 2004 to FY 2005. (See 4.1)

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Performance information is collected from grantees via several of vehicles: semi-annual RHYMIS data submissions and annual reports, statistics compiled by the National Runaway Switchboard, monthly grantee progress reports, and ongoing grantee monitoring. RHYMIS data (hundreds of variables on every youth in BCP and TLP) are used by FYSB to report the number of youths receiving services. Grantees are required to submit semi-annual reports to their federal project officers for the purposes of identifying successes and challenges in administering the grant. The progress reports, along with legislatively-required, on-site monitoring of programs, identify the training and assistance needs of grantees in providing services to the RHY population. This information is used to adjust program priorities, allocate resources, and monitor the activities carried out by grantees. When on-site monitoring and audits determine that a grantee is not meeting federal standards or significantly under performing, corrective action is taken; in some instances a successor grantee is chosen to replace the underperforming grantee. A demonstration project which began with FYSB and Region 10 now consists of a team of youth specialists from eight regions who have begun to review grantee-level RHYMIS data useful for improving safety, youth development (community service participation), and cost efficiency. RHYMIS data for all three programs has been cleaned, packaged, and distributed for use by regional offices, technical assistance providers, and grantees (on request) since FY 2002 .

Evidence: 3.1.1-NRS Statistics (examples from website) 3.1.2-RHYMIS Fact Sheet, Instrument, examples of data and variables (TLP example) 3.1.3-Monitoring manuals and Reports with Corrective Action, (See 3.CO2) 3.1.4-Steve Ice, Special Populations Team Lead, (2006), "Draft Analysis Using RHYMIS 2005 Data to provide project oversight to Region 10 FYSB programs," ACF Seattle Regional Office. 3.1.5-Sample letter. 3.1.6-Sample FYSB/regional office emails (See 4.1) 3.1.7-E-mail from FYSB Deputy Associate Commissioner to all ACF Regional Administrators sharing the Region 10 materials.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The RHY programs and structures are designed to ensure that Federal managers and program partners are held accountable for the efficiency and effectiveness of the program. The Assistant Secretary for Children and Families is responsible for three RHY measures in his performance contract with the Secretary of HHS. The FYSB Associate Commissioner and other ACF managers, including ten Regional Administrators, are held accountable for their performance through their Employee Performance Contracts for cost, schedule, and performance results. All of these plans include RHY measures. Federal Project Officers have to be certified by the HHS and/or government-wide contracting agency (e.g., Govworks) and are held accountable for cost, schedule, and performance of contracts and grants. Grantees are required to provide 10% of project costs and document this fact. They submit progress reports and semi-annual SF-269s (Financial Status Report) detailing activities and expenditures for a budget period. The project and grants office reviews the SF-269 for appropriate of use of funds. (See 3.6) Regional staff and social/youth services experts (peer monitors) conduct site visits and extensive audits of each grantee every three years. The evidence binder contains examples of stringent actions taken with nonperforming grantees. (See 3.CO2) Contractors are required to submit monthly invoices of services provided under the contract, which carefully examined by FYSB staff trained as contract officers' representatives (COTR) to ensure that expenditures are aligned with the contractual purpose. Staff and managers are held accountable for their oversight of grants and contracts in their annual performance plans.

Evidence: 3.2.1-ACF Assistant Secretary's EPMS plan 3.2.2-FYSB Associate Commissioner's EPMS plan 3.2.3-Regional Administrator's EPMS plan. 3.2.4-Cost sharing requirement from program announcements 3.2.5-Grantees semi-annual financial (SF-269) and progress reports (See 3.6) 3.2.6-Logic Models for RHY goals not achieved (See 2.7) 3.2.7-Sample (redacted) list of Region I grantees monitored during FY 2005 (See 3.CO2) 3.2.8-Sample (redacted) summary of recent Region I monitoring actions. (See 3.CO2) 3.2.9-Sample (redacted) results and report (See 3.CO2) 3.2.10-Sample (redacted) response to grantee response (See 3.CO2) 3.2.11-Sample (redacted) recent activity in Region II (See 3.CO2) 3.2.12-Monitoring Manual (Table of contents and introduction; original is 61 pages); and Instrument (Table of Contents, original is approx. 200 pages) (See 3.CO2)

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: With very minor local exceptions (which are highly regulated) national RHY program funds are obligated on schedule and are monitored to ensure they are used for intended purposes. A budget narrative is required as part of the funding requirements for all grantees. Prior approval of budget revisions is required in accordance with grants management policy. Grantee funds are obligated through the GATES system and monitored through the semi-annual SF-269 financial status report. Award recipients typically spend funds during the single fiscal year. FYSB grantees carry limited amounts of unobligated funds. However, when it is necessary to carry over funds, a carryover request has to be approved by FYSB, and funds can only be used for the purpose of completing unfinished prior approved projects. Federal staff review contractor expenditures on a monthly basis and approve or disapprove reimbursement items, according to their appropriateness under the statement of work. In general, non-financial reports in ACF and HHS grants programs (e.g., in FYSB's RHY programs) are required semiannually. Financial reports are semi annual. Many of the remaining grantees also automatically submit to independent audits as a condition of their organizational charters or affiliations (Salvation Army, Catholic Social Services), requirements of their boards of directors, other non-Federal funding sources, or their non-profit status.

Evidence: 3.3.1-GATES Detail CAN Activity Report for 2005 (note zero balance) 3.3.2-SF-269 Financial Status Report (See 3.6)

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: RHY competitively outsources evaluation, technical assistance, public education, logistics and RHYMIS. Federal staff review contractor proposed budgets to identify excessive and inappropriate costs prior to award. Contractors are consistently evaluated on their performance in several areas, including cost control. FYSB awards contracts in full compliance with Federal Acquisition Regulations Part 92 (award of government contracts through full and open competition). Generally, FYSB contracts are awarded on the basis of "best value to the government," not strictly on price, due to the indefinite nature of most of the work. Local contracts financed by the grantee must be for allowable costs (if FYSB-funded) and provide full and open competition under Part 92. Contracts are reviewed by FYSB for such compliance. Over the past several years, reengineering and continuous improvement of RHYMIS design, content and functionality have resulted in much more accurate national data and new data elements of significant interest, without increases in collection burden. The increased rate of grantee compliance with RHYMIS data submission is up to 99% in each of FY 03-05. It was 97% in FY 02, compared to 95% in FY 2001, 84% in FY 00 and 74% in FY 99. Prior to FY 99 it was as low as 45% in some quarters. This volume of reliable data has enabled FYSB to track a large set of indicators (e.g., more than a hundred variables for each youth). Many of these are appropriate for performance measurement and strategic outcome budget reporting, but they also guide policy, oversight, communications and technical assistance, allowing resources to be deployed most efficiently. Regional staff conduct on-site audits of RHY agencies on a regular schedule set by the Statute. (3.CO2) The following efficiency measure is part of the Performance Plan: Increase funding efficiency by increasing the percent/number of youth who complete the transitional living program (TLP) by graduating or who leave ahead of schedule based upon an opportunity. FY 2010 Long Term Target 55% FY 2007 Target 49.6% FY 2006 Actual Dec 06 FY 2006 Target 47.6% FY 2005 Actual 47.9% FY 2005 Target 45.6% FY 2004 Actual 45.6% FY 2004 Target 43.6% FY 2002-FY 2003 Actual/Baseline 42.6% This measure accounts for efficiency in the TLP because youth who complete the programs have best outcomes, as demonstrated by the correlation of successful completion with positive educational and employment outcomes. RHYMIS data also demonstrate that youth who drop out of a TLP are more likely to return to the streets, which FYSB classifies as a dangerous exit. In FY 2005, 55,067 youth received services from BCP at the cost in federal grants of about $796 per youth. The approximate cost for 3,048 youth in the TLP program was $11,888 (including all services and full-time housing). The SOP served 514,657 youth during FY 2005 at the cost of approximately $27 per youth contacted. The unit costs discussed in the preceding paragraph provide a basis for comparison with other service programs, but cannot be used to illustrate efficiency trends. They will vary widely with changes in the population of runaways, homeless and street youth. Given mostly level appropriations, an increase in the number of youth in crisis would look like fewer dollars spent per youth, but would not indicate increased efficiency. At the level of the individual grantee, costs per youth must be assessed in the light of varying service models. For example, in the BCP a drop-in center with overnight facilities may serve hundreds of youth during the same time that a temporary shelter with extensive services houses far fewer residents. Also, RHY youth have complex and resource-intensive problems, and, as in many social-service programs, cost per unit data has little to no value.

Evidence: 3.4.1-GATES Detail CAN Activity Report for 2005 (note zero balance)(see 3.31) 3.4.2-SF-269 Financial Status Report (See 3.6) 3.4.3-Sample RFP (pages 1-4) and government cost estimate (base year example) 3.4.4-Program Announcements (See 3.3) and grants competition process (See 3.CO1) 3.4.5-RHYMIS instrument and data elements (See 3.1) 3.4.6-Monitoring Manual and sample report (See 3.CO2) 3.4.7-2007 Performance Budget (see 2.1)

Does the program collaborate and coordinate effectively with related programs?

Explanation: RHY program administrators understand that the program's success relies on effective collaboration and coordination with other relevant organizations and ensure that it happens. FYSB's National Runaway Switchboard (NRS) coordinates with local BCP and TLPs, whether or not they are funded by RHY. All of these coordinate extensively with each other and with other pertinent service providers (e.g., education, housing, health, recreation, etc.) NRS operates the HomeFree program with Greyhound Bus (free tickets for runaways to return home). With FYSB support, NRS recently developed a MOU with the FBI to provide reunification services for minors lured away from home by internet predators. International agencies have recognized the national RHY system in the US as an exemplar of services to youth at risk. (See 4.4) because of the integration of the NRS hotline, street outreach programs, shelters, transitional programs (the latter three located throughout the U.S. and the Territories), and RHY support services at the national and regional levels. FYSB's RHY programs collaborate with ACF's Domestic Violence Prevention programs (also in FYSB), ACF's Children's Bureau, Housing and Urban Development (HUD), U.S. Depts of Agriculture, Education, Justice, Labor, Transportation, Corporation for National and Community Service, and the Office of Refugee Resettlement's trafficking initiative. Coordination with Domestic Violence Prevention includes skill sharing and joint training in common service elements between runaway shelters and domestic violence shelters. Additionally, during Hurricane Rita's approach towards Texas, RHY's NRS opened its facility to and shared telecommunications and database access resources with the Domestic Violence Hotline staff from Austin, TX, displaced by possible flooding and wind damage. NRS staff also helped research which DV sites were impacted and required relocation of shelter residents. The thousands of calls over the several days were handled seamlessly. FYSB's National Youth Summits are multi-agency conferences that address a broad spectrum of issues surrounding youth, not just risk reduction. Youth serve on planning committees for the National Youth Summits. FYSB's Positive Youth Development State and Local Collaboration Demonstration Project focuses on establishing partnerships and collaborative efforts among Federal, State and local agencies to improve conditions of runaway and homeless youth and other youth within the community. Projects strengthened individual positive youth development efforts. FYSB works with the Department of Juvenile Justice as a member of the Interagency Council on Juvenile Justice and Delinquency Prevention. FYSB collaborates and coordinates annually with the ACF Children's Bureau to host the national Pathways to Adulthood Conference which brings the transitional living and independent living programs (for older foster youth) together. FYSB's TLPs tend to be the mentors of the ILPs, which are new to the scene of youth development. Local programs supported by FYSB coordinate with education, law enforcement, housing, health, recreation, social services, and other resources, public and private. Regional Training and Technical Assistance Providers work with grantees to build capacity and develop referral resources in local communities. FYSB grants are able to partially cover costs of services. In many ways, the federal grant is more of a "door opener" and credentialing resource than a major support for operations. The FYSB "network of support" is also a significant asset for the field, including non-grantees. During FY 2003, FYSB loaned senior staff (full time/full year) and donated contract services (database systems development) to the White House Task Force for Disadvantaged Youth.

Evidence: 3.5.1-NRS web page (See 3.CO3) 3.5.2-Materials from UK services agency (See 4.4) 3.5.3-Training materials from Mid-Atlantic Network for Youth (MANY, a T&TA provider) on programmatic coordination between RHY and Domestic Violence Prevention. 3.5.4-Description from National Domestic Violence Hotline of NDVH Collaboration with RHY/NRS in Chicago during Hurricane Rita 3.5.5-4 National Youth Summits have brought together youth, community leaders, policy makers and practitioners to explore how to further the field of "positive youth development." (See 1.2) 3.5.6-FYSB/Children's Bureau Pathways to Adulthood conference materials 3.5.7-FYSB 10 training and technical assistance grantees provide youth related services and positive youth development to all RHY programs, including advising them how to coordinate local resources. (See 1.5) 3.5.8-Here is a fairly typical pattern of "braided funding" and coordination from Juneau Youth Services, a TLP in Alaska: 3.5.9-Funding sources have for TLP besides FYSB: "Other direct funding is from Medicaid, AK Housing Finance Corp Special Needs Housing Grant Program (capital), and possibly Denali Commission (capital). 3.5.10-"Indirect resources with whom we have MOAs include 2 WIA youth programs, ILP, McKinney-Vento, HUD Continuum of Care, SAMHSA Circle of Care and Teen Health funds, Native Organization funds for health and dental services, and other Medicaid through DD waiver services and dental services."

Does the program use strong financial management practices?

Explanation: In addition to the semi-annual program reports discussed in 3.2 and 3.3, FYSB's on-site monitoring system requires program and fiscal reviews of grantees every three years or on an as needed basis. (See 3. CO2) Prior approval of budget revisions is required, in accordance with HHS and ACF grants management policy, and is arranged through FYSB program specialists working with the Office of Grants Management. Allowable costs for expenditures of FYSB funds are detailed in the Project Summary portion of the Program Announcement, and programs are evaluated on meeting the allowable costs requirements. These costs include personnel, fringe benefits, travel, equipment, supplies, contractual, and indirect charges. In the "personnel" category, the employee's position and time commitment to the program (in months) must be outlined. Personnel costs do not include hiring of consultants, the personnel costs of delegate agencies, or other projects or businesses financed by the applicant. Grantees must report and justify program income and non-Federal resources, along with assurances such as non-construction. The agency's fiscal monitoring of grant awards is conducted through the Grants Management Staff (independent of FYSB), which tracks awards and obligations. Federal project officers review and approve quarterly reports, application renewals, and final reports. Signatory authority resides above FYSB and is exercised by the Commissioner of the Administration on Children, Youth, and Families, a Presidentially-appointed, Senate-confirmed senior official. Contractors are required to submit monthly, itemized invoices of services provided under the contract. Trained and certified federal project officers with extensive program knowledge guide projects, review expenditures for contracts on a monthly basis, and approve or disapprove reimbursement items. FYSB project officers must be certified by the Contracting Office or Govworks to manage contracts and are held accountable for cost, schedule, and performance. All FYSB project officers have received training and certification within the last three years. Independent financial management audits and inspections by the HHS Office of Inspector General (OIG) have identified no fraud, abuse or material internal control weaknesses in RHY programs. No evidence has suggested or necessitated such audits by the IG. Limited problems identified through monitoring at individual grantee sites are addressed aggressively by regional financial management and youth specialists, sometimes through the mandatory replacement of a grantee. All agencies in HHS have received a "clean" independent audit report for the past seven years.

Evidence: 3.6.1-RHY Monitoring Plan and materials (See 3.CO2) 3.6.2-program announcements (See 1.3) 3.6.3-Financial and Progress Reports 3.6.4-"Primary Sources of ACF Grants Management Guidance" 3.6.5-Departmental Financial Statement Audit: "For the seventh consecutive year, the Department received a "clean" opinion on its financial statements, meaning that the statements were reliable and fairly presented. However, the auditors noted two material weaknesses. First, the lack of an [HHS-wide] integrated financial management system and weaknesses in internal controls [i.e., oversight of State Medicaid and SCHIP] made it difficult for the Department to prepare timely and accurate financial statements. Second, CMS [Centers for Medicare and Medicaid Services] lacked comprehensive controls over the payment of Medicare managed care benefits." 3.6.6-- HHS Office of the Inspector General, "Semi-annual Report for the 6-month period ending March 31, 2006".

Has the program taken meaningful steps to address its management deficiencies?

Explanation: FYSB addresses management deficiencies at the local level through the use of its on site monitoring system. RHY program staff and peer monitors identify issues through interviews, audits, and observations. When deficiencies are discovered, the RHY program staff develops a plan for corrective action and provides appropriate training and technical assistance to correct the deficiency. If deficiencies are improved in a timely manner (as established by the grants management office), steps are taken to terminate funding for programs that are out of compliance with legislation/regulations. (See 3.CO2) The lack of accurate program data at the national level, a significant deficiency, was resolved by reengineering RHYMIS. Since then, data collection and availability has improved significantly.

Evidence: 3.7.1-Summary of Region I Monitoring Actions. (See 3.CO2) 3.7.2-Redacted monitoring report and corrective action table. (See 3.CO2) 3.7.3-RHY Monitoring Manual and Instrument, U.S. Department of Health and Human Services. (See 3.CO2)

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All grants are awarded through rigorous competition according to standards published in the program announcement. Program announcements are published according to standard HHS practices. The announcement itself goes through several levels of review outside within ACF and HHS. The criteria for application reviews are based on an OMB-approved "Uniform Program Description." Grantees must match 10% of the project costs. In-kind matching is permitted, but must follow the criteria spelled out in the program announcement and "terms and conditions" of funding. The Grant Review Handbook clearly delineates the process, identifies the responsibilities of each participant, and assists reviewers in being accurate and impartial. A grant award panel consists of a chair person and 3-4 reviewers, all chosen carefully by FYSB for their expertise in the field, as well as their ability to assess proposal quality critically and objectively. Grant review panelists are independent of FYSB and must provide explicit written assurance that they have no conflicts of interest or other disqualifying factors. Panels are led by chairs who ensure consistency and defensibility of scoring award recommendations. The scoring approach is systematic, explicit, and applied consistently to all applications. In-depth training in evaluation and scoring methods is provided to panelists. Final applications are reviewed and approved by Federal subject and project area managers as an assurance that all the federal guidelines are met. Final award decisions are made by the Commissioner, Administration on Children, Youth and Families, after a review and recommendations by federal staff. FYSB awards contracts in full compliance with Federal Acquisition Regulations Part 92 and adheres to all procedures for awarding government contracts through full and open competition. Contracts financed by the grantee must be for allowable costs (if Federally-funded) and provide full and open competition under Part 92. Contracts are reviewed by FYSB for such compliance. (See 3.4)

Evidence: 3.CO1.1-Program Announcements (See 1.3) 3.CO1.2-Training materials for (1) review panelists and (2) panel chairs 3.CO1.3-Sample RFP and Government Cost estimate (See 3.4)

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Continuous assessment of grantees is based on grantee monitoring reports, administrative review forms, and site visit protocol documents. Semi-annual progress reports and financial status reports are submitted to the Federal Project Officer and the Grants Management Office. Federal staff in the central office and in every region serve as project officers and receive data on grantees' activity through semi-annual progress reports. Furthermore, each program is monitored on site by federal project officers at least once every three years, or on an as needed basis. Fiscal oversight is monitored by Federal Project Officers and the Grants Management Staff. Federal regional project officers prepare an administrative review form that identifies whether the grantee is a continuing applicant, its monitoring status, and any material weaknesses. RHYMIS data helps identify strengths and weaknesses prior to on site reviews. ACF regional office youth specialists are developing procedures for using RHYMIS data in the oversight and program improvement process. (See 3.1) The evidence table provides an example of a stringent action taken against a Region I grantee following site visit findings.

Evidence: 3.CO2.1-Sample (redacted) list of Region I grantees monitored during FY 2005 3.CO2.2-Sample (redacted) summary of recent Region I monitoring actions. 3.CO2.3-Sample (redacted) results and report 3.CO2.4-Sample (redacted) response to grantee response 3.CO2.5-Sample (redacted) recent activity in Region II 3.CO2.6-RHY Monitoring Manual,, U.S. Department of Health and Human Services. (Table of contents and introduction; original is 61 pages); and Instrument (Table of Contents, original is approx. 200 pages) 3.CO2.7-RHYMIS Instrument and RHYMIS oversight report with sample letter (See 3.1)

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: RHYMIS enables FYSB to track an enormous set of indicators (more than a hundred variables for each BCP or TLP youth) that inform FYSB's strategic outcome budget reporting, policy guidance, communications, and technical assistance. RHYMIS data is reported by grantees every six months, and data is collected and compiled in a biannual Report to Congress. Annual performance data is summarized, incorporated into the Performance Budget, and made available on ACF's web site. RHYMIS data reports are available to the public on the ACF extranet. FYSB's National Clearinghouse on Families and Youth provides information dissemination services to the public, to researchers, and to practitioners. The National Runaway Switchboard collects extensive statistics on calls and referrals, which are included in the Report to Congress and posted on the Switchboard's website.

Evidence: 3.CO3.1-Agency performance reports and Reports to Congress are published on the ACF website. http://www.acf.hhs.gov/programs/olab/budget/index.html 3.CO3.2-Report to Congress (Cover page and Table of Contents) 3.CO3.3-National Clearinghouse for Youth website www.ncfy.com 3.CO3.4-RHYMIS data on performance by state, region, agency (available to the public at https://extranet.acf.hhs.gov/rhymis) 3.CO3.5-FYSB website http://www.acf.hhs.gov/programs/fysb Home page National Runaway Switchboard: www.Nrscrisisline.org 1-800-RUNAWAY

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: 1. Increase the proportion of youth living in safe and appropriate settings after exiting ACF-funded TLP services. (revised). FY 2010 Long Term Target: 85%; FY 2009 Target: 85%; FY 2008 Target: 85%; FY 2007 Target: 84%; FY 2006 Target: 83%; FY 2005 Actual: 82%; FY 2005 Target: 80%; FY 2004 Actual: 78%; FY 2004 Target: 80%; FY 2003 Actual: 78%; FY 2003 Target: 80%; FY 2002-2003 Baseline: 79%. Since RHY focused improvement efforts on TLP, in which safe exit rates have not been as high as BCP, the safe exit rate between FY 2004 and FY 2005 increased by four percent, exceeding the target by two percent. (See NOTE 1) BCP safety rates have been at or above 90% for the last four years, meaning that most youth exit BCPs into safe and appropriate settings. During their residence at the shelters, they are safe in 99.0% of cases. Few incidents of violence, serious injury, or other harm are reported in either program, which collectively serve about 60,000 high risk youth each year. This record is far better than is prevalent in jails, detention centers, boot camps, or other "alternative" treatments of at-risk youth. (Further discussion under 4.2) 2. Increase by 2% annually the proportion of youth who are prevented from running away after receiving BCP in-home/off-site services. (new) FY 2010 Long Term Target: FY 2006 + 8%; FY 2009 Target: FY 2008 + 2%; FY 2008 Target: FY 2007 + 2%; FY 2007 Target: FY 2006 + 2%; FY 2006 Baseline: Dec 2006. This new measure focuses on BCP prevention services. Again, because it is new, results are not yet available for evaluation. 3. Increase the proportion of TLP youth who are engaged in community service and service learning activities while in the program. (revised) FY 2010 Long Term Target: 35%; FY 2009 Target: 35%; FY 2008 Target: 34%; FY 2007 Target: 33%; FY 2006 Target: 32%; FY 2005 Actual: 31%; FY 2005 Target: 30%; FY 2004 Actual: 27%; FY 2004 Target: 30%; FY 2003 Actual: 29.8%; FY 2002 Actual/Baseline: 27.5%. Due to intentional efforts by FYSB, (See NOTE 2) this measure exceeded its FY 2005 target with an increase of 3 percent over FY 2004. The appropriateness and ambition of the long term and annual goals is explained in section 2.2 and 2.4.

Evidence: NOTE 1: Focus on improving TLP safe exit rate began upon analysis of the FY 2002-2003 RHYMIS data set. (See also efficiency measure) During FY 2003 and 2004, FYSB identified possible causes of unsafe exits among youth in the TLP (which had a 77.7% safe exit rate in FY 2004). The research found unsafe exits to be most prevalent among youth who dropped out of the TLP and lacked exit counseling and follow-up services. It was also correlated with certain characteristics of individual youth (particularly those with mental health and/or substance abuse problems). The analyses and a set of recommendations were shared with regional offices, grantees and technical assistance providers and presented at national and regional conferences during FY 2004 and FY 2005. Regional teams are using the data to focus on performance goal-related issues and posters for youth centers have been distributed with a message to youth (and staff) to "make a plan" before exiting or quitting program services. NOTE 2: Similar efforts to promote CSL have been part of FYSB's positive youth development approach for many years. FYSB has also highlighted the correlation of CLS with safe exits and program completions. 4.1.1-2007 Performance Budget (see 2.1) 4.1.2-Pergamit, M., Ernst, M., (2005), "Exploratory Analysis of the Basic Center Program Data in the RHYMIS Database," National Opinion Research Center/ University of Chicago. (Table of Contents, Introduction) 4.1.3-Trivits, L., (2005), "Preliminary Data Analysis of Basic Center RHYMIS Data," ACF Office of Planning, Research and Evaluation. (Introduction) 4.1.4-FYSB and ACF Region 10, (2004). "Post Service Safety in the Transitional Living Program" (DRAFT analyses and recommendations). 4.1.5-Conference presentation by FYSB of analyses and discussion of risk factors (Powerpoint) 4.1.6-Agenda from sample events where RHY performance goal issues (especially safe exits) were a focal topic: National Network for Youth Symposium, National T&TA meeting, Region III meeting. 4.1.7-Sample technical assistance materials developed during FY 2004-2006 stressing importance of safe exits, after care and exit planning 4.1.8-Sample FYSB/regional office team emails discussing data, performance, and oversight

Does the program (including program partners) achieve its annual performance goals?

Explanation: See tables in 4.1 for long term goals achieved and annual increments. Annual outcome goals for safety were surpassed by two percent and CSL by one percent. Both represented significant increases over previous year actuals of 4 and 3 percent respectively. Large extent is given because the program has achieved annual performance goals in recent years, with a few exceptions. NOTE: revisions and additions are explained in 2.1. and 2.2, which also address their appropriateness and degree of ambition. Goal 1 has been revised from: "Increase the proportion of youth living in safe and appropriate settings after exiting ACF-funded services." The Community Service Learning goal has been revised from "Increase the number of RHY youth who are engaged in community service and service learning activities while in the program." The BCP "prevention goal" is new, with data expected to be available in FY 2007, but possibly for the second half of FY 2006.

Evidence: 4.2.1-2007 Performance Budget (see 2.1) 4.2.2-Associate Commissioner Dear Colleague Letter

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Improvements in the efficiency measure have been consistent, substantial, and ahead of schedule since FY 2004, as explained in 3.4. Targets for the program's efficiency measure have consistently been met or surpassed: 1. Increase funding efficiency by increasing the percent/number of youth who complete the transitional living program (TLP) by graduating leaving ahead of schedule to pursue a positive opportunity. FY 2010 Long Term Target 55% FY 2007 Target 49.6% FY 2006 Actual Dec 06 FY 2006 Target 47.6% FY 2005 Actual 47.9% FY 2005 Target 45.6% FY 2004 Actual 45.6% FY 2004 Target 43.6% FY 2002-2003 Actual/Baseline 42.6% This measure accounts for efficiency in TLP since youth who graduate programs yield the best results. TLP RHYMIS data shows that youth who remain in the program for at least 60 days are more likely to demonstrate progress in educational and employment status, and the effect increases with length of stay. Dropouts are also more likely to return to street life. Unit costs cannot be used to illustrate efficiency trends, as they vary widely based on changes in the population of runaways, homeless and street youth. For example, given the history of fairly level appropriations, a significant increase in the number of youth in crisis would calculate as fewer dollars spent per youth, but would not indicate increased efficiency. For this reason, FYSB emphasizes the "program completion" factor.(See calculation at right.) Given the level funding of RHY programs over many years (See Budget document under 2.7), and the steady or improving performance, despite inflation and the challenges of serving target youth, indicates effective and efficient oversight and management. The efficiency and effectiveness of critical data collection has also improved dramatically as a result of the strategic reengineering of the RHYMIS instrument and software during FY 2001. Technical support and assistance is provided when grantees need help, and follow-up is aggressive when they are late in reporting. Grantee RHYMIS data submission has remained at above 99% since it was reengineered, while previously having been as low as 45%. Even the impact of the hurricanes in FY 2005, which knocked out a number of RHY centers, did not noticeably reduce the data transfer response level. Additionally, the program has continued to meet annual goals while funded on a flat level.

Evidence: 4.3.1-FY 2007 Performance plan (see 2.1) 4.3.2-Actual cost savings: bearing in mind the discussion at left that it would be perverse to claim improved efficiency if the number of endangered youth were to increase during a period of level funding, FYSB can still examine efficiencies in terms of the value of "residential slots well and appropriately utilized." 4.3.3-Because of "up and down" changes in caseloads, it is illogical to emphasize year to year dollar differentials. The growing percentages each year tell the story. However, for the sake of illustration, the national "efficient investment" during FY 2005 was $15,870,480 (1335 X $11,888) or a unit cost per completion (positive outcome) of $24,818. 4.3.4-See 4.1 and 4.2 for discussion of ACF's successful efforts during FY 2004 and 2005 to affect this measurement factor. 4.3.5-BCP achieved a cost of $888 per safe exit (compared to $796 per every youth who exited regardless of destination.) TLP spent $14,528 per safe or appropriate exit (i.e., excluding shelters) compared to $11,888 on each youth exiting the program.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Although it is difficult to find appropriate comparisons for RHY, RHY indicators fare well against any number of relevant comparison. TLP performed better than HUD's Homeless Assistance Grants program on similar measures. The HUD program provides transitional housing and employment services for homeless adults with economic, housing, behavioral and/or psychological problems, as TLP provides for youth. Many RHY grantees also receive HUD funding. In 2004, 59% of beneficiaries moved from transitional directly into permanent housing. In 2004, 65% of TLP youth entered private residences (their own apartments, homes of relatives, parents or guardians), with another 6% going to a residential program (not a shelter) after leaving. For youth in a TLP for at least 60 days, 73% went directly to a private residence and another 15% went into a residential program. RHY also operates housing facilities and group residences with much better conditions than the juvenile justice system provides. A 1991 USDOJ study of conditions of confinement in U.S. juvenile detention and correctional facilities found pervasive institutional crowding with more than 75% housed in facilities that violated one or more standards related to living space. Crowding was associated with higher rates of violence, suicidal behavior, and use of short term isolation. Over 30,000 staff or youth were injured by residents. There were 17,600 acts of suicidal behavior and even more incidents requiring emergency care. 9,700 juveniles escaped from custody. Since then, GAO and press reports suggest that this situation remains troubling. BCP and TLP house youth with some similar risk factors, but rarely experience similar problems, and adhere to staffing ratios, limits on numbers of youth per site, and other quality factors. The TLP drop out rate cannot be compared with "escapes from custody," since TLP is not a detention program and youth are free to leave any time. During their time in RHY facilities, youth are rarely endangered. FYSB's extensive and integrated system of shelters, transitional programs, street outreach, a hotline and other services for runaways and homeless youth is regarded as a model by advocates and policy makers in the United Kingdom. During the last several years, FYSB hosted fact-finding delegations from China, Brazil, Russia, Hungary, Czech Republic, Mexico, Great Britain (an earlier visit), and West Bank, each seeking guidance in setting up RHY-type programs. BCP has had an 89-90% "safe and appropriate exit" rate for several years, despite the fact that RHY youth have complex problems and are often difficult to serve. (More than 40 % of TLP youth report mental health issues, many critical. Even more have substance abuse problems.)

Evidence: 4.4.1-Children's Society (UK) invitation to FYSB Director of Youth Programs (20 September, 2005): 4.4.2-Children's Society (UK) Press Release (October 10, 2005): 4.4.3-National Runaway Switchboard Newsletter, March-April, 2006 (pages 1 and 2) 4.4.7-Pergamit, M., Ernst, M., (2005), "Exploratory Analysis of the Basic Center Program Data in the RHYMIS Database," National Opinion Research Center/ University of Chicago. (Table of Contents, Introduction) (See 2.6) 4.4.8-Trivits, L., (2005), "Preliminary Data Analysis of Basic Center RHYMIS Data," ACF Office of Planning, Research and Evaluation. (Introduction) (See 2.6) 4.4.9-FYSB and ACF Region 10, (2004). "Post Service Safety in the Transitional Living Program" (analyses and recommendations). 4.4.10-FYSB Conference presentation of analyses and discussion of risk factors. (Powerpoint) (See 4.1) 4.4.11-HUD PART http://www.whitehouse.gov/omb/expectmore/detail.10001234.2005.html (See 2.2.2) 4.4.12-Abt Associates, Inc. CONDITIONS OF CONFINEMENT IN JUVENILE DETENTION AND CORRECTIONAL FACILITIES: [UNITED STATES], 1991 [Computer file]. Conducted by Abt Associates, Inc. ICPSR edition. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [producer and distributor], 1996. 4.4.13-Government Accountability Office, "Foster Care: HHS Should Ensure That Juvenile Justice Placements Are Reviewed," HEHS-00-42, June 9, 2000 and "Noncriminal Juveniles: Detentions Have Been Reduced but Better Monitoring Is Needed," T-GGD-91-30, May 22, 1991

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: There is considerable independent research on RHY services and program that indicates the programs have positive effects; however, the picture painted by the research is incomplete. The statutory mission of RHY is immediate: remedy the endangerment and urgent needs of RHY youth through outreach, temporary shelters, developmental services and opportunities. Rigorous and independent national and regional evaluations have shown RHY as effective in achieving its purpose, not only while youth are in programs but also over longer terms. Six months is a more significant term in a youth's life than an adult's, and status during tenure and at discharge are directly relevant to the Statute's most urgent objectives. FYSB and contractors regularly analyze annual and multi-year data sets with over 100 variables for each youth, the largest scope available. TLP records cover periods of up to 36 months. RHY does not maintain contact with beneficiaries (e.g., to mail monthly benefits), and youth are not required to report their progress. They are often untraceable because of protected identities (fleeing abuse). Given the detailed information needed to demonstrate results, some studies must use interviews, increasing costs. CSR found that successfully completing TLP correlated with self-sufficiency and independence. TLP youth, compared to a control group of similar at-risk youth, were more likely, 6 months after completing TLP, to be employed, attending school, in college, and saving money. Westat found that 88% of BCP youth reported being very or somewhat satisfied with services they received. BCP youth rated BCP services as very or somewhat helpful in six-month follow up surveys: transportation: 100%; peer counseling: 93%; food: 89%; AIDS education: 89%; a place to live: 87%; help with life goals and plans: 87%; 6 months after their shelter stay, most youth positively rated their own situation in 12 problem areas. 77% of youth with physical abuse as a presenting problem were either "better" or "much better" at follow-up. 68% of youth with law enforcement problems were better or much better after services. A one year examination of data was attempted, but attrition made the results unreliable. A regional study, "Short Term Outcomes for Youth Receiving Runaway and Homeless Shelter Services," (Thompson, Pollio et al, 2002) conducted follow-up interviews with shelter youth in four Midwestern States. "Every outcome variable demonstrated improvement post intervention [school behavior, sexual activity, self esteem, current employment, etc.]" Youth in the BCP had better outcomes than at-risk peers 6-weeks after using shelter services. "The research provides evidence for the short-term effectiveness of crisis shelter services for runaway/homeless youth." The Pollio study looked at pre and post-service outcomes by interviewing youth and examined variables that may indicate best practices for the population. The study has a high degree of validity and reliability. The research, although admittedly imperfect, demonstrates positive results after intervention and is of high quality. Large extent is given based on the difficulty of conduction such evaluations for such a transient populations, and because a comprehensive longitudinal study of BCP outcomes has been approved and is in planning stages.

Evidence: These studies indicate that significant progress is made in helping youth make more positive choices (school, employment, risky behaviors), some of which may be day-to-day but nevertheless are vital foundations for long-term positive outcomes. RHY program structure, legislative requirements and other characteristics have changed little since the nineties, at the time of the earliest studies in this section. 4.5.2-Thompson, Pollio, Constantine, Reid, Von Nebbit, Washington University. St Louis, MO. (September, 2002), ""Short Term Outcomes for Youth Receiving Runaway and Homeless Shelter Services," (long and short term outcomes of youth in youth shelters in four Midwestern States, N = 261), Research on Social Work Practice: 12:5. P. 589-603 (4.5.3 deleted) - 4.5.4-CSR Inc. (1997), Evaluation of the Transitional Living program for Homeless Youth, N = 285; 175 TLP participants and 110 comparison youth. (Phase 2 Table of Contents and Executive Summary up through "Findings of the Impact Evaluation" and "Conclusions;" (The original is approx. 200 pages and has been transmitted to OMB examiner separately), U.S. Dept of Health and Human Services. 4.5.6- Westat, Inc., and Urban Institute (1997), National Evaluation of Runaway and Homeless Youth (BCP program). N = 969. (in depth interviews, comparisons calculated against Youth Risk Behavior Survey (CDC, 1995) and purposive samples of street youth in seven cities (Macro international, 1992). (Table of Contents and Executive Summary, pages 1 & 2 and "Status at 6 month follow-up," pages 15-20, U.S. Dept of Health and Human Services. (The original is approx. 200 pages and has been transmitted to OMB examiner separately). 4.5.7-Cauce, A.M., & Morgan, C.J. (1994), "Effectiveness of intensive case management for homeless adolescents: Results of a 3-month follow-up," Journal of Emotional and Behavioral Disorders, 2(4). 4.5.8 Findings of Runaway and Homeless Youth Act identifying urgent factors such as risk reduction, safety, insufficient basic resources and shelter, as well as developmental opportunities, as the purposes of the RHY program. The interlocking system that has evolved is based on the "long term strategy for responding to the needs of this population."