May 2004 Progress report on VFC - AFIX Initiative:  Natl Strategy to Improve Quality of Immunization Practices among VFC Providers

Background/Introduction:
In 1995, Congress directed the Centers for Disease Control and Prevention (CDC) to set guidelines for assessing coverage levels in all public clinics as part of the federal funding for immunization programs.  A continuous quality improvement strategy was developed to assess coverage levels and provide insight on how to improve coverage levels.  This strategy is known by the acronym “AFIX.”  The four components of “AFIX” are Assessment of immunization coverage levels, Feedback of the assessment findings to providers, Incentives to motivate and acknowledge change, and eXchange of information on best practices.  This strategy has been documented in the CDC Guide to Community Preventive Services as an effective method to improve and sustain immunization coverage levels.

Private provider participation in the VFC program has created a shift in the provisions of vaccine services over the last ten years from public health clinics to private provider offices.  Recognizing this shift, an objective related to the assessment of immunization coverage levels was included in the Healthy People 2010 goals.  The objective is to “increase the proportion of providers who have measured the vaccination coverage levels among children in their practice population within the last two years.”  The Healthy People 2010 report included a 1997 baseline measurement for providers that had assessed their coverage level in the last two years as 6% for private providers and 66% for public providers.  

In 2000, CDC/NIP launched the VFC-AFIX initiative which linked AFIX, the continuous quality improvement strategy, with the VFC program. The year 2003 represents the third year that the 61 eligible VFC grantees requested and received funding for the VFC-AFIX project.  The purpose of this project is to assess and improve immunization delivery practices at the provider level to assure that VFC eligible children are receiving quality services.


2003 NIP Programmatic Activities
A major focus of the VFC-AFIX project in 2003 was the redesign of the Clinical Assessment Software Application (CASA).  NIP has developed and implemented a usability study. This particular activity involved testing the aspects of the new software with six participants in a laboratory setting over several days.  Other CASA redesign related activities included the development, implementation and evaluation of a users’ survey and the provision of several different trainings for programmers assigned to the CASA redesign project.   CDC is also involved in the development of the functional specifications for this project. The current CASA redesign is estimated to be completed during the first quarter of 2005.

In addition, many other 2003 activities enhanced the VFC-AFIX project at the federal and grantee level such as:
* VFC-AFIX project staff coordinated, moderated or presented at the following workshop sessions at the 2003 National Immunization Conference: “Strategies for Evaluating the Impact of AFIX: Case Studies from Chicago and Indiana;” “Strategies for Enhancing and Leveraging AFIX in the Private Sector;” “Successful Strategies to Improve the Quality of Immunization Services in Private Practices;” “Integrating CASA, ACASA, AFIX and VFC: New Developments;” and “Building a Quality AFIX Program: Foundation to Finishing Touches”.
* VFC-AFIX project staff coordinated and facilitated discussions during quarterly conference calls between grantees and NIP staff regarding VFC-AFIX activities.
* VFC-AFIX staff was actively involved in developing a Health Insurance Portability and Accountability Act (HIPAA) “Dear Colleague” letter signed by the Director of NIP and sent to all Immunization Program Managers.  Additionally, a frequently asked questions (FAQ) fact sheet was developed as a companion piece to the “Dear Colleague” letter.  This document is posted on the Vaccines/AFIX website. (www.cdc.gov/vaccines/programs/afix ).
* VFC-AFIX staff revised the VFC-AFIX evaluation software for use in 2004.


2003 Grantee Programmatic Activities
Grantees actively improved VFC/AFIX operations.  This section of the report illustrates some of the VFC/AFIX activities conducted by the grantees in 2003: 
* The Arizona Immunization Program (AIP) offers a “Learn at Lunch” opportunity for medical office staff to be trained on immunization practices. The programs are a collaborative effort between The Arizona Partnership for Immunization (TAPI) and AIP office staff. Immunization experts provide tips on how to make provider offices more parent and immunization friendly. In addition, TAPI partnered with WIC to design and provide baby bibs to be given to parents at the six-month visit.
* The Florida Immunization Program conducted an immunization summit in 2003 to address improving Florida’s childhood immunization levels. Directors from the county health departments, WIC, and the Children’s Medical Services (CMS) participated. Based on this meeting, a statewide initiative was developed to include public and private providers with the goal of improving immunization coverage levels of two-year-old children. The initiative is called “85 by ‘05”, representing the goal of achieving 85% coverage of two-year-olds in Florida by 2005.
* Idaho immunization program staff conducted a random telephone survey to measure the overall satisfaction of providers with the immunization staff providing quality assurance visits and training. Overall, the results were positive. 
* Illinois collaborated with the Illinois Chapter of the American Academy of Pediatrics through a grant agreement to conduct VFC-AFIX visits within four geographical areas of need and provide physician education with the “Reaching Our Goals” curriculum. 
* Indiana promotes reminder/recall systems during all AFIX visits and continues to support and fund the use of autodialing equipment by public immunization providers. 
* In Iowa, an immunization assessment report of all the public clinic results including a ranked listing of clinic specific results was compiled and published. A copy is sent to each clinic site as well as the county board of health. In the private sector, a ranked listing of clinic results was also compiled, but a coded identifier was used instead of the provider’s name. Each provider receives a copy of the report. 
* In Kansas, the numbers of Standards for Pediatric Immunization Practices implemented by providers were documented during site visits, and six private providers were recognized at the annual state conference for Best Practice. 
* Missouri formed an immunization workgroup in July 2003, and this workgroup has been meeting monthly in an effort to standardize the VFC Site Visit and AFIX process.  The workgroup consists of central office staff, district staff and contract staff who either perform AFIX/VFC site visits or work with the reports and data generated by these site visits.  A policies and procedures manual is being developed and should be finalized by summer 2004.
* Oregon established valuable partnerships with the Oregon Health Science University to improve immunization practice in rural clinics and New York Academy of Medicine to determine how length of enrollment in managed care affects immunization rates.
* San Antonio awarded 32 certificates to deserving facilities that reached at least 80% immunization coverage for the 4:3:1:3:3 series. Twelve of these facilities are members of San Antonio’s “VFC STAR PROGRAM.”  They have been awarded a certificate and VFC Silver Star in recognition of exceeding the National Immunization Program (NIP) goal of 90%.  Three facilities achieved 100% for their 2003 immunization coverage level.  
* Within the Utah Immunization Program, the staff who conduct AFIX visits are called “Provider Relations” staff, which reinforces the idea of building relationships between the immunization program and the immunization providers. These staff also provide support and coordination of CASA/AFIX with Intermountain Healthcare (IHC).  Providers with good rates are recognized in the program newsletter, and information on CASA/AFIX has been provided to various provider groups.   
* Washington State continued collaborative efforts for supporting and implementing AFIX among public and private providers. Appropriate state organizations and stakeholders shared information at meetings and placed educational items on appropriate websites, including, (Washington Chapter of the American Academy of Pediatrics (WCAAP), Washington Academy of Family Practice (WAFP), managed care plans, and the Immunization Action Coalition of Washington (IACW).
* In 2003, the Wyoming Immunization Program established goals for private practice reviews throughout the state and created electronic spreadsheets for reviews that document immunization level trends in private providers’ offices. 


VFC AFIX FY 2003 Awards to Grantees
In 2003, a total of $15.4 million in VFC AFIX funds was awarded to grantees.  Approximately $850,000 of the total amount apportioned in 2003 was unobligated and reapportioned in 2004.  


Summary of Program Findings
The VFC/AFIX project has shown successful outcomes in the private sector.  At the State and local levels, grantees continued to measure changes in coverage levels both in the public and private sector.  Many grantees also focused on improving the implementation of the Incentives and exchange of information components of the AFIX strategy.  The 2003 data reported by the grantees reflects an increase in the number of site visits from 2002 to 2003.  The number of visits made to public and private providers increased by 2,247 from CY 2002 to CY 2003.  Many grantees report activities in CY 2003 promoting and working in collaboration with other organizations and individuals to improve both immunization coverage levels and participation in the VFC/AFIX initiative.  Some of these collaborations are scheduled to continue in 2004.  

20 grantees documented an increase in average public provider vaccination coverage levels.  21 grantees documented an increase in average private provider vaccination coverage levels.  

Despite significant success, challenges in fully implementing the VFC AFIX program remain.  The data suggest that selected grantees need continued detailed assistance/instruction on the use of the VFC|/AFIX evaluation software.  Additionally, selected grantees may need more individualized technical assistance on how to move this project from a conceptual basis to an active dynamic public health program.  


Analysis of 2003 Grantee Submitted Data 
CDC/NIP created the VFC/AFIX Evaluation Software as a tool for grantees to monitor their VFC/AFIX activities. This software is a Microsoft ACCESS database that can be used to store the VFC/AFIX site visit data in accordance with the report requirements due at the end of each calendar year. If this software is used by the grantees, then the aggregate data required for submittal to CDC/NIP can be automatically generated by the report options built into the software. The VFC/AFIX Evaluation Software was initially distributed to the grantees in 2002 and a revised copy was provided in 2003. 

Grantees are not required to use the VFC/AFIX Evaluation Software; however, they are advised to develop their own tracking system that would capture the same data fields if they choose not to use the software. Grantees were required to submit documentation of their 2003 VFC/AFIX activities as part of the Annual VFC Management Survey due March 1st, 2004. The Annual VFC Management Survey data were collected using a web-based reporting method. Of the 61 eligible grantees, 59 submitted data for the Annual VFC Management Survey.  

VFC/AFIX Staff
The number of full-time employees (FTEs) working on VFC/AFIX activities is tabulated in Table 1 below. In total, 202 FTEs are employed at the state level, and 147 FTEs are employed at the local level. Together, 349 FTEs are currently working on VFC/AFIX related activities across the country. 

As shown in Table 1, these numbers can be categorized by new and existing positions for calendar year 2003.  The grantee VFC/AFIX projects can be carried out at the state level or the local level by immunization program staff or by staff hired through contracts with outside agencies. 

Table 1. Number of FTEs Working on VFC/AFIX Project CY2003
Personnel
Program Staff*
Contract Staff**
Total

State
Local
State
Local
State
Local
New Positions
6
6
10
1
16
7
Existing Positions 
196
141
65
7
262
148
Total 
202
147
76
8
278
155
*State Program Staff: state-employed staff working on VFC/AFIX at the state immunization program level. Local Program Staff: local health department staff funded with federal funds who work on VFC/AFIX.
**State Contract Staff: VFC/AFIX staff hired by the state immunization programs through third party contracts with federal funds. Local Contract Staff: VFC/AFIX staff at local health departments hired through contracts with outside agencies using federal funds. 

Provider Information
Table 2 below includes the number of provider sites that received at least 1 visit during 2003 for each category of provider. Public providers are divided into three categories. The “Public” category includes local health departments and Indian Health Service clinics.  Community or Migrant Health Centers (C/MHC) are an individual category and the “Other Public” category captures all other facilities that are not included in the other two categories. “Private” represents all private providers that received at least 1 visit in 2003. The providers are categorized into “VFC-Enrolled sites” and “Non-VFC Enrolled sites.” Including both types of provider sites, a total of 23,609 provider sites received at least one visit in calendar year 2003. Of the 23,609 provider sites, 22,732 (96 %) were enrolled in the VFC program. 

The 22,732 VFC-Enrolled provider sites that were visited constitute 52% of the total 43,532 active VFC-Enrolled provider sites. Table 3 illustrates the percentage of visits for each site type (Public, C/MHC, Other Public and Private). 

Table 2. Number of Provider Sites Receiving at Least 1 visit, CY2003
Provider Information
Public
C/MHC*
Other Public
Subtotal All Public
Private
Total
VFC-Enrolled Provider Sites 
4,035
2,324
1,022
7,381
15,351
22,732
Non-VFC Provider Sites 




    877
877
Total 
4,035
2,324
1,022
7,381
16,228
23,609
*Community or Migrant Health Center
Table 3. Number and Percent of VFC-enrolled Provider Sites Receiving at Least 1 Visit, CY2003
Provider Information
Public
C/MHC
Other Public
Subtotal All Public
Private
Total
Number of VFC-enrolled provider sites
4,366
3,877
2,939
11,182
32,350
43,532
Number (Percent) of VFC-enrolled providers who received at least 1 visit during CY2002
   4,035 

92%
2,324

60%
1,022   

35%
7,381

66%
15,351

47%
22,732

52%


Site Visit Information
While 22,732 VFC-Enrolled provider sites were visited in 2003, the actual number of VFC Only, AFIX Only, VFC/AFIX combined and educational visits to a VFC-Enrolled provider site totaled 26,442. Table 4 below details the number of visits to VFC and Non-VFC provider sites by visit type and provider type. As the table shows 8,846 visits were conducted in public VFC Enrolled sites and 17,596 visits were conducted in private VFC Enrolled sites. In addition, 1,054 visits (AFIX Only and Educational) were conducted in Non-VFC Enrolled provider sites. 


Table 4.  Total number of visits by provider type, CY2003
Type of Visit
Public
VFC enrolled
Public Non-VFC
C/MHC VFC Enrolled
C/MHC Non-VFC
Other Public VFC enrolled
Other Public Non-VFC
Private VFC enrolled
Private Non-VFC
VFC Only*
1,492

763

515

5,622

AFIX Only**
949
35
187
4
36
0
1,453
103
VFC/AFIX Combined***
1,921

1,318

349

7,607

Educational
663
36
425
35
228
0
2,914
841
* VFC Only is defined as a visit to a VFC enrolled provider to ensure compliance with VFC program requirements. 
** AFIX Only is defined as a quality improvement strategy utilizing assessment of immunization records, feedback, incentive, and exchange of information through performance measurement, diagnosis of service delivery problems, and data feedback during a visit to a medical practice.  One AFIX visit should contain an assessment and a feedback component even though more than one physical visit to the provider site may be required to complete the assessment and the feedback sessions. 
*** A VFC/AFIX Combined site visit is defined as a visit to a VFC-Enrolled provider site which integrates the review to ensure compliance with VFC program requirements and immunization record assessment and feedback activities. 


Table 5 includes additional documentation regarding site visits in CY 2003. Repeat AFIX visits are a subset of the AFIX visits documented in Table 4, and they are included to illustrate the number of visits that occurred at a site that had previously received an AFIX visit between January 1, 2000, and December 31, 2002. The nature of AFIX as a continuous quality improvement strategy requires that provider sites are visited on more than one occasion to evaluate incremental progress. The number of repeat AFIX visits allows CDC/NIP to track the grantees progress in implementing this ongoing quality improvement strategy. The VFC Follow-Up visits describe the number of visits that occurred as a result of a problem and/or concern found at the initial VFC visit. The information in Tables 4 and 5 reveals that grantees are actively visiting provider sites, following up on problems identified in previous visits, providing education as well as service, and ultimately building relationships with staff.  

Table 5. Additional Visits, CY2003
Type of Visit
Public
VFC Enrolled
Public Non-VFC
C/MHC VFC Enrolled
C/MHC Non-VFC
Other Public VFC Enrolled
Other Public Non-VFC
Private VFC Enrolled
Private Non-VFC
Repeat AFIX*
1,404
2
426
2
122
1
1,532
19
Follow-up VFC**
584

320

59

1,967

*Repeat AFIX: the number of AFIX visits (includes AFIX only and Combined VFC/AFIX) from Table 4 that are repeat assessments (e.g. the provider received an assessment during a previous year).
**Follow-up VFC: the number of visits completed to evaluate provider response to previously identified problems found during the initial VFC site visit.  


As part of the annual grant application process, grantees are required to specify the proposed number of site visits to be conducted in the upcoming calendar year. For the 2003 grant applications, the planned number of site visits included three categories: VFC Only, AFIX Only and VFC/AFIX Combined. In an effort to examine the accuracy with which a grantee can estimate VFC/AFIX activities, the number of proposed site visits from the grant applications were compared with the actual number of site visits (for the categories of VFC Only, AFIX Only and VFC/AFIX Combined). Grantees were not able to conduct the total number of proposed site visits in CY 2003 (see Table 6); however, the grantees did exceed the number of visits completed in CY2002 by 2,247.

Table 6.  Proposed and Actual Number of Site Visits, CY2003. 
Type of Site Visit
Public Provider*
Private Provider
Total Visits

Proposed
Actual
Proposed
Actual
Proposed
Actual
VFC only
1,960
2,770
4,794
5,622
6,754
8,392
AFIX only
1,389
1,172
1,440
1,453
2,829
2,625
VFC/AFIX combined
7,038
3,588
9,066
7,607
16,104
11,195
Total visits
10,387
7,530
15,300
14,682
25,687
22,212
*Public provider includes Public, C/MHC, and Other Public.




Assessment Outcome Measures for Public Providers 
Of the 61 grantees that receive funds for VFC-AFIX activities, 55 submitted public provider data for the annual report (see Appendix A, Table 1). Of the 55 that submitted data, 44 use the CASA, Mini-CASA or equivalent method (designated as “CASA”). Three grantees use the Hybrid method, and 7 grantees use a combination of CASA and Hybrid (designated as “Both”). One grantee did not report this information. (The difference between the CASA and Hybrid methods is found in the results from the assessment.) Grantees that use CASA as the assessment method will receive an estimate immunization coverage level for each provider assessed. Grantees that use the Hybrid assessment method will receive a result indicating the provider is “above” or “below” a pre-determined threshold immunization coverage level. (See Appendix B for more in-depth explanation.) 

The majority of the grantees (24 of 55) assess who are children 24-35 months of age. Other age groups reported include 19-35 months (12 grantees), 12-23 months (2 grantees), and nine reported that they assess both ages 12-23 and 24-35 months. Seven grantees responded to this question with “other,” and one grantee did not report this information. 

For those grantees using CASA for the assessment, eight grantees reported provider vaccination coverage levels averaging 80% or higher. Sixteen grantees reported provider vaccination coverage levels averaging 70-79%.  Of the remaining results, 12 grantees indicated vaccination coverage levels averaging 60-69% and 14 indicated vaccination coverage levels below 60%. One grantee did not report this information. 

For those providers using the Hybrid Method, threshold levels of 70, 75, 80 and 90 percent were used. The most common threshold levels were 80 and 90 percent.  For those grantees that used the Hybrid methodology, only 1 grantee reported the number of providers above the selected threshold level as greater than 80% of the providers assessed. One grantee did not report its Hybrid method outcomes. 

Note: The above numbers are not mutually exclusive – meaning one grantee could have done some AFIX visits using the Hybrid method and some visits using the CASA, Mini-CASA, or Equivalent method. 

Assessment Outcome Measures for Private Providers
Of the 61 grantees that receive funds for VFC-AFIX activities, 58 submitted private provider data for the annual report (see Appendix A, Table 2). Of the 58 that submitted data, 46 use the CASA, Mini-CASA or Equivalent method (designated as “CASA”). Three grantees use the Hybrid method, and 7 grantees use a combination of CASA and Hybrid (designated as “Both”). Two grantees did not report this information.  

A large number of the grantees (25 of 58) assess children 24-35 months of age. Other age groups reported include 19-35 months (14 grantees), 12-23 months (2 grantees), and 9 grantees reported that they assess children ages 12-23 and 24-35 months of age. Seven grantees responded to this question with “other,” and 1 grantee did not report this information. 

For those grantees using CASA for the assessment, 12 grantees reported provider vaccination coverage levels averaging 80% or higher. Fifteen grantees reported provider vaccination coverage levels averaging 70-79%.  For the remainder, 12 grantees indicated vaccination coverage levels averaging 60-69%, and 14 indicated vaccination coverage levels below 60%.  

For those providers using the Hybrid method, threshold levels of 70, 75, 80 and 90 percent were used. The most common threshold levels was 80 percent.  Although no grantees using the Hybrid Method reported 80% of the providers assessed as above the designated threshold level, one grantee did report 79% of providers were above the designated threshold. One grantee did not report Hybrid method assessment outcomes.

Note: The above numbers are not mutually exclusive – meaning one grantee could have done some AFIX visits using the Hybrid method and some visits using the CASA, Mini-CASA, or Equivalent method. 

Change in coverage when using CASA to assess public providers
Thirty-eight grantees used the CASA method to assess coverage during previous and CY2003 assessments in the public sector (see Appendix A, Table 3). The majority (30 of 38) assessed for 4 diphtheria, tetanus, and pertussis (DTaP) doses; 3 polio doses; 1 measles, mumps and rubella dose (MMR), 3 haemophilus influenzae Type B doses (Hib) and 3 hepatitis B doses (HepB) for previous and CY 2003 assessments. This series of vaccinations is documented as 4:3:1:3:3. Six grantees assessed for 4 doses of DTaP, 3 doses of polio and 1 dose of MMR for previous and CY 2003 assessments; commonly referred to as the 4:3:1 series. One grantee assessed for the 4:3:1:3 series for previous and CY 2003 assessments, which is 4 doses of DTaP, 3 doses of polio, 1 dose of MMR, and 3 doses of Hib. One grantee assessed for the 3:2:2:2 series for previous and CY 2003 assessments, which is 3 doses of DTaP, 2 doses of polio, 2 doses of Hib and 2 doses of HepB. 

Eight grantees documented an increase in average public provider vaccination coverage levels to be 10% or greater. Twelve grantees found the average public provider vaccination levels improved 1-9%. The average private provider vaccination coverage levels did not change for 1 grantee.  The change in average public provider vaccination was negative for 17 grantees. This negative change ranged from -1% to -13% for all but three grantees. The other three grantees observed the average percentage point change in coverage to be -16%, -22% and -27.3%.

Grantees that demonstrated either a significant increase or decrease in change in coverage were contacted to elicit possible factors for this change in coverage. A decrease in coverage was most often attributed to access-related issues. Specifically, these include an increasing number of clinics with appointment-only scheduling systems, an inability to immunize managed care children without referral, and some clinics that are not providing simultaneous immunizations at appointments.  Finally, one grantee explained the decrease in coverage as a result of the deferral of the 4th DTaP dose due to the vaccine shortage. Reasons for significant coverage increases were also provided. One grantee explained that providers are becoming more familiar with the assessment process and, as a result, are able to identify ways to increase and maintain coverage rates.
Change in coverage when using CASA to assess private providers
Thirty-nine grantees used the CASA method to assess private provider coverage levels during previous and CY2003 assessments in the private sector (see Appendix A, Table 3). The majority (30 of 39) assessed for the 4:3:1:3:3 series for previous and CY 2003 assessments. Seven grantees assessed for the 4:3:1 series, 1 grantee assessed for the 4:3:1:3 series, and 1 grantee assessed for the 3:2:2:2 series for one year olds. 

Five grantees documented change in the average private provider vaccination level to be 10% or greater. Sixteen grantees found the average public provider vaccination level to change between 1-9%.  The average private provider vaccination coverage levels did not change for 2 grantees and the change was negative for 16 grantees. This negative change ranged from -0.5% to -8.3%

Grantees that demonstrated either a significant increase or decrease in change in coverage were contacted to elicit possible factors for this change in coverage. The grantee with the most significant increase in coverage attributed it to the fact that their performance-based contracts now require objectives directly related to increases in immunization coverage levels.  One factor that contributed to the decrease in coverage levels was the deferral of the 4th dose of DTaP during the vaccine shortage.


2003 NIP Training and Education Activities
In 2002, NIP staff working on the VFC/AFIX project focused on improving grantee training and education. A revised training request application was created which provided the VFC/AFIX staff with the ability to customize trainings for each grantee. These customizing trainings have allowed the VFC-AFIX staff to develop a large presentation library that can be made available to grantees so they can utilize the presentations for local training.

In 2003, this training request application was absorbed into a larger training plan that was developed to assist the NIP VFC/AFIX staff in the planning of 2004 events, particularly the need to designate significant time to properly learn, test, distribute and implement the new, redesigned CASA, titled “Comprehensive CASA” or “CoCASA.”  NIP decided in 2003 that training in 2004 would have to be limited.  As a result, the training plan would strategically organize all grantee requested training to be completed by mid-2004, thus allowing the second half of the year for CoCASA planning.  This new training plan was posted on the AFIX website in the fall of 2003. 

During 2003, NIP staff conducted on-site AFIX and/or CASA trainings to the following grantees:
* Alabama
* New Mexico
* Oklahoma
* South Dakota
* Tennessee
* Utah
* Western New York State

The following grantees requested educational presentations or limited trainings on specific aspects of AFIX or CASA in 2003:
* Arizona
* Oklahoma Southeast Area Health Education Center (AHEC)

In addition, extensive conference calls were held with the following grantees to provide guidance on revising, restructuring and/or improving their respective AFIX programs: 
* Kansas
* Michigan
* Oklahoma













APPENDIX A:
Tables



Table A-1: Assessment Outcome Measures for Public Providers by Grantee, CY2003
Grantee





Age Group Assessed (Months) (1)



Assessment Method (2)




Vaccination Series Measured with CASA (3)

Number Assessed with CASA (4)


Minimum Coverage Level (5)



Maximum Coverage Level (6)



Crude Coverage Level (7)



Vaccination Series Assessed with Hybrid (8)

Number Assessed with Hybrid (9)



Hybrid Threshold Level (10)



Number of Providers at or above Threshold Level (11)

Alabama
19-35
hybrid    





4:3:1:3:3
6
80
2
Alaska
24-35
casa      
4:3:1:3:3
26
40
94
77




American Samoa
Other
casa      
4:3:1:3:3








Arizona
24-35
both      
4:3:1:3:3
23
21
85
58.5




Arkansas
24-35
casa      
4:3:1:3:3
96
63
67
65




California
Other
both      
4:3:1:3:3
358
11.3
100
66.4
4:3:1:3:3
3
80
1
Chicago
12 & 24
both      
4:3:1:3:3
25
19
88
58.6
4:3:1:3:3
4
75
3
Colorado
Other
both      
4:3:1:3:3
34
19
97
52.3
4:3:1:3:3
43
70
5
Connecticut
19-35
casa      
4:3:1:3:3
29
50
100
81.7




Delaware
19-35
casa      
4:3:1:3:3








District of Columbia
19-35
casa      
4:3:1:3:3
15
25.64
90
54.39




Florida
24-35
casa      
4:3:1:3:3
156
13
100
64.4




Georgia
19-35
casa      
4:3:1:3:3
149
0
100
78.2




Guam*











Houston
12 & 24
casa      
4:3:1:3:3
24
53
96
75




Idaho
19-35
casa      
4:3:1:3:3
30
50
100
77.3




Illinois
12 & 24
casa      
4:3:1:3:3
133
29
100
75




Indiana
19-35
casa      
4:3:1:3
170
0
100
78.7




Iowa
12 & 24
casa      
4:3:1:3:3
21
43
100
78.4




Kansas
12 & 24
casa      
4:3:1:3:3
132
0
100
58.3




Kentucky
24-35
casa      
4:3:1:3:3
152
15
100
71.2




Louisiana
24-35
both      
4:3:1
106
13
95
54.4
4:3:1
31
90
15
Maine
24-35
casa      
4:3:1:3:3
63
0
100
68.1




Maryland
24-35
casa      
4:3:1:3:3
39
33
100
82.7




Massachusetts
24-35
hybrid    





4:3:1:3:3
46
80
20
Grantee





Age Group Assessed (Months) (1)


Assessment Method (2)





Vaccination Series Measured with CASA (3)

Number Assessed with CASA (4)


Minimum Coverage Level (5)



Maximum Coverage Level (6)



Crude Coverage Level (7)



Vaccination Series Assessed with Hybrid (8)

Number Assessed with Hybrid (9)



Hybrid Threshold Level (10)



Number of Providers at or above Threshold Level (11)


Minnesota
24-35
casa      
4:3:1:3:3
14
14
84
61.5




Mississippi
Other
casa      
4:3:1
386
0
100
74.4




Missouri
12 & 24
casa      
4:3:1
215
16.7
100
70.5




Montana
24-35
both      
4:3:1:3:3
99
35
100
90.7
4:3:1:3:3
100
90
69
Nebraska
19-35
casa      
4:3:1:3:3
48
28
100
72.3




Nevada
24-35
casa      
4:3:1:3:3
5
32.4
80
61.1




New Hampshire
24-35
casa      
4:3:1:3:3
39
25
100
79




New Jersey
24-35
casa      
4:3:1:3:3
74
22
100
80




New Mexico
12 & 24
casa      
4:3:1:3:3
69
11
90
64




New York City
24-35
casa      
4:3:1:3:3
28
45
91
70




New York State
24-35
casa      
4:3:1:3:3
84
0
100
67.5




North Carolina
12 & 24
casa      
4:3:1:3:3
100
14
95
71.77




North Dakota
24-35
casa      
4:3:1:3:3
18
50
100
80.9




Ohio
24-35
casa      
4:3:1:3:3
61


57.43




Oklahoma
24-35
casa      
4:3:1:3:3
42
6
78
54.9




Oregon
12 & 24
casa      
4:3:1:3:3
62
20
77.78
56.55




Pennsylvania
24-35
casa      
4:3:1:3:3
65
81.5
84.5
83




Philadelphia
19-35
casa      
4:3:1:3:3
8
35
76
63




Rhode Island
12-23
casa      
3:2:2:2
19
61
100
83.5




San Antonio
Other
casa      
4:3:1:3:3
29
3.3
93.3
60




South Carolina
24-35
casa      
4:3:1:3:3
23
13.6
100
61.3




South Dakota
19-35
casa      
4:3:1
83
0
100
92




Tennessee
19-35
both      
4:3:1:3:3
19
0
94
50.4
4:3:1:3:3:1
5
80
1
Texas
Other
casa      
4:3:1
534
0
100
56.98




Utah
24-35
casa      
4:3:1:3:3
45
0
91
67.3




Virgin Islands
24-35
hybrid    





4:3:1:3:3
2
80
2
Virginia
12-23
casa      
4:3:1:3:3
65
27
86
57.2




Washington
19-35
casa      
4:3:1:3:3
50
0
100
49.53




Grantee





Age Group Assessed (Months) (1)


Assessment Method (2)





Vaccination Series Measured with CASA (3)

Number Assessed with CASA (4)


Minimum Coverage Level (5)



Maximum Coverage Level (6)



Crude Coverage Level (7)



Vaccination Series Assessed with Hybrid (8)

Number Assessed with Hybrid (9)



Hybrid Threshold Level (10)



Number of Providers at or above Threshold Level (11)

West Virginia
24-35
casa      
4:3:1:3:3
54
0
100
53




Wisconsin
19-35
casa      
4:3:1:3:3
10
38.1
96.2
73.7




Wyoming
Other
casa      
4:3:1:3:3
56
25
100
76.43






*Although Guam conducted site visits, it did not have a standardized assessment protocol and, therefore, was not able to report any measured assessment outcomes. 

1. Age Group Assessed: the age range of the patients included in the assessment
2. Assessment Method: method used to evaluate vaccine coverage level (CASA or Hybrid)
3. Vaccination Series Measured: vaccine series (type and number of doses of vaccine) used to evaluate up-to-date status during the assessment visit. All coverage levels will refer to the completion of this series. 
4. Number Assessed with CASA: number of providers that received an assessment using the CASA method.
5. Minimum Coverage Level: among all providers assessed, the lowest determined vaccination coverage for the vaccination series measured
6. Maximum Coverage Level: among all providers assessed, the highest determined vaccination coverage for the vaccination series measured
7. Crude Average Coverage Level: an unweighted average vaccination coverage level among all provider sites
8. Vaccination Series Assessed with Hybrid: vaccine series (type and number of doses of vaccine) used to evaluate up-to-date status during the assessment visit when using the Hybrid Method.  
9. Number Assessed with Hybrid: number of providers that received an assessment using the CASA method.
10. Hybrid Threshold Level: vaccination coverage level at which providers are expected to be performing at or above
11. Number of Providers At or Above Threshold Level: number of providers at or above threshold level 



Table A-2: Assessment Outcome Measures for Private Providers by Grantee, CY2003
Grantee






Age Group Assessed (Months) (1)


Assessment Method (2)




Vaccination Series Measured with CASA (3)

Number Assessed with CASA (4)


Minimum Coverage Level (5)



Maximum Coverage Level (6)



Crude Coverage Level (7)



Vaccination Series Assessed with Hybrid (8)

Number Assessed with Hybrid (9)


Hybrid Threshold Level (10)



Number of Providers at or above Threshold Level (11)

Alabama
19-35
hybrid    





4:3:1:3:3
1
80
0
Alaska
24-35
casa      
4:3:1:3:3
11
42
82
60




Arizona
24-35
both      
4:3:1:3:3
131
0
100
56.5




Arkansas
24-35
casa      
4:3:1:3:3
116
35
39
37




California
Other
both      
4:3:1:3:3
67
1.92
100
61.2
4:3:1:3:3
161
80
99
Chicago
12 & 24
both      
4:3:1:3:3
108
0
100
54
4:3:1:3:3
21
75
6
Colorado
Other
both      
4:3:1:3:3
4
25
87
57.9
4:3:1:3:3
49
70
15
Connecticut
19-35
casa      
4:3:1:3:3
119
16.7
100
86.8




Delaware
19-35
casa      
4:3:1:3:3
86
17
100
78




District of Columbia
19-35
casa      
4:3:1:3:3
39
14.29
100
58.12




Florida
24-35
casa      
4:3:1:3:3
381
22
100
73.4




Georgia
19-35
casa      
4:3:1:3:3
83
33.3
100
87.3




Guam*











Houston
12 & 24
casa      
4:3:1:3:3
205
10
96
53




Idaho
19-35
casa      
4:3:1:3:3
95
13
100
69.1




Illinois
12 & 24
casa      
4:3:1:3:3
131
16
100
72.2




Indiana
19-35
casa      
4:3:1:3
510
0
100
80.4




Iowa
12 & 24
casa      
4:3:1:3:3
57
55
100
83.1




Kansas
12 & 24
casa      
4:3:1:3:3
46
0
100
51.2




Kentucky
24-35
casa      
4:3:1:3:3
27
16
95
71.8




Louisiana
24-35
both      
4:3:1
140
0
100
58.1
4:3:1
34
90
7
Maine
24-35
casa      
4:3:1:3:3
150
0
100
63.3




Maryland
24-35
casa      
4:3:1:3:3
532
0
100
85.9




Massachusetts
24-35
hybrid    





4:3:1:3:3
252
80
161
Michigan
19-35
casa      
4:3:1:3:3
127
6
100
60




Minnesota
24-35
casa      
4:3:1:3:3
27
7
100
74.6




Grantee






Age Group Assessed (Months) (1)


Assessment Method (2)




Vaccination Series Measured with CASA (3)

Number Assessed with CASA (4)


Minimum Coverage Level (5)



Maximum Coverage Level (6)



Crude Coverage Level (7)



Vaccination Series Assessed with Hybrid (8)

Number Assessed with Hybrid (9)


Hybrid Threshold Level (10)



Number of Providers at or above Threshold Level (11)

Mississippi
Other
casa      
4:3:1
71
0
100
75




Missouri
12 & 24
casa      
4:3:1
107
36.4
100
71.1




Montana
24-35
both      
4:3:1:3:3
66
0
100
83.5
4:3:1:3:3
69
90
37
Nebraska
19-35
casa      
4:3:1:3:3
90
20
100
78.3




Nevada
24-35
casa      
4:3:1:3:3
60
0
95
52.4




New Hampshire
24-35
casa      
4:3:1:3:3
77
18
100
88




New Jersey
24-35
casa      
4:3:1:3:3
45
0
100
63.6




New Mexico
12 & 24
casa      
4:3:1:3:3
71
17
100
67




New York City
24-35
casa      
4:3:1:3:3
87
0
90
45




New York State
24-35
casa      
4:3:1:3:3
174
0
100
74.8




North Carolina
12 & 24
casa      
4:3:1
99
33
100
80




North Dakota
24-35
casa      
4:3:1:3:3
25
30
100
86.2




N. Mariana Islands
Other










Ohio
24-35
casa      
4:3:1:3:3
247


62.69




Oklahoma
24-35
casa      
4:3:1:3:3
11
0
0
0




Oregon
12 & 24
casa      
4:3:1:3:3
148
3.45
100
57.12




Pennsylvania
24-35
casa      
4:3:1:3:3
510
81.3
82.9
82.1




Philadelphia
19-35
casa      
4:3:1:3:3
108
0
100
70




Rhode Island
12-23
casa      
3:2:2:2
55
0
100
84.1




San Antonio
Other
casa      
4:3:1:3:3
79
0
10
60




South Carolina
24-35
casa      
4:3:1:3:3
41
17.3
100
63.8




South Dakota
19-35
casa      
4:3:1
166
0
100
78




Tennessee
19-35
both      
4:3:1:3:3
131
0
100
65
4:3:1:3:3:1
10
80
4
Texas
Other
casa      
4:3:1
1415
0
100
59.13




Utah
24-35
casa      
4:3:1:3:3
113
0
100
67.6




Vermont
24-35
casa      
4:3:1:3:3
6
81.3
97.6
91.3




Virgin Islands
24-35
hybrid    





4:3:1:3:3
19
80
15
Virginia
12-23
casa      
4:3:1:3:3
162
17
100
73.1




Washington
19-35
casa      
4:3:1:3:3
143
0
100
54.51




Grantee






Age Group Assessed (Months) (1)


Assessment Method (2)




Vaccination Series Measured with CASA (3)

Number Assessed with CASA (4)


Minimum Coverage Level (5)



Maximum Coverage Level (6)



Crude Coverage Level (7)



Vaccination Series Assessed with Hybrid (8)

Number Assessed with Hybrid (9)


Hybrid Threshold Level (10)



Number of Providers at or above Threshold Level (11)

West Virginia
24-35
casa      
4:3:1:3:3
136
50
100
75




Wisconsin
19-35
casa      
4:3:1:3:3
81
54
100
70.4




Wyoming
Other
casa      
4:3:1:3:3
42
33
100
74






*Although Guam conducted site visits, it did not have a standardized assessment protocol and, therefore, was not able to report any measured assessment outcomes. 

1. Age Group Assessed: the age range of the patients included in the assessment
2. Assessment Method: method used to evaluate vaccine coverage level (CASA or Hybrid)
3. Vaccination Series Measured with CASA: vaccine series (type and number of doses of vaccine) used to evaluate up-to-date status during the assessment visit. All coverage levels will refer to the completion of this series. 
4. Number Assessed with CASA: number of providers that received an assessment using the CASA method.
5. Minimum Coverage Level: among all providers assessed, the lowest determined vaccination coverage for the vaccination series measured
6. Maximum Coverage Level: among all providers assessed, the highest determined vaccination coverage for the vaccination series measured
7. Crude Average Coverage Level: an unweighted average vaccination coverage level among all provider sites
8. Vaccination Series Assessed with Hybrid: vaccine series (type and number of doses of vaccine) used to evaluate up-to-date status during the assessment visit when using the Hybrid Method.  
9. Number Assessed with Hybrid: number of providers that received an assessment using the CASA method.
10. Hybrid Threshold Level: vaccination coverage level at which providers are expected to be performing at or above
11. Number of Providers At or Above Threshold Level: number of providers at or above threshold level 


Table A-3: Change in Vaccination Coverage Levels, Grantees that Used CASA Method 
for Previous and CY2003 Assessments (n=42)
											
Grantee









Total Number Assessed (1)




Vaccination Series Measured (2)






Crude Average Coverage Level (Previous) (3)



Crude Average Coverage Level (CY2003) (4)






Average Percentage Point Change In Coverage (5)




Crude Average Coverage Level (Previous) (3)



Crude Average Coverage Level (CY2003) (4)




Average Percentage Point Change in Coverage (5)



Alaska
31
4:3:1:3:3
61
77
16
59
60
1
Arizona
43
4:3:1:3:3
54.7
61.7
7
55.3
51.7
-3.6
Arkansas
98
4:3:1:3:3
77.25
67
-10.25



California
314
4:3:1:3:3
59.2
66.4
15.4



Chicago
284
4:3:1:3:3
49.2
58.6
9.4
58.8
53.7
-5.1
Connecticut
26
4:3:1:3:3
69.4
81.6
12.1
72.5
86.3
13.8
Delaware
27
4:3:1:3:3



80.3
78
-2.3
District of Columbia
82
4:3:1:3:3
61.69
54.39
-7.3
61.74
58.12
-3.62
Florida
84
4:3:1:3:3
75.9
64.4
-11.5
77.7
73.4
-4.3
Georgia
174
4:3:1
90.9
94.2
3.3
92
95.3
3.4
Houston
169
4:3:1:3:3
65
75
10
44
53
9
Idaho
61
4:3:1:3:3
76.4
82
5.6
75.2
76.5
1.2
Indiana
566
4:3:1:3
79
79.2
0.2
80.3
81
0.7
Iowa
18
4:3:1:3:3
55.7
82.5
26.8
66.1
83.2
17.1
Kentucky
406
4:3:1:3:3
84.1
71.2
-12.9
78.8
71.8
-7
Louisiana
103
4:3:1
81.3
54
-27.3
70.5
62.2
-8.3
Maine
158
4:3:1:3:3
77.5
67.4
-10
71.9
65.6
-6.3
Maryland
576
4:3:1:3:3
76
82.7
6.7
87
85.9
-1.1
Michigan
35
4:3:1:3:3



48
60
12
Minnesota
2
4:3:1:3:3
53
50
-3.0



Mississippi
384
4:3:1
76.8
71.1
-5.7
75.5
71.8
-3.7
Grantee









Total Number Assessed (1)




Vaccination Series Measured (2)






Crude Average Coverage Level (Previous) (3)






Crude Average Coverage Level (CY2003) (4)




Average Percentage Point Change In Coverage (5)







Crude Average Coverage Level (Previous) (3)



Crude Average Coverage Level (CY2003) (4)




Average Percentage Point Change in Coverage (5)



Missouri
322
4:3:1
73.1
71.6
-1.5
72.8
72.3
-0.5
Nebraska
93
4:3:1:3:3
73.4
70.6
-2.8
72.2
75.5
3.3
Nevada
65
4:3:1:3:3
81
65
-16
70.2
70.9
0.7
New Hampshire
116
4:3:1:3:3
83
79
3
87
88
1
New Jersey
149
4:3:1:3:3
78.9
79.9
1.1
70.5
66
-4.7
New Mexico
134
4:3:1:3:3
61
64
3
61
67
6
New York City
33
4:3:1:3:3
63
76
13
40
44
           4
New York State
166
4:3:1:3:3
70.8
67.7
-3.1
73.7
75.4
1.7
North Carolina
31
4:3:1



72
77
5
Ohio

4:3:1:3:3
61
57.43
-3.57
65.72
62.69
-3.03
Oregon
112
4:3:1:3:3
56.81
56.78
-2.52
64.73
58.8
-4.56
Philadelphia
112
4:3:1:3:3
61
63
2
64
70
6
Rhode Island
67
3:2:2:2
81
82.8
1.8
89.2
82.3
-7
San Antonio
108
4:3:1:3:3
55.8
60
4.2
60.5
60.9
0.4
South Carolina
75
4:3:1:3:3



70
65.3
-4.7
South Dakota
249
4:3:1
90
92
2
74
78
4
Texas
1715
4:3:1
64
56.98
-7.02
59
59.13
0.13
Utah
115
4:3:1:3:3
82.9
70.9
-12
52.3
67.5
15.2
Virginia
101
4:3:1:3:3
41.48
57.2
15.72
67
73.1
6.1
West Virginia
57
4:3:1:3:3
75
53
-22
74
75
1.0
Wisconsin
202
4:3:1:3:3
55.3
73.7
18.4
59.1
70.4
11.3


1Total Number Assessed: total number of providers that received an AFIX visit
2 Vaccination Series Measured: vaccine series (type and number of doses of vaccine) used to evaluate up-to-date status during the assessment visit. All coverage levels will refer to the completion of this series. 
3 Crude Average Coverage Level (Previous): an unweighted average vaccination coverage level among all provider sites based on vaccination coverage levels from assessments performed between 1/01/01 and 12/31/02
4 Average Coverage Level (CY 2002): an unweighted average vaccination coverage level among all provider sites based on vaccination coverage levels from assessments performed during CY 2003
5 Average Percentage Point Change in Coverage: increase or decrease between previous coverage level and coverage level CY 2002


















APPENDIX B: 
Assessment Methods


Appendix B: Assessment Methodology Options


Method

Description

Advantages

Disadvantages

Diagnostic Tools

CASA

Sample selection: random sample
Sample size: approximately 100 per cohort
Inputs:	child’s demographic information; date of each immunization; other information related to diagnostic analysis
Outputs:	diagnostic information on late starts, drop-offs and missed opportunities
baseline measure of coverage

Precise estimates of immunization coverage levels; evaluation of missed opportunities; evaluation of late starts, etc.

Large sample size
Resource burden (staff, time)


Registry based

Sample selection: census of all  eligible records
Sample size: all eligible records
Inputs:	 data from registry downloaded to CASA 
Outputs:    diagnostic information on late starts, drop-outs and        missed opportunities

Minimal time and effort for data collection
no sampling error since estimates based on census of records. 

Potential biases:
- Registry may not contain 100% of provider’s records
- Reliability of registry data


Mini-CASA

Sample selection:  consecutive, convenience or random sample
Sample size: 40-60
Inputs: 	  same as CASA
Outputs:     diagnostic information on late starts, drop-off and missed opportunities

Smaller sample sizes

Coverage estimates have less precision; sample may not be randomly selected

Hybrid

Hybrid
(LQA/CASA) 

Sample selection: random sample
Sample size: 30
Inputs:	  same as CASA
Outputs:	determines if a provider has immunization coverage       above or below a specified threshold
                    vaccine histories of not up-to-date clients as examples to discuss with provider and staff

Smallest sample size
Rapid assessment
More feedback information than LQA alone
Identifies providers who may benefit from a diagnostic assessment

Computer needed (not paper and pencil as LQA)
Does not give point estimate of coverage
Smaller basis for diagnostic feedback



12