Tribal SG Evaluation Feasibility Study
Technical Work Group Meeting
February 3, 4 and 5, 2003
Washington, D.C.

Attendance:

Monday, February 3, 2003

1. Welcome and Introductions
   After a brief welcome, the facilitator requested that all participants introduce themselves.
2. Project Overview
   • Andy Rock thanked everyone for their attendance and mentioned that the combined efforts of Westat, Inc., Project HOPE and Kauffman and Associates, Inc. brings with it a wealth of experience to the study. He stated that the Technical Work Group (TWG) will participate throughout the study and provide advice when necessary. He said the study will provide information needed on a possible future evaluation. He stressed that this is a feasibility study and not an evaluation. Designing and conducting an evaluation are not part of this immediate process. Rather, he stated that what the TWG is engaged in is preliminary work that will lead up to the possible design and conduct of an evaluation in the future. Also, he said the primary focus of this process is on health services programs and programs that the demonstration study report (regarding non-IHS programs) recommended be included in a project extending Self Governance. He mentioned that there are 11 of these programs. He stated that the Office of Intergovernmental Affairs will be conducting consultations with the Tribes and he introduced Gina Tyner-Dawson of that office who is responsible for these consultations. (She mentioned that these will be the first consultations since 1999 and the forum will be open for budget issues and that information regarding the Evaluation Feasibility Study will be included.) Andy Rock then stated that role of the TWG is to advise the contractor on all aspects of the study. He stressed that this will be a productive activity and should not be viewed as critical of Self Governance, but rather it is seeking constructive ways of advancing it.
   • Jo Ann Kauffman explained her role in facilitating the TWG process and recording information to documentation the Work Group's deliberations and advice. She also provided her tribal affiliation, background on KAI, and role as a subcontractor. Jo Ann provided a brief overview of her past projects and advocacy activities as they relate to SG, as well as her philosophy on SG.
   • Sherm Edwards of Westat, Inc. explained that Westat is the "prime contractor" and this project was being conducted as one of its Task Order projects under its broad contract with the Office of the Assistant Secretary for Planning and Evaluation. Westat will be looking at some of the data and the logistical details. He mentioned that Westat has an administrative role and final review of documents before they are submitted to the Task Order Monitors (Andy Rock and Tom Hertz). In response to the question from a member of the TWG of whether Westat, Inc. has a Native American on its staff, familiar with Self Governance, Sherm responded affirmatively and indicated that this staff member would be working on the project.
   • Kathy Langwell, with Project HOPE a subcontractor, mentioned that she has directed and participated in many quantitative research projects and has been designing programs and short term and multi-year projects. She mentioned that in the last two years, she has been working on a CMS project for eligibility of NAs in rural areas and conducting site-visits in tribal communities. She has several grants relating to NA issues.
   • Alvin Windy Boy, Sr. expressed concern that he said he has spent 9 1/2 months in a process similar to this one with no results. He stressed that Tribes just want the ability to provide a service. He said that the trust responsibility has broken down and that Indian Country is no further along in 10 years and has not increased funding since 1955. Research is not a strange word in Indian Country. The research has not benefited tribes. If this (evaluation feasibility study) process results in things he can utilize to benefit Indian Country, he is behind it. But there needs to be a plan.
   • Dan Jordan mentioned that studies often do not produce the desired results. He stressed that Self Governance has allowed Tribes to become empowered to address problems on reservations and that Tribes generally respond to studies in a constructive manner, but this information is not always reflected in studies. He suggested thinking about Self Governance just like we do democracy. He stated that Self Governance makes the economy work and creates jobs and this needs to be captured in any evaluation study. He stressed that a tribe's ability to control its environment is the fundamental purpose of Self Governance and program issues are secondary. He mentioned that we would be looking at analysis of increase in services and funding. Services have increased but not funding. Has SG changed the lives of those living on reservations? Have these services improved the lives of Tribal members? Does SG (????) the study actually empower the Tribes comparable to the rest of society?
   • Tim Martin mentioned that Alabama is mentioned in the Data Matrix Report, Page 3. He pointed out that state-recognized tribes are listed, but not federally recognized tribes in the state. Self Governance only applies to federally-recognized tribes. He indicated that we should purge this data. Kathy Langwell responded that this table was taken from the Federal Register listing federally recognized tribes and there shouldn't be any State Tribes in this report. She will double check and make the corrections.
   • Mickey Peercy stressed the importance of staying focused on our purpose and that the ultimate questions are whether there is or is not data available to do a study. He suggested also making a list of areas the group does not want to address in order to stay focused as it will be easy to move down different directions.
3. TWG Ground Rules
   A discussion was conducted to seek agreement about procedural matters.
   1. TWG Agreements on the Ground Rules:
      1. Stay focused on:
         1. Purpose of the study
         2. Feasibility of an evaluation process
         3. Is data available to conduct the study
      2. Written charge from the HHS to the TWG is to be developed
      3. TWG is advisory to the contractor
      4. Formal Tribal Consultation is a separate process
      5. Protocol will be developed and reviewed (This was completed before the 3-day meeting adjourned.)
4. Legislative History & Literature Review
   1. TWG Recommendations about the Legislative History Report:
      1. Allow written comments
      2. The History paper needs to capture why Tribes became involved in Self Governance
         1. Complexity and nuances have been lost in this report. It's too sanitized. It doesn't appear to fully take into account the Tribal perspective.
         2. Should reflect failure of non-BIA DOI SG effort.
         3. It should reflect the frustration of Tribes over 1988 regulatory process.
         4. Should indicate also that Tribes may exercise their SG even when electing to keep HIS direct services arrangements.
         5. Title IV preamble of proposed and final rules describes this process very well and it is a federal document that has already been published. It should be used to convey this history.
         6. There are gaps in what is currently available. Perhaps will need an oral history of tribal leaders that were involved to capture some of this history.
         7. The difference between negotiated rule making and the previous process for establishing regulations should be reflected.
         8. The Lummi Demonstration Project is a good source for information.
         9. Early SG reports are available and there is a "Blue Print for Self Governance" which is a good source.
         10. Paula Williams of IHS/HHS/OTSG mentioned that there is a case study with Valerie Davidson that is coming out and she can distribute to the TWG when it comes out.
      3. There needs to be clarification that Self Governance does not stem from PL 93-638, but from a different process; a government-to-government process.
      4. The reasons why some Tribes opt for PL 93-638 contracting and not SG compacting should be addressed.
   2. TWG Recommendations for the Draft Literature Review:
      1. Obtain DHHS evaluation on EPSDT Medicaid Expansion at Yukon Kuskokwim through HRSA.
      2. Need section describing quantitative and qualitative methods, and the history of each approach in regards to SG. The study seems to imply that only quantitative methods are good. The paper should identify what needs to be examined and how both qualitative and quantitative methods could be used to achieve answers to these questions.
      3. Need to ask IHS for existing quantitative evaluative studies on Self Governance and Non-Self Governance and make effective use of RPMS data system.
      4. Move emphasis on limitations of the study and indicate it may be difficult to conduct evaluation because:
         1. Small tribal populations;
         2. Dispersed populations through out country;
         3. Move this limitations section to the front;
         4. Constraints in longitudinal approach;
         5. Causal relationship to health status versus relationship to health care; and
         6. Define what would be statistically significant and what this research would cost.
      5. Need to note that Indian programs are starting from a point of under-funding, for example LNF. (level of need funding????)
      6. Page 4 regulation statement needs to be reworded, it says compacting is not subject to regulation and that is not true.
      7. Reference relationship of poverty to health status, and provide more emphasis on the economic stimulus that SG can have on communities.
      8. Be careful in wording, so that phrases cannot be lifted out of context to the unfair detriment of tribes.
      9. Better transition from IHS to non-IHS programs needed.
      10. Is there a risk of documenting successes? i.e. :
          1. Federal cutbacks?
          2. Unfair comparisons?
      11. Conclusion needs rewording.
      12. Add 477 successes experienced by tribes combining jobs, vocational rehab and child care resources.
      13. Better explanation and application of "user population" is needed as this will impact quantitative data.
      14. Discuss importance of Tribal buy-in or support for research/evaluation.
      15. Note Tribal investments; e.g., Contract health services, Staff, Construction, etc.
      16. Obtain report on Tribal shares [94-95 JAMAW (???) report through Mr. Lincoln or Mickey Peercy].
          
          Additional Project Team Comments:
          • The paper reflects what research has been conducted to date.
          • More literature was found than anticipated.
          • Most of the existing research is qualitative with focus on process, perceptions, documentations, and descriptions. For example, NIHB study did excellent job conducting a survey and seeking perceptions of SG tribes.
          • It is not surprising that there is little quantitative data.
          • What we were looking for in the literature review is a comparison of existing literature.
          • There is quantitative data and studies being done on Indian programs, ie there is a Mathematica Quantitative Data Report regarding TTANF; and Westat, Inc. is doing an evaluation of Head Start projects.
          • This is an important document in laying groundwork of where the project and data might go.
          • It may not be worthwhile for us to dig into everyone's data sets to tease out information. What is the relative standard of comparison is an issue.
          • Pre and post comparisons are a good way of looking at changes.
          • Cross-sectional comparison is another method.
          • We are exploring IHS data.
          • In the end, politics and "telling the story" and not quantitative data is important to Congressional staff when considering legislative changes.
5. Methodology
   1. TWG Recommendations for Evaluation Methodology:
      1. Feasibility of quantitative evaluations
         1. a. IHS and non-IHS
            1. What data is there? Is it accessible? Is it comparable?
         2. Six (6) Tribal site visits
            1. What data is there on site?
            2. Prepare Tribes to address non-IHS questions
         3. What is the baseline to compare the progress of Self Governance?
            1. IHS activity pre-Self Governance?
            2. Treaty obligations?
         4. Describe the process of success
      2. No single measure will apply to all Tribes. This must be flexible.
      3. Given literature supporting local control of health care, can we then document process of increased Tribal leader and community involvement?
         1. Consumers are more involved in priority setting under SG
         2. Increased role of Tribal leaders in health resources and priority setting
      4. Determine feasibility to evaluate IHS Self Governance to better inform potential non-IHS Self Governance expansion and evaluation considerations
      5. Track the time needed to produce necessary data for this study
      6. Questions of tribes engaged in gaming should not be an issue for this study; a focus on tribes' efforts to maximize their resources could be viewed negatively by some policy-makers.
      7. Consider the geographic and demographic differences of Tribes across the country
      8. This process should be carried out so as to move us closer to implementing a Title VI Demonstration
      9. How can we measure "Empowerment" under SG?
         1. Participation of community via quantitative measures
         2. Probably qualitative would be most descriptive method
      10. Health outcome measures probably not realistic for this purpose.
          1. Is this realistic for small populations and time with Self Governance?
          2. Is it realistic to assume health status changes due to SG given the short time frame?
          3. How do you make causal link between health status changes and SG?
      11. Recommendations for evaluation comparisons that should be made:
      1. Pre/Post Self Governance of the same tribe;
      2. Program processes; and
      3. Measures for reorganization, administration or governance.
      12. Elements of qualitative measurement that focuses on SG could include:
      1. How much program redesign was done;
      2. Whether financial systems modified/improved;
      3. What changes were made to personnel and administrative systems;
      4. Differences under local decision making; and
      5. Whether services are more comprehensive under SG.
         
         Project Team responses to comments:
         • The feasibility study seeks to answer whether it is feasible to do an evaluation. We are not actually doing the evaluation now.
         • We started with a broad range of topics.
         • If funding and patient utilization data aren't acceptable measures, what quantifiable data would you like to see that would be of interest to all stakeholders?
         • We seek to define the broader issues and then go to IHS and do a review of all the data and then recruit Tribes to participate in the site visits (Tribes, program managers, etc.).
         • If this is structured to address the impact of Self Governance on key outcomes, etc., then we need to know what was there before (IHS data) and then go out to the Self Governance Tribes. We're going to have to rely upon individual Tribes who are willing to participate.
         • We hope to recruit Tribes to participate in the feasibility study and who would also be willing to participate in an evaluation that may follow. Interested Tribes represented on the TWG are encouraged to consider such participation.
         • We are looking at this (TWG) group to provide advice on the parameters of the study. We might end up with recommendations for an evaluation emphasizing qualitative measures if that is the only feasible way of collecting data.
         • The Tribal Self Governance Demonstration report shows a consensus and willingness to proceed on the part of the Department. The decision of when and how to proceed with a demonstration project would need Congressional legislative action. We would expect an evaluation would be part of any mandated demonstration project enacted.

Tuesday, February 4, 2003

6. Major Study Areas & Questions
   1. TWG Recommendations for Study Area Parameters:
      1. Focus on Governance - Should answer the question: "What is the impact on Tribes"
         1. Administration or Governance changes
         2. Structural changes
         3. Finance changes
      2. Health Indicators (utilization, systems, status)
      3. What did we do and how did we do it to fill the shoes of the federal government?
      1. FTCA
      2. Budget
      3. Staffing
      4. JCAHO Ratings
      4. Proposed "Guiding Principles" for this feasibility study:
      1. Evaluation must start with consultation and negotiation with specific Tribes involved in the evaluation process;
      2. What are the do's and don'ts for SG?
      3. Identify Tribal specific measures
      4. Identify where/if data might be available
         
         Technical Work Group Comments:
         • Tribal leaders and tribal representatives caucused. Discussion resumed.
         • Discussion regarding qualitative and quantitative means to answering the same questions. The TWG recommended that we go through the questions and that there be both a quantitative piece relating to Self Governance as well as qualitative. Questions drafted to date are health indicator questions and we want management level information as well.
         • Some Tribes were doing these things prior to Self Governance via 638.
         • Check with the government for existing records and reports, as it is disrespectful to the Tribal administrator and staff, for federal contractors to come to the community and ask for data that has already been submitted to federal agencies.
         • There is a copy of the Health Indicators Baseline Measures. However, none of the health indicators are mandated and usually it is not used.
         • It's critical that you speak to Tribal Chairman and leaders and get their perspectives.
         • You need to talk about more programs than just health care. We cannot just look at one program by itself.
         • What is the array of providers and the relationships in place?
         • All the people at the table are willing to help regarding coming up with some questions. We have a question about your contractor's understanding of what the leaders are saying and what the contract is about.
         • What is the change in staffing and what is the change in programs of similar size in 94 are two controls worth looking at.
         • We assume that IHS hasn't been stagnant either; that it has also changed services, provided greater access, etc.; and that they have the data.
         • Because programs are so different, Tribes must you pick and choose what is applicable for them and an evaluation needs to take into account these choices.
         • An evaluation would need feedback on the quality of data. The data cannot just be taken by itself; it must be explained and the context provided.
         • You have to ask how many providers you have access to, etc. to broaden the questions and not get hung up on the words as this will be redrafted.
         • Will we be asking about socio-demographic data?
         • Open ended questions will allow for more qualitative data. The qualitative questions will allow for additional information. There needs to be both qualitative and quantitative components.
         • Part of what we want to focus on is the importance of Self Governance that allowed for getting a doctor or other opportunities. An increase or decrease may not be bad. Having no doctors isn't necessarily bad since the tribe may have contracted for these services with a local clinic.
         • To measure success, you have to look at how an individual tribe achieves its own goals rather than universal goals. They control the resources more now than before Self Governance.
         • The tribe can provide a certain matter of added value. This study doesn't indicate that there are study rules up front. Tribes could use this as an option. There should be health and non health criteria/standards out there. There have been a tremendous amount of studies conducted.
         • The source of funding is not critical to the evaluation of the program.
         
         Project Team responses to comments:
         • We are primarily looking at quantitative data when doing site visits.
         • We are looking at both qualitative and quantitative data. Whether information is available is the main focus. We are not getting the data for purposes of evaluation. Information availability is not a criterion for site visit selection.
         • Can we get the information from the site visits is the question. We might find there are ways of getting the data, but it could be cost prohibitive. We need to answer these questions. Another question could be: can you get historical information today?
   2. TWG Recommendations for Study Questions - The questions should be reworked and consider the following issues:
      1. Need to Recognize Phases in the SG Process. Not all features are fully in place from the outset; they occur over time and SG tribes are learning organizations that change.
   1. Stabilize/trust
   2. Management changes
   3. Growth/expansion
   2. Governance
   1. Formal connection to the community
   2. Organizational structure (lines of communication)
   3. Management Issues
   1. Process and changes, etc.
   2. How did Self Governance help in addressing management issues?
   3. TWG Recommendations for Non-IHS Study Questions --- the non-IHS study questions should be reworked and consider the following issues:
      1. Describe organization/governance
   1. Advisory Boards
   2. Inter program coordination already underway
   1. 477
   2. TANF
   3. Historical documentation
   1. Original Annual Funding Agreements (AFA), (Congressional Appropriations)
   2. Tribal Plans
   3. 638 Transition history
   4. Program Budgets
   5. Inter-Tribal Coordination
   4. TWG Recommendations for Program and Benefits Section:
      1. Expand and supplement…..
      2. Reorganize
      3. Annual Report?
      4. Annual ACF 700 report and other Title VI reports/regulations
   5. TWG Recommendations for Accreditation Section:
      1. List
      2. Years
   6. TWG Recommendations for Facility Section:
   1. Numbers
   2. Accreditation
   7. TWG Recommendations for Quality/Satisfaction Section:
   1. Background
   2. Compare to National/Data?
   3. Identify Existing Standards/Data Collected by IHS and Non-IHS
   8. TWG Recommendations for Management Challenges Section:
      1. Add "and Opportunities" to read Management Challenges & Opportunities
      2. Add: What changes do you contemplate would result in improved services to Tribal members?
      3. Add question: How does operating as a SG Tribe allow you to address problems differently?
         
         Project Team Comments:
         • There may be times when Self Governance is relevant and times when it isn't. What are the management issues you were trying to tackle before and after Self Governance?
   9. TWG Recommendations for Availability of Services/Access to Care Questions (Page 2):
      1. Add: What changes did you make and why?
      2. Add: What problems still exist and why?
      3. Add: What could fix the problems?
      4. Add: What factors are most important in the decision to bring services in-house versus Contract Health Services?
      5. Discuss with community issues considered in order to decide how services are delivered (direct, CHS?) relative to "in house" question
      6. What is the CHS Risk Management under Self Governance
      7. Did you first do a feasibility study to take over CHS?
      8. Management opportunities under Self Governance and CHS
   1. Clear up old debts?
   2. Negotiate Better hospital/provider rates?
   3. Hired direct provider?
   9. Waiting Time
   1. What has a Tribe done to reduce waiting times?
      
      Technical Work Group Comments:
      • Services are different everywhere.
      • Questions need to be refined.
      • 2001 User Population is the most current total tribal client data (per Cliff Wiggins).
      • How do you manage the risk with contract health?
      • It is possible to bankrupt a system when becoming involved in CHS.
      • What was the reasoning behind doing Contract Health?
      • CHS under IHS is a difficult system with long delays in payment.
      • What did the Tribes do to manage Contract Health after taking it over from the IHS?
   10. TWG Recommendations for Quality of Care/Process Questions (Page 3):
       1. IHS/Non-IHS
       2. Accreditation as a measure
       3. Licensure as a measure (or at least to describe what is followed)
       4. National Standards as a measure
       5. Did you change application requirements with Self Governance?
          
          Technical Work Group Comments:
          • Look at Alaska reports to outline problems with data collection in all areas not just AI/AN data collection.
          • Remember to look into questions about IHS not being adequate or labeled as insurance.
          • Immunization gap between fed and non-fed programs.
          • How to differentiate that the data collected will go along with regional differences.
          • NIH Research into diabetes, but remember diabetes funding is not included under Self Governance.
          • Preliminary qualitative diabetes data rather than quantitative.
   11. TWG Recommendations for Quality of Care/ Health Outcomes Questions (Page 4):
       1. Eliminate all original questions
       2. IHS GPRA Seminal Health Indicators
       3. Baseline Measures (95-96)
          
          Technical Work Group:
          • All of these questions should be eliminated.
          • Mickey Peercy will come up with some additional health indicators. He indicated that there are 110 Baseline Measures in the Report, but these are not required of tribes.
          • Cliff Wiggins mentioned that IHS is going to use seminal indicators that are part of the overall budget, which is more measurable than the overall assessment.
   12. TWG Recommendations for Quality of Care/Patient Satisfaction Questions (Page 5):
       1. Eliminate: What proportion of the population eligible for services uses the SU annually.
       2. Add: Have you opened a new facility?
       3. Add: Is there data on change patterns?
       4. Add: What percent of potential users utilize your facility?
       5. JCAHO Customer Satisfaction Survey
          
          Technical Work Group Comments:
          • There is no common denominator to who is eligible. Census could be used, but it is self reporting.
          • Standards of eligibility varies from tribe to tribe.
          • The information you are seeking is not reliable enough to make the links you are trying to make.
          • Every facility has some measure of patient satisfaction. Ask interviewee if you can quantify this data and what changes have taken place. Is there a pattern and do they have the data.
   13. TWG Recommendations for Financial Performance Questions (Page 6):
       1. Eliminate the last four questions
       2. Add qualitative questions and identify as such
       3. Regarding question 2, units of service, need to first ask whether there have been changes in the types of care rendered.
       4. It is unclear what the value of the numbers is; the data here may not answer the question of why they are higher or lower; the interpretation of the data is the essential key; merely collecting it could be misleading or worse.
       5. Add question relating to the pre/post collection of dollars
       6. Regarding the question concerning collection from third parties; a more important issue may be the efficiency of the business office.
       7. Add question relating to financial incentives for MD's, etc.?
       8. Add question relating to traditional practitioners
       9. Add question relating to negotiated rates
       10. There may be other proxies rather than dollars for the effectiveness of financial performance; such as audit information.
       11. Add question relating to providers per square footage of facility
       12. Add question relating to the efficiency of the business office
       13. Add question relating to 3^rd party billing system reserves devoted to tribal priorities
       14. Add question relating to the availability of a reserve fund
           
           Technical Work Group Comments:
           • The questions need to start with numbers rather than percentages.
           • You have to measure the level of effort on the part of the tribe. You can get this information of 1 and 2 through RPMS.
           • You have to put all the answers in some sort of a context. Numbers always mean something to the reader and they can be read in many ways, thus, it is important that they are put in some sort of a context. The data is feasible to collect, but does it make any sense.
           • Why is the break even and surplus question important? You can look at the proxies and the audits.
           • If at the end of the report, you mention that there is a lot of data available from the Federal government, this would be helpful to Congress.
           • Remove the last four questions.

Wednesday, February 5, 2003

7. Small Group Discussions
   1. TWG Recommendations for the Discussion Groups --- Contractor will rework the Discussion Group format based upon TWG input:
      1. Focused on Self Governance Tribes
1. Evaluation Feasibility
2. Title VI Expansion evaluation issues
3. Target and invite Tribes that are running 11 programs
4. National Tribal 477 programs
5. Originally planned for early in the project to help inform site visits and focus of inquiry.
6. TWG proposes moving to end of study to test draft findings
   
   Technical Work Group Comments:
   • Schedule small group discussions after the site visits instead of before like a Focus Group testing what you have found out.
   • You can do the Focus Groups with specific questions.
   • Are the six sites representative of the whole?
   • Wait until you have the preliminary findings in hand and then ask if these are consistent with the experiences of the 6 discussion groups.
   • If this was an evaluation of a project, which it is not, then one would have to go out in advance and ask for comments on the evaluation protocol. This study is seeking the feasibility of an evaluative study. The critical stage of this is the preliminary findings.
   • There is always an update at conferences. The discussion groups make sense at the end. A combination of Tribes considering Self Governance, TANF, Head Start, etc.
   • NCAI would be a good forum for Discussion Groups.
   • Tribes running 11 programs should be invited. We should use 477 programs as a good resource.
   • August through October is a difficult time for good conferences.
   
   
   Project Team Comments:
   • We will rework a draft schedule and format for the discussion groups looking at an Aug through Oct window.
   • This will likely require an extension of the contract.
   • May not be possible to cover agenda with 10 people in two hours unless the agenda was focused/limited.
   • We've scheduled into the contract 6 groups at 2 separate locations for 2-3 hours each. We also talked about doing this process in two different segments. Pre and Post.
8. Tribal Matrix
   1. TWG Recommendations for the Tribal Matrix:
      1. Project HOPE will correct and re-send to the TWG
1. Delete State Tribes
2. Add Federally Recognized Tribes that are missing (i.e., Havasupai, Tonto Apache, etc.)
2. IHS User population in addition to census data
3. Use footnotes to address limitations of data sources
4. Seek volunteers for participation in the site visits and group discussions at the April SG Conference
   
   Technical Work Group Comments:
   • Use user population data from HIS; it has already been reviewed by tribes.
   • Cliff Wiggins mentioned that the TWG can obtain user population with little effort for health programs.
   • Census data that is self reported doesn't mean anything in the context of who a tribe provides services to.
   • User population has limitations too.
   • Census data however might be used for other non-IHS programs;
   • We deal with our service areas differently.
   • FDI/LNF data is useful also and should be reviewed.
   
   Project Team Comments:
   • This data is used not as a benchmark, but more relative than absolute numbers.
9. Site Selection
   1. TWG Recommendations for Site Selection Criteria - the TWG was not willing to identify site selection criteria until it is clear what the purpose of the site visits will be and what the study questions will be for the site visits --- Issue addressed included:
      1. Cross section of tribes, sizes, demographics, locations, etc
      2. Recruitment process
      3. Governance profile
      4. FHDI/LNF data
      5. Regional disparity
      6. Visit those with the most experience
      7. Tribes must volunteer
      8. Self Governance Tribes with experience in the 11 Title VI programs will narrow the focus
      9. Size stratification
      10. Service Area demographics
      11. TWG needs to review study methodology first
      12. TWG will draft the site selection criteria for review
          
          TWG Comments:
          • During site visits, we are looking at data. We need to define the characteristics of selected sites.
          • We need to know what we're testing and why first.
          • Criteria for site selection should include volunteer sites, geographic distribution, multiple HHS programs operated, size/demographics of service areas, land based status, governance status.
          • Need to use IHS rather than Census user population data.
          • Questions need to be broader than health care as there are many different programs.
          • Need to indicate what the data limitations are and how the matrix data will be used.
          • Are the sites all going to be Self Governance?
          • We need to start with the questions before selecting sites.
          • Questions need to reflect 638 and what it brings to Tribes.
          • Tribes are very sensitive. They may feel that this study seeks to evaluate Self Governance.
          • This is about advancing Self Governance and not comparing them. This is not comparing one tribe to another. Rather, does Self Governance allow Tribes to be more successful or not and how do these Tribes do business.
          • We don't want this to be an impact on non-Self Governance.
          • Assessing data on services needs to be the focus, not comparisons.
          • The difference between why Tribes are doing what they are doing is more the question.
          • Mandated reallocation of funds by Tribes is not the issue, rather showing how Tribes match and exceed the dollars that are received is the issue.
          • There is an opportunity to move forward the Title VI efforts.
          • How will the money flow smoothly through the system is a key issue.
          • A good assessment can help us with legislation.
          • Need to emphasize levels of need and federal health disparity index.
10. Redrafting Communication Protocol
    1. TWG Recommendations for Redrafting the Protocol --- The redrafted protocol was eventually agreed to and signed between the Title VI Advisory Committee and the DHHS --- There were discussion within the TWG regarding this process:
       
       Technical Work Group Comments:
       • We need to define the relationship between Title VI Advisory Committee and TWG. Not all the tribes represented on the TWG are involved with the Title VI Advisory Committee, and it is not clear to them what this relationship is.
       • Why is the TWG approving or reviewing an agreement between the Title VI Advisory Committee and ASPE?
       • There is a responsibility of HHS to provide information/communication to Title VI Group.
       • There are members of the TWG on the Title VI Group and thus there is a linkage.
       • There is a responsibility to communicate with all tribes, not just those on the Title VI Advisory Committee. Will the Advisory Committee take on that additional role?
       • The Title VI Advisory Committee needs funding to continue the link and communication between the two groups. Protocol defines communication, but this will require funding.
       • The Title VI Steering Committee provided the start to this evaluation feasibility study. This protocol provides support to the TWG and the Title VI Steering Committee. There is a need to cover the costs to Sense, Inc., for facilitating the tribal communication process.
       • Those members of the Title VI Steering Committee have not been communicating properly with the TWG.
       • This document was developed before the TWG. It sets protocol for the study and not for the TWG. This is the Title VI Steering Committee telling the TWG what to do rather than vice versa.
       • This background information would have been helpful earlier. We had on the agenda Ground Rules and why we are here. Some on TWG originally viewed this as there was not support for the Title VI Steering Committee by the Department and that the Title VI Steering Committee was trying to get their support through the TWG.
       • The TWG should have a list of those on the Title VI Steering Committee and a list of their meetings.
       • The TWG requests to see a breakdown of the prime and sub-contracts and who does what and the percentage of the total used for communications. The TWG requests to see a breakdown of the dollars being used on this project.
       • This isn't about nitpicking, but rather dialogue. We on the Title VI Steering Committee don't have much control, but we do have an interest.
       • We wish we had more input into the contract negotiations to ensure success and efficiency.
       • TWG expressed concern about what is the role of Westat, Inc.. and concern about contractors who don't know anything about Indian Country doing Indian projects.
       • Two meetings of the TWG are not going to get this job done.
       
       Project Team Comments:
       • Contractors will be in communication with TWG. We also have tribal leader representatives from the NIHB and NCAI who are not at this meeting, but may want input on the protocol.
       • In principle we should support the cost of communication, but we have to know what we are talking about before we seek more funding.
       • Exclusively quantitative studies aren't appropriate. Management changes that are quantitative, not just health indicators. You can measure things like daycare, etc. over time for turnover.
       • We have to be careful not to deviate to far from the scope of work that has already been agreed to and signed between the contractors and the government.
       • The Steering Committee works at a higher policy level with DHHS. They will continue after the TWG ends. The TWG is advisory to the contractors on this particular study.
       • What can be accomplished is based on the workload and tasks. There has been much work that went into the project based on last year's budget.
       • The timing wasn't in DHHS' and the project expanded from its original proposal.
11. Next Steps
    TWG Recommendations for Next Steps:
    1. Myra Munson redrafted the protocol (See attachment A), which was signed and is posted on the ASPE web site.
    2. Contractors will redraft all materials to go out to the TWG by March 8^th
    3. TWG (tribal only) teleconference to be conducted to review redrafted materials.
    4. TWG will propose site selection criteria and comments on the redrafted materials by March 30^th
    5. Follow-up with ASPE, contractors and leadership to identify possible sources of funding to support Sense, Inc., tribal communication efforts.
    6. Review budget
    7. Formal TWG (all participants) teleconference to be conducted in April (to be scheduled).
    8. Six sites need to be identified after agreement on site selection criteria.
    9. Six site reviews in June, July and first half of August
    10. Next (not budgeted) TWG Meeting will be May 2^nd at Salt River, if possible. This meeting will be held to coincide with Spring Self Governance Conference at the end of April.
    11. 2^nd Budgeted Meeting will be tentatively targeted for September.
    12. Identify best opportunities to conduct small Discussion Groups at two locations (3 discussion groups in each location), ie., NIHB Consumer Conference, NCAI Convention, national TANF or Head Start gatherings, etc.,

Adjournment