A r c h i v e d  I n f o r m a t i o n

Volume I: Executive Summary (cont'd)

The following section contains brief summaries of selected findings at each of the 10 Regional Educational Laboratories. For each of the selected strengths or areas needing attention, excerpts from the synthesis report prepared by the panel chair are provided. These excerpts are direct quotations found in the synthesis report; ellipses have been used to indicate where text has been omitted. In a few instances, changes have been made to the text for clarification or to correct a grammatical error; these changes are indicated in brackets.

Where appropriate, excerpts from the written response prepared by the Lab have also been included. In some cases, Laboratories provided a broader, contextual response to the synthesis report rather than addressing every specific point raised by the panel. Consequently, some of the summaries do not contain a Lab response for each of the highlighted areas needing attention. All summaries include a list of other strengths and areas needing attention that were noted by the panels.

The issues selected for inclusion in these summaries represent some of the major findings reflected in the synthesis report and are in no way meant to represent the totality of important findings. Further, the number of findings highlighted does not necessarily represent the relative distribution of strengths and areas needing attention included in the synthesis report. Rather, the summaries included here are simply meant to provide the reader with a sense of some of the areas in which particular Labs excel, as well as some of the areas in which the panelists felt that special attention was needed for improvement. In some cases, panelists noted only concerns within the areas needing attention, while in others, actual recommendations were provided. Both concerns and recommendations are included in the following summaries.

The findings included in these summaries were selected collaboratively by DIR and the panel chair at each Lab. In all cases, Laboratories were found to have met their contractual obligations or were able to justify changes in the scope of work or in the deliverables schedule. Similarly, all Labs were found to have more strengths identified than areas needing attention.

For more detail on the evaluation findings, refer to the complete versions of the synthesis reports, Lab responses, and individual evaluation reports compiled by Lab in Volumes III — XII.

• Within the Region, AEL is developing products and services of unusually high quality. The products and services emanating from AEL’s operations appear to be enhanced by the knowledge of the applied areas by project staff, knowledge and reviews of the relevant research literature, and input and reviews by appropriate clients and other target audiences.

• A close examination of all these publications reveals outstanding examples of successful efforts to communicate Lab findings broadly within the Region and to provide timely information in a clear and easily understood manner.

• [T]he area of "utility" represents an area of major strength for AEL in terms of the products and services they provide to the educational community and policymakers of the four-state Region. As one principal from a QUEST project schools put it, "I can always rely on the AEL network to provide a continually high level of professionalism, and they have been continually helpful in finding resources when needed." AEL honors what it refers to as the "wisdom of practice" as it captures the experience of its clients in its product development.

• From the composition of the governing Board, to the membership of program and Lab advisory committees, to the development and revision of individual projects, products, and services, there is a consistent attention within AEL to focusing on customer needs within the Region.

• The Lab appears to operate in a genuinely collaborative, interactive, and supportive way with its various client groups. This focus on customer needs was evident in virtually all the documents reviewed and in interviews with a wide selection of project participants interviewed during the visit.

• AEL is clearly focused in its work on students and their teachers, and service to the field enjoys great emphasis across projects. Products and services emerge from the experience and articulated needs of regional customers.

• Each of the states appears sincere in its encouragement of local (rural) schools to respond to the state-mandated system of standards and assessments with their own bottom-up approach to implementation. AEL is deeply involved in this process of supporting local schools as they struggle to discover "locally appropriate" solutions to systemic reform initiatives.

• Several elements of AEL’s organizational structure are critical to making certain that resources converge in a functional manner. All of AEL’s key programs report to a single individual, the Program Leader, which allows coordination across programs to be facilitated on a daily basis. AEL’s Program Leader joins with program directors throughout the Lab as a member of the decision-making Corporate Leadership Group (CLG), headed by the Executive Director. In this manner coordination of programs across AEL is facilitated. In addition, a more recent structure places Resident Directors in each of the Region’s four states who meet monthly, and the organizational structure also includes a Policy Staff which targets services to key statewide policymakers to assist them in decisionmaking.

• On all advisory and governance committees there appear to be sufficient and balanced representation from all expected constituencies, including teachers, administrators, parents, policymakers, the private sector, higher education, and state education departments.

• The outstanding strengths of AEL are its clear vision, focus, and unwavering attention to its mission of serving the Appalachian Region over the past 33 years, and its leadership team headed by the Executive Director, whose tenure spans much of the Lab’s lifetime. He and his staff have unqualified support from a very committed and involved Board of Directors.

• The staff and leadership at AEL are clearly committed to the goal of bettering the life chances of the children in the areas they serve. This strong leadership and shared common vision, and the significant efforts being made to move toward that vision, help create another great strength of the Lab, its positive organizational culture.

• Strong regional focus
• Self-monitoring/evaluation above and beyond that required by OERI
• High utility of products
• Efforts in the area of Comprehensive School Reform Demonstration (CSRD)

• There was consensus among panelists that AEL needs to take a more proactive role in establishing itself as a national leader in the specialty area of rural education. One panelist referred to AEL’s "absence of vision" in the specialty area.

• Rural education dominates the Region and AEL’s attention, so much so that AEL has not well-defined rural education or clarified its programmatic and research strategies in rural-specific ways . . . This lack of clarity can present difficulties for those in rural areas in other parts of the country that do not share the same attributes as Appalachian rural areas.

• While maintaining its commendable regional focus, AEL should maintain a more balanced portfolio by providing special incentives for a larger number of staff members to achieve greater recognition for major AEL programs by making national presentations and publishing in refereed journals . . . AEL needs to emphasize an increased national awareness and perspective in all projects, from proposal to the completion stage.

• As AEL moves to expand its overall perspective from a regional to a national focus, it should also expand its understanding of "customer" to be more nationally inclusive.

• In pursuit of its exemplary service to the Region, AEL has put little emphasis on cultivating its national profile . . . The reluctance of the Lab to engage, systematically, in "marketing" or to pursue national recognition is regrettable because the nation is denied much of what it could learn from AEL, and because AEL does not receive the recognition and resulting resources that would accrue from that recognition.

Selected excerpts from the AEL response:

• Years of responding to RFPs have led to organizing AEL (and other Regional Labs) according to projects. This programmatic compartmentalization has resulted in making communications across programs difficult.

• The Review Panel found evidence that considerable AEL accumulated experience and expertise does not flow readily from one AEL program to another. Although some important steps are beginning to be taken to address this concern, the interactions among programs that do occur appear to be more serendipitous than to happen by design.

• Communication across programs is much less consistent than communication within project teams. Although there are obvious recent efforts being made to address this issue (for example, the recent staff retreat and the Learning Lab strategy), more attention needs to be given to keeping all staff "on the same page" in terms of lessons learned and strategies applied.

• Several panelists also recommended the need to minimize potential areas of duplication in the improvement of products and services by a more formalized process of intra-Lab interactions.

• There is agreement among panelists that communications within the organization appear to be somewhat fragmented, though not unusually so. The staff attributes many of the communication obstacles to years of responding to calls for proposals that have created programmatic compartmentalization. The panelists were made aware of a variety of formal in-house communication structures, including retreats, seminars sponsored by the Rural Center, an organization of program officers, and the recently established "Learning Lab," which, when combined with informal communications and a strong Labwide collaborative spirit, encourage communication across programs. Nevertheless, project leaders and project staff in their presentations rarely attributed ideas and strategies employed within their activities to other AEL programs.

Selected excerpts from the AEL response:

Other areas needing attention noted by the panel:

• Increased use of student outcome data and intermediate measures to assess impact
• "Engineering" view of scaling-up school reform products (i.e., developing a working model and engineering it so it can be produced in greater numbers)

• The LAB has benefited from the utilization of Brown’s management systems (e.g. budgeting, personnel, grants-management, and legal services) as well as some of the communications services . . . Efforts have been made to leverage Brown’s considerable intellectual resources in support of LAB activities and programs.

• The Board is a strong, guiding force, constituted by leaders in education, business and industry, and the public sector.

• Partners and alliances have contributed to establishing the LAB’s credibility in the region, both effectively and efficiently. The LAB has placed a priority on forming, sustaining, and extending strategic alliances and partnerships with education agencies such as the Chief State School Officers in the region, associations of school administrators, school districts, teacher professional organizations, and other networks.

• The State Liaison System of the LAB is a valuable source of needs assessment: The system effectively informs the LAB’s work.

• Customer needs are a major focus of the LAB, which is sensitive to and attempts to be responsive to the needs of its clients. Most of the input from the clients the Panel met during the on-site visit and from the materials reviewed, was highly positive, even laudatory . . . Without exception, the LAB staff members were complimented for their credibility, visibility, and non-threatening support provided to clients.

• The NEASC [New England Association of Schools and Colleges] accreditation review process has been revised as a direct result of the Secondary School Restructuring project . . . The Director of this NEASC project expressed her strong satisfaction with both the process and outcome of the LAB’s contributions.

• The LAB uses customer feedback to refine products and services. One example of this strength is the effective use of the State Liaison system described above. Increased communication and a high level of service to the field is a direct result of the system.

• The focus on cultural and linguistic minority students and in particular English Language Learners permeates the work of the LAB.

• The collaborative relationship with NABE [National Association of Bilingual Educators] in the "Portraits of Success" and with CAL [Center for Applied Linguistics] in the Nanduti Web site exemplify the well designed nationally recognized effort to improve the education of English Language Learners.

• The LAB is willing to conduct "in the trenches" applied research projects: As a consequence, existing and planned projects reflect significant R& D ideas whose anticipated results will be useful to the field.

• The LAB is conducting excellent research efforts that are significant, well-designed, informed by state-of-the-art research, and competently executed. For example, the Implementing Standards with English Language Learners — the Lowell middle school professional development project is an example of a school improvement effort that has been . . . conducted with a high degree of competence.

• Knowledgeable, capable, and dedicated staff
• Self-monitoring and quality assurance program
• Work addresses major policy issues
• Useful research reviews and application of that research to practice and policy
• Contributions in the area of comprehensive school improvement (e.g., "best practices" electronic site)
• Recognition that student achievement and other student outcomes are an important focus for programs of educational development and applied research
• Useful products that are widely used by educators
• Anecdotal evidence of positive student impact
• Prototypes for interactive collaboration supported by technology

• There is a concern regarding the efficient and effective application of technologies (such as the Internet and video-teleconferencing) that can be applied throughout the programmatic and management elements of the LAB. Documentation and assessment of the effectiveness of LAB programs can be supported to a greater extent by technologies, resulting in greater efficiencies and opportunities for cost reduction.

• Access to LAB materials and documents can be documented by obtaining systemic measures of utilization and calibrating production and dissemination in response to feedback . . . The concern here is to verify the utilization by multiple users and re-design documents to achieve the highest level of usage by clients.

• Disseminate publications and other products in multiple modes and formats, both "really" and virtually.

• The efforts of the LAB in all venues, and in particular in the specialty area, can be complemented with the commitment of broadly initiating and expanding collaborative activities utilizing the newly available interactive technologies . . . For example, activities including videoteleconferencing and streaming video can be accomplished via the Internet without requiring special facilities and with little additional costs to the host institution.

• The LAB is challenged to develop a comprehensive needs assessment plan…for meeting customer needs in the future…The comprehensive needs assessment can guide future planning for LAB strategies and priorities.

• Develop a comprehensive needs assessment plan and timely method for implementation. This plan can include multiple measures including, but not limited to, client satisfaction. Particular attention should be paid to . . . reaching out to help those who reside in localities who have not engaged the LAB in the past.

• Use the results of systematic needs assessment to inform strategic planning. This may include raising questions that deserve investigation, even though a specific request for assistance has not been received . . .

Other areas needing attention noted by the panel:

• More diversity among staff to reflect populations served in the region
• Increased dissemination of research to the national arena
• Higher priority placed on measurement of student outcomes to assess impact
• Development of specific plans to pilot, evaluate, and scale-up successful work
• Expansion of collaborative activities to broaden customer base

• LSS uses a data-driven, quality control system that includes collecting a wide array of data, and the Lab has a self-monitoring process in place to be able to plan and adapt activities in response to feedback and customer needs. LSS’s user satisfaction surveys include "event evaluations" (which determine the quality of customer satisfaction with external events sponsored by LSS), "tracer studies" . . . , "product satisfaction surveys" . . . , and "semi-annual needs assessment surveys."

• Reaction of the peer panel to the evaluation process and related documents indicates that the assessments are taken seriously. That is, satisfaction surveys, technical reviews, and tracer studies are done systematically and in abundance, and Lab staff seem to pay attention to the findings. One has the sense that there is a culture of continuous improvement in the Lab, and that the quality assurance process is an important element of that culture.

• Further quality assurance issues are addressed through ongoing procedures involving feedback from the Governing Board, Stakeholder Advisory Board, Technical Review Board, field-based collaborators, LSS clients, and self-assessments of Lab staff.

• Methods and programs, such as 20/20 data disaggregation method and CFL [Community for Learning], assist schools and districts in user-friendly ways to harness the strength of research in practice. An important strand of LSS field work is based on understanding how to build capacity among school staff in making intelligent choices in strategic reform, in creating more adaptable delivery systems, and to be persistent in maintaining change.

• Through this series of invitational conferences, the Lab brings in top caliber individuals–both researchers and practitioners–around issues of national, boundary-spanning importance. Because of its high visibility, the LSS National Invitational Conference Series program is under close scrutiny by its stakeholder groups. The publication of the proceedings in refereed journals and the review processes of their widely respected publishers, such as Laurence Erlbaum Publisher, are additional indicators of the quality of the series. Further quality is evidenced by the request for copies of conference proceedings by LSS stakeholder groups and requests for follow-up activities.

• Community for Learning was cited as an exemplary program in the November 1997 Congressional Report on the CSRD legislation. Indeed, high praise for the CFL model was a constant theme during many conversations with representatives from regional schools during the site visit.

• Because CFL is highly data-driven, LSS often ends up working with districts and states both to improve their collection of data and to help analyze the data. This goes well beyond what is actually called for in implementing the CFL model, but it provides an extremely valuable service for the district or state overall, with benefits that may go well beyond the CFL sites themselves.

• In sum, across LSS programs, user representatives characterize the LSS work within the larger constellation of available research and technical assistance as follows:

• The work is non-political, high quality, research-based, and not biased
• LSS fits a niche that the SEA, universities, and national reform innovations can’t fill
• LSS is having a direct impact on the classroom

• One of the first steps in determining whether or not the Lab’s work is contributing to student success is putting a sound model for measurement in place. In sites where CFL is in place, the Lab monitors changes in teacher behavior related to the degree of implementation of the CFL model and examines the target district’s data to measure student achievement progress.

• Schools in some of the nation’s most impoverished inner city areas have achieved positive results following CFL implementation. A study of the first year of implementation of five CFL schools in the District of Columbia which have been identified as among the lowest performing in the District found that teachers were making significant changes in classroom practice. The study also examined changes in student reading scores on the Stanford 9 and found that scores improved at all five schools, that program schools improved more than other elementary schools in the district, and that the districtwide ranking of program schools climbed considerably (one school jumped from 119^th to 46^th, for example).

• At a middle school in inner city Philadelphia . . . students have shown significantly higher academic progress than students at a control school. A follow-up study of students . . . reported that they had a significantly lower dropout rate than their high school peers (19 percent vs. 60 percent) and that 48 percent of them were performing at grade level in the eleventh grade compared to 26 percent of their peers. A similarly situated elementary school in Houston also witnessed improvements in student achievement, along with positive changes in students’ and teachers’ attitudes about their school.

• Another noteworthy finding in the pattern of progress in student achievement in math and reading is the gains made by students who scored at the top 20 percent across all D.C. CFL demonstration schools, which is reflective of the design focus of the CFL program.

Other strengths noted by the panel:

• Strong leadership team
• Leveraging of resources provided through relationship with Temple University
• Quality Assurance processes
• Research-based product and service development
• Large presence in the region with progress being made toward a national reputation
• Useful products and services that are used by customers
• National, regional, and state recognition for products and services
• Accessibility of products and services in a variety of media
• Products tailored to customer needs
• Strong mentoring environment and diverse staff
• Overall culture of the organization

• A great deal of the effectiveness of the LSS appears to rest upon the special skills, energy and relationships of the Executive Director. How can LSS institutionalize the many contributions of its Executive Director so that it can continue its success, even if she were to function in a reduced role?

Selected excerpts from the LSS response:

• For example, evaluation data for the CFL program was not available in two forms that would be helpful: (1) longitudinal profile data on the Degree of Implementation (DOI) at all CFL schools, and (2) outcomes data on student achievement at sites other than at the DC schools where CFL was in place. Fuller outcome/impact data will hopefully become available as the schools continue to participate in CFL. LSS is encouraged to place a priority on both longitudinal DOI profiling and student achievement outcomes.

• Most interviewees involved in school reform efforts talked about a wide variety of positive changes in teacher, student, and parental behaviors within their schools that are not as easily measurable or quantifiable as student scores on achievement tests, yet they are no lesser indicators of school improvement and student "success," broadly conceived. LSS can make an important contribution to the school change literature and to the "procedural knowledge base" by attending to those "other successes" and by learning how to document and give credibility to them.

Selected excerpts from the LSS response:

Other areas needing attention noted by the panel:

• CFL model values teachers as instructors and implementers, but seems to lack in valuing their capacity to reflect, articulate and document their experience
• Lack of systematic program evaluation for some implemented programs
• Increased user-friendliness of some published material (while still maintaining the subtleties of the research findings)
• Sustainability of projects

• To the extent that standards-based reform can be thought of as encompassing much of curriculum, learning, and instruction, the Laboratory has indeed developed a national and international reputation in its specialty area. McREL is currently providing services or having its products and programs used in places throughout the U.S.

• The Laboratory has developed and disseminated print resource materials, monitored and integrated research literature, provided extensive professional development opportunities, sponsored numerous conferences and workshops, presented at regional and national forums, and developed extensive partnerships with U.S. Department of Education-funded institutions and service providers.

• Standards are at the heart of comprehensive school reform. The work of McREL to make standards available, understandable, and usable by SEAs and LEAs is of tremendous assistance to the field. Comprehensive school reform strategies begin with standards. McREL uses its expertise and prominence in the field of standards to help states and districts implement reform.

• Alliances/partnerships help keep McREL grounded in the reality of the classroom and build long-term relationships. Such alliances allow McREL to leverage its resources while allowing states and local districts to leverage their resources as well. The Laboratory has established formal long-term, field-based collaborative action-research partnerships that promote systemic reform with six of the seven states in the region. In addition, it has developed less formal, shorter-term relationships with associations and schools both within the region . . . and outside the region.

• Formal collaborative partnerships that are responsive to local, state, and regional needs have been established in various configurations such as State Facilitation Groups, Regional Field Services Teams, and Collaborative State Action Teams.

• The Laboratory has cooperative working relationships on specific projects with nationally prominent institutions or agencies such as the Jet Propulsion Laboratory, Lockheed Martin Astronautics, National Education Association, Phi Delta Kappa, the International Reading Association, the National Council of Teachers of English, the American Psychological Association, and the New York Times. The Laboratory and its partners such as ASCD [Association for Supervision and Curriculum Development] have published numerous Laboratory-developed materials.

• The Laboratory regularly collects feedback from clients through diversified methods.

• Most of the feedback from clients as well as from OERI has been highly positive, with several interviewees reporting that the Laboratory was very responsive and really met their needs . . . By conducting regular needs assessments, the Laboratory remains informed of the type of support schools, LEAs, and SEAs need and want.

• Clients interviewed by the Panel suggested that they considered the services to be of high quality, in large part because McREL staff were willing and able to tailor what they did to fit local needs. In some cases, the clients felt that McREL staff went out of their way to customize their services.

• By trying to be current and poised to help schools on short notice, the Laboratory has sometimes been able to serve as a resource much needed by schools under pressure to respond to mandates and initiatives.

Other strengths noted by the panel:

• Leveraging of funds and resources
• Multi-talented, motivated staff
• Quality assurance process in place
• Regular needs-sensing in the field
• Frequent collection of client satisfaction data
• Useful products that are widely used
• Annual evaluation reports
• High quality products and services

Ensure that products and services are solidly research based

• So great is the reputation of McREL regarding standards that it must now be more circumspect in its pronouncements about the subject. Being among the most highly reputed experts on the matter gives the Laboratory great credibility and influence.

• During the most recent Congressional debate about funding research, there were complaints that much laboratory and research center work was simply ideology. McREL needs to be sensitive to this issue and make sure that its work is not speculation or ideology masquerading as research-based findings.

• To more fully serve the emerging measurement and research needs that defining and implementing standards have created, McREL should bring its technical staff resources into the research and assessment planning and evaluation collaborations more frequently. One panelist suggested that if current staff are overcommitted and hiring more permanent staff with expertise in measurement and research is difficult to carry out, then a possible alternative is to form a long-term national technical advisory group that convenes quarterly.

• When McREL external reviewers raised significant technical issues such as low response rate with regard to a survey conducted by the Gallup organization, such concerns were dismissed, unconvincingly to the Panel, by Gallup . . . Despite the various flaws noted by the external reviewers of the Gallup-conducted study, McREL plans to have Gallup conduct similar surveys to obtain industry’s views and teachers’ views.

• Despite concerns expressed about the quality of the method used in the aforementioned Gallup survey, McREL published the results with what the Panel views to be an overly bold title, What Americans Believe Students Should Know: A Survey of U.S. Adults.

Selected excerpts from the McREL response:

• By permeating the Laboratory’s activities, the concern for standards-based reform seems to be more integrated than would be the case if standards were seen as a more discrete entity. Although the Panel found this permeation to be a strength, at the same time, it is concerned about over-reliance on standards as the (only) way to successful school reform.

• Standards must be embedded in a larger view of school reform to improve student performance. McREL should reflect on its view of standards and how they fit in with assessment, professional development, school climate, finance equity, and other important aspects of school reform.

• McREL’s focus on standards may limit its ability to respond to various other customer needs unrelated to standards . . . It may make more sense to treat the standards-based approach as one of several different approaches that McREL can propose and facilitate.

• [T]he Laboratory has not shown concern for what may be the negative effects of a headlong implementation of standards-based reform on students, teachers, and state school systems. Many of the states, and many schools within the states, have historically been very successful educationally. Care should be taken to not risk what already works.

Selected excerpts from the McREL response:

Other areas needing attention noted by the panel:

• Improved methods for self-monitoring
• Increased ethnic and gender representation on the board and in staffing
• Greater focus on student outcomes or changes in teacher behavior
• More reliance on external peer review using reviewers with technical expertise
• Measures of meeting client needs other than client satisfaction
• Lack of baseline data to allow for monitoring change/effect over time
• More rigorous, systematic collection of evaluation data
• Expanded efforts to provide culturally and linguistically appropriate products and services

• NCREL has a strong focus on the use of technology to support instruction and learning . . . the Lab focuses its energy on harnessing the strength of technology for design of user-friendly products and processes, for dissemination on a broad scale in a cost-effective manner, and for broad, open access to information through technology.

• The publication Plugging In is a successful, widely disseminated product devoted to choosing and using educational technology. The indicators of engaged learning are one of the real contributions that NCREL is making to the challenge of integrating technology into the curriculum.

• The Lab has extensive projects within its region, in partnership with others outside the region, and in partnership and coordination with other Labs. The DOD [Department of Defense] Education Activity for professional development in technology applications sends a strong message of support for the Lab’s sound national reputation.

• The quality of the products and services provided by NCREL is generally high . . . (One reviewer commented that the wider the intended dissemination and the more the product focused on technology, the higher its quality tends to be.) Certainly, there is a wide array of printed and web-based materials addressing a broad range of interests and target audiences . . . Certain products produced by the Lab seem to have "star" quality and attracted particular interest among panel members.

• Lab staff appears to do everything they can to ensure that their products are useful, ranging from careful needs-sensing as the basis for developing products and services to modifying and customizing products and services based on feedback from the field.

• In the area of print materials, NCREL has an impressive library of works. NCREL’s print library runs the gamut from draft academic papers to bound training materials, to glossy, magazine style reports. The overall quality of these materials, as reviewed by this panelist, appears to be very high.

• Using the concept of engaged learning, NCREL has developed a number of strong products and processes. CTLC [Center for Teaching, Learning and Curriculum] and CSCD [Center for School and Community Development] have focused in particular on the teachers’ role in delivering instruction and in understanding how students use that delivered instruction to learn, with products such as the Engaged Learning Safari offering educators clear benchmarks for understanding engaged learning and with potential for scale-up inherent in the product.

• Quality Assurance procedures are well-documented in the NCREL Quality Assurance and Evaluation document provided for the panel’s review . . . An overall unifying theory drives the self-monitoring process. This framework is a customized TQM cycle with four major parts: Assessing Needs and Setting Goals; Designing Plan and Evaluating Alternatives, Implementing Plan, Evaluating Plan and Renewing Efforts. NCREL uses this framework to drive the collection of data and the refinement of activities across the spectrum of work.

• As outlined in the NCREL Quality Assurance and Evaluation Document, external feedback is solicited from a variety of sources: through partnerships with R&D organizations, regional feedback from key client groups, and regional surveys of educators and agencies.

• In addition to documenting the review process for individual projects, NCREL undertakes a portfolio review with its Board of Directors. This review allows the Board members to review NCREL’s entire portfolio of products and services, organized by the four centers.

• Leveraging of funds and staff to support the Laboratory scope of work
• Co-development of products and services
• Usefulness of products and services
• Focus on customer needs/"issue scanning"
• Work in comprehensive school reform

• Deliberately shift focus for the last part of this contract to scale up issues of both delivering products and services out and issues of consumer use on a broad scale. Focus deliberately on the products and processes already developed or firmly underway, looking in great depth and with a researcher’s eye at how to move them to scale by increasing their potential to connect with consumers in context of the practitioner’s reality.

• An area of concern is the sustainability of NCREL’s impact after it pulls out of a project. NCREL staff spoke about the challenges associated with the movement of products from the Lab to the field.

• The "lessons learned" from intensive sites have to be used to inform the work on scale up. Therefore, a major recommendation focuses on the need to make the link between the intensive work and scale up.

• Various obstacles in rolling out services on a larger scale have been identified by NCREL and their constituents whom the panel interviewed. NCREL staff spoke about the difficulties of working with intermediate state agencies in disseminating training to their teachers.

• The issue of scale up is a thorny one. NCREL can make a tremendous contribution to this literature and to the systemic school reform movement in this area . . . the "collective wisdom" of the organization could be more utilized in regard to scale up.

• Given that projects cut across centers, it would seem important to have very well-established systems of communication. For example, the theme of engaged learning was identified in the presentation of the Technology Center as one of the most powerful "big ideas" to emerge from their work. Yet, when staff from CSCD were asked about the indicators of engaged learning, they were unable to respond.

• The organizational structure, while flexible may lead to stress within the organization as staff begin to have too many masters; potentially wear too many hats. Diffusion of treatment is an issue. Perhaps a new center devoted to training and delivery could increase fidelity of transfer, but this would be very expensive.

• Rethinking the structure. This . . . should be based on three questions: (1) does the structure reflect the need to "put the pieces together"? (2) does the existence of a center called Scaling Up actually contribute to the success of doing scale up? (3) does the center structure really reflect the priorities of the Lab, including technology and professional development?

• The structure issue is complex because it reflects not only the priorities of the Lab, but the way they see themselves. My impression is of a relatively traditional hierarchical organization, which lacks the elements of cross-fertilization of ideas that create a learning community.

• Revisit the organizational structure of the Lab. The 4-phases of implementing projects require major interdepartmental efforts; NCREL itself has described how different departments interface around one project. What I heard from the presentations, however, was a process by which centers and departments became involved in projects in a sequential, rather than an interactive, manner.

Other areas needing attention noted by the panel:

• Lack of hard outcome data demonstrating improved student outcomes
• Lack of questionable methodologies for data collection in some areas
• Practical application/implementation of some products
• Need for an increased use of literature and research based approaches
• Need to be selectively strategic

• NWREL has an outstanding Director and a highly qualified staff dedicated to the development, production and use of excellent resource-based products and services. Its reconfigured governance board is strong, hard working and committed to NWREL’s present and future.

• Program planning and management are definitely among NWREL’s many strengths. The structures and processes used to guide planning and operational decisions have been reviewed and refined during this contract period, and include needs assessments, implementation plans, advisory committees, monitoring and evaluation.

• The Lab’s Institutional Strategic Plan is dynamic, not static: it is a working plan.

• NWREL’s careful attention to the aspects of managing a complex organization support and enable this area of work in educational reform to be successful. To recap a few of the "basics": staff at every level is hardworking and capable; the Board of Directors has been "renewed", reinvigorated, refocused during this contract period; planning processes are in place to ensure Quality and Utility; institutional planning includes strategies for action; and, needs sensing is taken seriously. These fundamentals of NWREL’s infrastructure underpin successful results.

• The planning, development and implementation of high quality products and services is, without a doubt, one of the hallmarks in this Lab’s long history of providing educational services in its region and throughout the country.

• The panel agrees almost unanimously that the Lab’s attention to developing high quality products and services exceeds expectations. It has generated a reputation for being prolific in developing high profile products and services that are in demand at both the regional and national levels.

• Services and products are designed to be user-friendly, and consideration is given to the resources, such as technology, available to teachers, trainers, principals.

• Product and service materials and interactions are available through several modes, for example: print, web sites, videos, CD ROMS, and in-person workshops.

• NWREL uses many mechanisms to obtain internal and external assessments of their work.

• NWREL has a well-developed three-level standardized system in place to assure quality of products and services.

• Panel members offer positive comments on NWREL’s annual Institutional Evaluation "that gives, in essence, a quick status report on Lab operations and accomplishments." Although this type of evaluation is not required by contract, the Lab conducted external evaluations to gain additional feedback.

• The Lab uses external panels, expert panels and consultants and peer review in this R & D area; it conducts action research in "real-world" settings; and it develops high quality products and services as indicated by users. Products and services of the Lab are used regionally and nationally and data and other feedback from users shapes the improvement of products and services.

• NWREL uses data to improve and expand use of products and services. Examples of revisions in products and delivery of services in response to user data are readily available with OTE [Onward to Excellence] and other products.

• Leveraging of funds and resources
• Successful alliances, in the region and nationwide
• Action research conducted in real-world applied settings
• Products and services used regionally and nationally
• Impressive rapport between staff and their users

• Continuous focus on current and future needs
• Products and services focused on student success
• Extensive and accessible work on comprehensive reform

• High potential for policy impact
• Regional and national reputation in specialty area

• Expand services and products to engage broader audiences. Audiences here means, for example, educators/parents where schools are based on other than behaviorist based learning programs, or other theory based programs, for example, constructivist based programs. "Audience" also includes community members, teachers, non-English speakers.

• Expand efforts to provide culturally and linguistically appropriate products and services to widen the customer base.

• A primary concern here in regard to increasing outreach and scaling up, is the need for all technical assistance and dissemination organizations to increase efforts in reaching educators, parents, students who are difficult to reach, as partners in their research, development and dissemination work.

• NWREL should reach out even more aggressively to the Laboratory Network Program, the alliance of seven Labs working with NWREL on this specialty area, and deans of the colleges of education in the northwest. Broader and deeper collaborative arrangements should be encouraged and facilitated over the next two years.

Selected excerpts from the NWREL response:

• NWREL should encourage and support users of products and services to collect and analyze student achievement data. "Encourage" as used here also means "enable": through expert assistance; through NWREL and/or other finances or arrangements with a district or state agency; through teacher and other LEA training in evaluation processes.

• The panel agrees that improved student success is important to a Lab’s work and further agrees that the influence of a Lab’s contribution to improved student success although difficult to determine, needs more work . . . A great deal more work needs to be done in this entire area — by everyone, not just NWREL.

• The Mississippi Weyerhauser/NWREL sites for OTE and OTE II lack significant documentation in the materials provided to assume that the fifteen-year-old OTE project has made substantial improvements in student achievement. Rather, a case can be made for an improved research design so as to ensure that any gains made using either OTE I or OTE II are substantiated within an accepted design framework.

Selected excerpts from the NWREL response:

Other areas needing attention noted by the panel:

• Representation of parents, students, and teachers on the Board of Directors
• Staff development opportunities
• Improved inter-Lab sharing of products and services

• Improved methodologies for self-monitoring
• Increased involvement of higher education institutions in the use of products and services
• Increased outreach and scaling-up efforts for school renewal
• Sustainability of projects
• Collaboration efforts and joint ventures with other Labs and relevant entities (in specialty area)

• All Lab program areas have local capacity-building modules built into them. Interviews with officials from the entities confirm that this capacity building is, in fact, occurring.

• To a great extent, the focus of the Lab is on developing the capacity of its customers to meet their own needs, using the services of the Lab without developing a dependency upon them. These processes engage the customers directly, and often intensively, in the service being provided.

• The overall approach to working with education entities appears to have been "comprehensive" in that it focused on educational systems and on building capacity to improve instruction from the organizational to the instructional levels.

• It appears that the overarching approach to service delivery by the Lab is toward meeting customer needs. A great deal of the Lab work engages the customers as partners in the service-delivery process.

• Lab has a number of routines built into its service and governance processes that assure that systems of services are "customer driven."

• Once an approach has been selected to address a need, the general Lab approach to working with clients–work with clients, not for clients–has a built-in assurance that the interests or perceived needs of the customer are foremost throughout the process. The Lab operates almost exclusively so that customers are partners in planning, delivering, and receiving services. The "customers" influence the services from initial planning through use of the results.

• PREL’s products are culturally and linguistically appropriate for the entities it serves. This means that customer service is real, not a surface imitation of service. Furthermore, the Lab’s products are carefully constructed to have a level of detail that permits use by all educators across the region, whether minimally educated (for an educator) or well educated.

• PREL, operating at 25 percent of the budget it proposed for its specialty area, has made significant contributions to the national understanding of Pacific language and culture and of issues of language and culture generally. The Lab has formed alliances with national professional organizations that recognize its expertise in these areas.

• PREL is "breaking new ground" in research on the impact of language and culture on teaching and learning, and in establishing and maintaining productive working relationships across multiple political jurisdictions and political boundaries.

• With an original focus mainly on work within its region and specialty area, the Lab has begun to provide leadership in working with SEDL and LAB, the other RELs that share its specialty area. As a result, PREL is beginning to develop a national reputation in the area. It is doing this both by producing good work that clearly addresses issues and by rejecting "quick fixes."

• The Lab has engaged educators from throughout the region in planning, staff development, research, and materials development in learning and reading a second language, and it has begun steps to disseminate results throughout the region and through the other RELs with related specialty areas. An important amount of language materials has been produced in the several major languages of the region, and teachers are being trained to use these materials in instruction.

Other strengths noted by the panel:

• Commitment to quality
• High quality staff and strong leadership
• Concern for its own organizational effectiveness
• Leveraging of other resources
• Reputation and respect in the region
• Quality assurance process which includes extensive internal and external review
• High level of customer satisfaction
• Large number of partnerships throughout the entities
• Utility of PREL’s services
• Several vehicles for broad dissemination
• Continuous and sustained needs-assessment activities
• Frequent contact with constituents
• Culturally and linguistically appropriate products
• Assistance to entities engaged in comprehensive school improvement
• High intensity services of sufficient duration for important results to be achieved

• The original proposal outlined a mission that was very broad in coverage and not definite in its statement of expectations. Furthermore, there is little about the statement that distinguishes PREL from any other agency that has a broad purpose to improve schools, schooling, or student learning.

• While those original mission and goal statements were approved by the funding agency and maintained without much change during the first three years of the current contract period, they probably could serve the organization better if they are modified to reflect what actually is expected to be accomplished, to reflect the uniqueness of PREL, and if they are expressed within the context of a strategic plan.

• [T]here is an apparent need in the organization, indicated by its own self study and by difficulty responding to questions raised by the panel, to revisit the organization mission and goals to consider whether they serve well the needs of the organization.

• It seems imperative for Lab decision-making and planning that it elaborate its strategy for capacity-building, including identification of targets within its audiences and specific capacity benchmarks from the highest decisions to classroom practices.

Selected excerpts from the PREL response:

• [N]eeds are mainly identified by some form of expression, often through questionnaires. Adopting a strategic approach to meeting needs of the region will require the Lab to be strategic in identifying needs and setting priorities. Some form of generalized "environmental scanning" process, matched with serious analyses of the "root causes" of conditions revealed, should be considered.

• [S]ome panel members thought current needs-assessment processes and reports are not as precise or sophisticated as they should be. Also, at least one panel member thought that classroom teachers and their ideas are not represented adequately in the processes for assessing needs.

• Consider adopting a variety of processes for needs identification, including processes such as "environmental scanning," that allow the Lab to be strategic in addition to being responsive in determining the needs that become organizational priorities. This recommendation is not intended to imply that the Lab should discontinue using its "customer-driven" system for determining the needs it will serve. Rather, it is intended to encourage the use, also, of "large-view" approaches to setting priorities . . .

Selected excerpts from the PREL response:

Other areas needing attention noted by the panel:

• Process for resource allocation
• Capacity of the Lab to continue delivering services in-person (heavy staff burden)
• Quality assurance process for services similar to that used for products
• Identification of Lab customers
• Improved customer feedback systems
• Results-management system
• Standards for juried research
• Increased impact of PEC [Pacific Educational Conference] on regional and Lab purposes
• Careful consideration of the opportunity cost of each product or service
• Direct, objective evidence to document improvements in student achievement
• Systematic, formalized methods for internal and external evaluation

• Shortly after his appointment in 1997 and with the support of the Board of Directors, the current CEO initiated a major internal review of all SEDL programs…As a result of the internal review and compensation study, a major restructuring effort was completed of programs and staff positions. This effort has enabled SEDL leadership to begin to realize its vision as a client-responsive, future-oriented organization.

• In order to get board members more involved in the work of SEDL and enhance their capacity to advise the staff, the current CEO asked each board member to choose one of the six goal areas for a more intensive examination. Board members were invited to "research" the goal area, visit sites, and, at the end of a specified period of time, make presentations of their findings to the full board. Overall, the relationship between the Board of Directors and SEDL appears to be excellent.

• Two management councils — one for SEDL as an organization, the other for the REL — have been established. Both meet twice monthly, the first focusing on the overall management of the organization and the second focusing specifically on the work pertaining to REL contract. Task forces are established to address specific issues or problems (e.g., the compensation study, facilities remodeling, and the performance appraisal system).

• SEDL is very client-oriented with respect to its overall operations, its work in intensive implementation sites, and its work with policy makers.

• SEDL carefully distinguishes among and attends to the different audiences and different purposes for their products (e.g., those for training, those for information dissemination, those for use as resource materials). Revisions based on the results of pilot and field tests often lead to site-specific modifications that increase the likelihood that the programs will be implemented successfully.

• SEDL regularly determines client needs, both formally and informally.

• All clients interviewed appreciated the follow-through assistance provided by SEDL staff as well as the scope of the relationship with SEDL staff members. Statements such as "they gave us direction," "they helped us develop a vision for our school," they had an impact on the quality of instruction and collaboration among teachers that aroused interest of others because of the excitement of students and teachers" attest to the impact SEDL has had in the intensive implementation sites.

• SEDL recognizes the critical need to monitor its services and products in order to optimize its value to the region. A major strength, discussed earlier, is the conduct of the internal review process that led to restructuring.

• A second strength is the use of a quality assurance (QA) process within the Laboratory. SEDL’s QA process, described in its original proposal, was fully implemented as of March, 1996.

• A third strength is the use of feedback from clients and customers to improve the quality of products and services.

• High quality products and services
• Satisfied customers
• Multiple ways of sharing information
• Focus on "whole school" change and improvement
• Use of the Regional Policy Analysis and Advisors Network to support comprehensive school improvement

• Talented and dedicated staff
• Site work approached with flexibility and adaptability
• Effective field-based developmental model

• Presence of comprehensive evaluation plans

• Two leadership positions remain vacant: the program manager for REL Goal 2 and SEDL’s specialty area, language and culture diversity, and the program manager for the Office of Evaluation and Policy Studies . . . SEDL’s executive leadership acknowledges the importance of both positions. Moreover, they are critically aware of the need for leadership in language and cultural diversity for this Laboratory in this region. Thus, these positions should be filled as soon as possible.

• The vacant leadership positions in the specialty area and in evaluation must be filled expeditiously, with SEDL staff seeking creative solutions to this staffing problem.

Selected excerpts from the SEDL response:

• The panel members understand that the major thrust of SEDL’s work is professional and/or community development. However, there is a long-term need to evaluate school, community, and teacher development in terms of the influence on students.

• Although the panel believed that the programs and services could potentially lead to improved student success, there was a concern that insufficient attention was being paid to documenting improved student success. Stated somewhat differently, although it is expected that students should be more successful as a result of participating in SEDL’s programs, student success should be not be assumed; rather, it must be directly determined. Thus, SEDL must maintain vigilance in collecting and analyzing student success data.

• According to the clients interviewed, the need for [student success/impact] data is a "fact of life" in today’s world. Such data would enable them to secure additional support for the SEDL programs and services, both political support and financial support.

• It is unlikely that SEDL has sufficient resources to collect and analyze student success data at every site for every program. Thus, the panel recommends that SEDL should assist personnel on-site in the identification of relevant data, the collection and analysis of those data, the interpretation of the data, and the use of the data to make informed decisions about programs and, ultimately, about students. In some cases, SEDL will likely need to build local capacity in this area.

Other areas needing attention noted by the panel:

• Lack of diversity of SEDL staff
• Insufficient collaboration across programs and goal areas
• External review as an integral, required part of the QA process
• Generalizability and scaling up efforts
• Need for building and disseminating knowledge
• Need to make SEDL synonymous with language and cultural diversity
• Outreach on the regional and national scene

• Effective connections operate from the state policy level, to the governance level, to the implementation level. State level analysts offer the Lab strong partnerships with potential for direct impact on policy.

• With highly trained policy analysts advising top state officials in every state, SERVE has the ability to impact policy formation, while with its strong professional development and training infrastructure created by program-level personnel, the organization can also impact implementation in a major way.

• This arrangement has afforded SERVE a continuous presence in state policy circles, while providing a conduit of information on evolving policy priorities.

• Despite distance and diversity of state contexts, the customer’s voice is heard at SERVE thanks to the remarkable infrastructure the REL has created.

• The Lab is strongly conscious of customer satisfaction and engages in a continuous process of monitoring in order to more effectively serve its diverse customers.

• The Lab uses customer feedback such as Delphi techniques and focus groups.., surveys and informal networks to determine customer priorities and responds accordingly, tailoring products, training, and other services to the intended audience, and refining them as needed. The REL has documented a close match between what the Lab thus determines its actual and potential clients need and want and what it does programmatically.

• Utilization statistics . . . testify to a high degree of regional awareness of the Laboratory and the popularity of the full range of its products in all their variety and scope.

• [C]ollaboration primarily for conducting special projects of interest to individual states, is characteristic of SERVE; there are few activities that the Lab seems to carry out solo. This is the hallmark of a strong institution seeking to maximize its effectiveness and stretch its dollars.

• SERVE organized a consortium of superintendents and school leadership teams to share ideas on how to build school cultures for long-term school improvement efforts.

• The LNP Assessment Toolkit for Professional Developers is a major example of cross-Lab collaboration nationally…A mature product and part of SERVE’s [Assessment and Accountability program], it is employed by the Laboratory in a number of venues.

Other strengths noted by the panel:

• Coherent and sustained program of work
• Committed Board
• High quality products and services
• Products and services that are useful to and used by customers
• Programs that are consistent with the broad education focus of the states in its region and the nation as a whole
• Programs that uniformly do good things and accomplish much for children
• Highly evolved quality assurance processes
• Work with schools on comprehensive school reform
• Regular and ongoing evaluation
• Organizational reliability
• Continuous and interactive engagement of customers

• Improve the research base from which the REL derives its programmatic thrusts and seek exposure in the field--both to get input from researchers and from content experts currently working elsewhere on relevant topics, and to share with the wider audience of peers and practitioners what SERVE is doing.

• While SERVE has become visible in its region, little has been done to assert its presence in national research and development settings, e.g. AERA, nor has much been published in refereed journals or more popular vehicles like the Phi Delta Kappan.

• For planned interventions, reach beyond the region to ensure they reflect the most current thinking and research in the field. Use of outside content experts, not necessarily from the region (depending on the issue), is critical to ensure accuracy and timeliness of information being imparted. Also, by consulting more outside reviewers SERVE might be better guided to build on work done elsewhere rather than create duplicative products.

Selected excerpts from the SERVE response:

• After meetings with staff, some members felt that obvious "content connections" are not being made, nor are there structures in place to encourage staff to discuss their activities so as to promote collaboration and coordination. For instance, while there would seem to be a content connection between the alternative assessment school projects and the Senior Project, a distinct alternative assessment, the panel heard no evidence that one informs the other.

• Similarly, the Assessment, Accountability and Standards Program seems to operate in relative isolation from the School Development and Reform Program, except for joint planning of a conference for low-performing schools.

Selected excerpts from the SERVE response:

Other areas needing attention noted by the panel:

• More formal and rigorous evaluation process for products and services
• Consistent, rigorous monitoring of student achievement and student success data
• Critical analytical processes to learn from programmatic failures
• More visibility throughout the region
• Greater emphasis on purposeful selection of participating districts and sites to ensure variation in key demographic and contextual factors.
• Strength of specialty area (early childhood education)

• WestEd appears to be well-managed, and much work has been accomplished to date. One of its major strengths, in fact, lies in its management structure. This structure is supported by a knowledgeable and active board of directors and WestEd’s close relationship with its OERI program officer. The latter factor allows for sufficient flexibility so WestEd can provide timely and responsive service to its clients and appropriately regroup when difficulties arise and well laid plans must be revised.

• The leadership has made focusing on student success a virtual mantra in the WestEd organization. The issue of student success, for instance, undergirds the five framing questions which WestEd has begun to use throughout the organization to critique its work.

• The WestEd board members we interviewed during the site visit were clearly committed to the Lab and its mission. The testimonials evidenced detailed knowledge of WestEd’s work . . . We also saw evidence of structural arrangements which made it possible for committed board members to exercise meaningful oversight and have a significant impact on policies and practices within the WestEd organization.

• A further strength is apparent in WestEd’s knowledge of its region and its region’s needs. Extensive needs assessment procedures are in place, including a highly effective strategy of drawing upon first-hand knowledge of its board.

• To a large extent, WestEd uses an informal, but highly effective approach to needs assessment. A key actor in the process is the state liaison who spends considerable time in his or her designated state attending meetings and talking with key stakeholders.

• There was ample evidence that WestEd translated good intentions into meaningful action. WAC [Western Assessment Collaborative] clients brought to the Lab office for the evaluation site visit, for instance, emphasized that they were continually helping to shape and refine the work of that cutting edge initiative. There also was convincing evidence that the Lab continually attempts to get feedback from customers of its other programs about how well its programs and strategies are working for customers.

• The customers we interviewed certainly found WestEd’s services useful. Although this evidence was primarily in the form of testimonials, the testimonials normally included a great deal of specificity about what happened because of WestEd’s interventions.

• WestEd’s specialty area of assessment and accountability is recognized nationally. Staff members have provided technical assistance to the National Skills Standards Board and to the Arizona Governor’s Division of School to Work, assisted in the development of assessment materials in collaboration with a number of other organizations supporting reform including local school districts, and worked with various groups involved in activities related to assessment.

• Particularly laudable is the work that the specialty area staff performs in conjunction with other programs within WestEd. Staff members work with programs within the REL on assessment issues and collaborate with other projects in the joint development of products.

• WestEd also serves as the lead REL in its specialty area for the Laboratory Network Program; this leadership has, among other things, resulted in the creation of an updated version of the assessment toolkit. Staff members have conducted "training-of-trainer" institutes centering on this toolkit, both regionally and nationally. In addition, staff members have presented workshops and presentations related to WestEd’s specialty area to national research and evaluation groups.

• Work on systemic reform
• Detailed procedures for quality assurance (QA) of products and services
• Quick response to emerging needs
• High quality products and services

• Innovative responses to the "scaling up" expectation
• Leveraging of external resources
• Impact at the school and school district levels
• Valuable and at times strikingly novel insights about the reform process in education
• Generous use of the internet to develop and disseminate products and services
• Products and services that help policymakers and practitioners implement standards based reform
• Widespread access to information on best practices

• At a more general level, the panel recommends more systematic evaluation efforts as the Lab begins the final two years of its REL contract. Whenever possible, student achievement measures should be part of this evaluation work. When such measures are not appropriate, other convincing indicators of impact should be employed and a compelling case for using these indicators — rather than measures of student achievement — should be presented.

• In fact, to date, for all of its talk of the importance of focusing on student success, WestEd has collected little if any student achievement data, at least not in the intensive research sites used by the WAC signature program . . . By design the WAC "treatment" varies from site to site and particular sites are not so much intervening variables as significant developers of the "treatment" which influences--or fails to influence--student success. WAC, however, could still attempt to document systematically (1) the impact of its work on schools and school districts and (2) the impact that transformed districts have on student achievement. At the very least, such systematic documentation might reveal the circumstances under which WAC-like intervention are likely to produce positive results and when other, possibly less collaborative, strategies may have a greater chance for success.

• At mid-point in its current five-year contract, dissemination efforts require more attention. Not only what is to be disseminated, but how to most effectively reach the most relevant and broadest array of audiences should be a concern. Publication in major journals needs to be increased, not only as a way of validating WestEd’s work, but also because such journals represent important forums for dissemination.

• There is also a need to document and disseminate information about how WestEd staff members play the support service role and to describe what the various strategies employed by WestEd reveal about supporting educational reform. To state this another way: the WestEd staff has developed a great deal of craft knowledge about supporting reform efforts; this information should be documented and shared.

• WestEd still needs to document more systemically the impact of some of this work and as well as the impact of many of its other initiatives. It should also put more emphasis on disseminating its work through traditional mechanisms such as practitioner and scholarly journals, in addition to using its own publications and the impressive technological forums it has developed for dissemination purposes.

Selected excerpts from the WestEd response:

Other areas needing attention noted by the panel:

• Intensified scope of work in the area of Language and Cultural Diversity
• Representation on the Board
• Consistent implementation of QA mechanisms
• Technical quality of WestEd’s formal evaluations
• Clarification of target audience for selected publications
• The soundness of performance indicators
• Scalability issues
• Equity across states in R&D site selection

• Efforts to reach a broader audience

Volume I: Executive Summary (Part 1)

Volume II: Study Design and Methodology