IV. Research Activities

The issues I have raised so far do not always constitute well-defined research activities; rather, they are questions that need to be answered through various kinds of information. In this section I briefly outline the kinds of research implied by the three previous sections, the issues each kind of research could address, and some of the major problems in designing the research. The detailed designs for research must await another time and place.

In considering the research that NAVE can undertake, its timetable must be carefully considered. If NAVE is to report to Congress by July 2002, and requires the first half of 2002 to evaluate all its findings, and if researchers therefore need to complete their reports by the end of 2001 and require the fall of that year for writing, then at best data collection can take place between summer of 1999 and summer of 2001. But interim state plans were due in April 1999, and many states will not have definitive plans until April 2000; the school year 2000-2001 will be, in effect, the first school year under Perkins III, though it will be the last year of data collection for NAVE. Therefore research on changes that Perkins III has introduced-particularly those having to do with accountability-must rely on plans and perceptions rather than steps already undertaken. Similarly, the participation of PSOE institutions with WIA and adult education must rely on reports of early planning stages, since WIA need not be implemented until July 1, 2000. Fortunately, this timing problem is largely confined to accountability and WIA since in other important issues-the other issues related to Perkins III in Section I, the larger issues of effectiveness in Section II, and the other topics raised in Section III — are relatively enduring ones, where the questions have been relatively stable all during the 1990s.

IV.1: Examining State Plans

State plans are usually compliance-oriented documents, often containing relatively little useful information. However, they do at least describe the state performance measures and the any local indicators developed by the states, and there will be useful in examining patterns in accountability (see I.1).

IV.2: Surveying or Interviewing State Directors of PSOE

Several issues can be investigated through interviews or questionnaires directed at state directors of postsecondary education, including the nature of performance measures (I.1), the decisions on the allocation of state funds (I.4), the nature of postsecondary institutions in the states (I.5), the strengths and weaknesses of state departments with responsibility for PSOE (useful in III.3), and the normative questions of what federal policy should be (III.4). While forced-choice questionnaires are cheaper, they do not permit exploration of interesting topics. A hybrid approach would therefore be to administer questionnaires in all states, and then to choose a sample of state directors for interview. This sample could include, for example, the states corresponding to local case studies (in IV.4) and community case studies (IV.5); a sample of large states and a sample randomly chosen; or a purposive sample of states known to have strong state policies with another of states with weak policies.

IV.3: Questionnaire Surveys of Local Institutions

Many of the issues covered above require information from local community colleges and other PSOE institutions. Indeed, almost all the topics in Section I and II require information from local institutions, to ask them how they use federal funds (I.2, I.3, I.4) and what practices they now follow related to effectiveness (II.1 through II.4). In addition, the critical question of what a completer is (see III.1) can be partially answered, by asking local institutions what information they collect about student intentions and alternative credentials, and what that information reveals. Because this information will be inconsistent among institutions, however, it will be impossible to aggregate such results to the national level; instead such information will be useful primarily for determining the quality of current data and the range of responses.

Questionnaire surveys are the most efficient ways of getting such information. The questionnaires developed for the prior NAVE can be used as the basis for updated questions, in order to examine trends over time. However, because local responses are likely to be exaggerated, my recommendation is that no questionnaires be administered without case studies to check on their validity, as I propose in the next section.

IV.4: Case Studies of Local Institutions

Local case studies are more reliable because they can interview many different respondents-local PSOE administrators, PSOE instructors, administrators and faculty from the academic side (to check, for example, on the participation in curriculum integration), local interest groups like senates and unions, and students (but see IV.7). They can also examine practices cited; for example, if a college claims to be integrating academic and occupational education in a particular way, or reshaping their curricula for tech-prep students, investigators can visit the classes and instructors to confirm these reforms. The issues to be included in local case studies include all those incorporated into the questionnaires mentioned in IV.3, including virtually all of the issues in Sections I and II. In addition, a case study should include questions about completers and noncompleters, relying on local data (see III.1), and on local perceptions of the "new fluidity" (III.2). Finally, local perceptions of state policy, a necessary component of examining responses to federal legislation (III.3), could be included in these case studies.

One way to carry out case studies is to identify a small set of community colleges and PSOE providers and carry out such studies after responses from questionnaires have been completed, so that the validity of questionnaire responses can be checked. It's difficult to know how many such case studies are necessary, since they need to reflect the variety of urban, suburban, and rural institutions, with some other institutional variants in states that have them; perhaps 20 community colleges and 10 other types of PSOE institutions would be sufficient to generate real results.

In my view, a central component of local case studies should include classroom observations. Only through observation is it possible to determine the extent of integration of different subjects including remedial education (see I.2), for example, or the approaches to teaching of occupational faculty (see II.2). If such observations can be videotaped, then they could begin the creation of a permanent data base for better understanding teaching in community colleges and PSOE specifically. (See Stasz 1999, for a similar argument at the secondary level.)

IV.5: Community Case Studies

Community case studies, examining all the education and training providers in a community, could be carried out for two purposes: to examine the linkages among programs (see II.4), particularly as programs begin planning for WIA; and to describe the "new fluidity" (see III.2). Communities for such case studies should be selected from the communities where local institutional case studies (IV.4) are carried out, so as not to duplicate the information collected from certain PSOE institutions. The communities selected need to vary in certain dimensions-particularly by income or unemployment rates and the composition of local employment-and such communities need to be relatively self-contained, so that it is possible to set limits on the providers who are defined as operating within the community. In prior NCRVE research, two communities in each of four states, or eight communities in all, provided a wealth of information (McDonnell and Grubb 1991; Grubb and McDonnell 1991 and 1995), and such a design might be replicated.

IV.6: Questionnaire Studies of Faculty

The responses of PSOE faculty on questionnaires would be principally useful in identifying the methods of integrating academic and occupational education, and how widespread such practices are (see I.2); any changes in response to tech prep (I.3); the role of employers and other outside groups in setting curricula, including the role of industry-generated credentials and standards (II.2 and III.1); the approaches of instructors to pedagogy, including the question of whether local institutions play any role in helping instructors with their teaching (II.2); the interaction of instructors with support services, particularly developmental education and guidance and counseling (II.3); any influences on instructors of federally required performance measures (I.1) or other locally generated accountability standards (II.4); and faculty perceptions of the issue of completion and the "new fluidity" (III.1 and III.2). In addition, there are at least two other studies of postsecondary faculty — one supported by NCRVE (Brewer and Gray 1997) and one by the Carnnegie Foundation for the Advancement of Teaching (Huber 1998), and these ought to be consulted in designing any new faculty surveys.

IV.7: Questionnaire Studies and Interviews of Students

Questionnaires to students would be useful in ascertaining their perceptions of the value of many practices, including integration (I.2), tech prep (I.3), and student support services like guidance and counseling and remediation (II.3). However, the most critical focus for student services should be the issue of student intentions, raised in the issue of who counts as a "completer" (see III.1); and student activity in attending multiple institutions, as part of examining the "new fluidity" (III.2). Unfortunately, for these purposes questionnaires may not be sufficient, since what counts is not responses to forced — choice questions — which are likely to exaggerate-but rather the stability and precision of these plans. Therefore some student interviews should be used to check on the reliability of student questionnaire responses. Of particular value in such questionnaires would be deriving their school and work histories — that is, the different periods of work and schooling they pursue after high school, trying to determine how related and coherent-rather than fluid and unplanned — such pathways are.²⁴

Locating samples of community college students-especially samples of central interest like noncompleters or leavers-can be quite difficult, particularly since the records community colleges maintain may not be accurate. However, it is particularly necessary to budget in any studies of students for efforts to find fugitive students since otherwise the extent of fluidity will be underreported.

IV.8: Questionnaire Studies of Employers

The principal value of collecting information from employers is to learn more about their demand for PSOE (see II.1), particularly the conditions under which they hire form community colleges, the value they place on credentials including certificates and associate's degrees, licenses, and industry-generated credentials compared to other dimensions of qualifications including experience (see II.1 and III.1 on alternative credentials). An additional purpose is to determine the ways in which local institutions involve employers (see also II.1).

The issues of getting adequate response rates from employers are perhaps as difficult as they are from students, though some researchers have solved this problem (e.g., Osterman 1994). And all the problems with force-choice questionnaires are applicable here too. One alternative would be to include interviews with employers in the community case studies (IV. 5), in order to get their perspectives on the variety of programs within the community.

IV.9: Reanalyses of National Data

As I have stated in reviewing the existing literature (Grubb 1999b), any subsequent analyses of national data ought to focus on several issues that are currently unclear: the value of small amounts of course-taking, where fields of study and student intentions need to be disentangled; the role of local labor markets and employment conditions on the returns to subbaccalaureate education; and the timing of benefits, since it is unclear whether the employment benefits of SOE materialize relatively soon after completion or whether they take several years to develop. NAVE should first determine the appropriateness of national data sets to answering these questions before commission any further research.

IV.10: Reanalyses of State UI Data

NAVE should consider reanalyzing the UI wage record data already collected by several states, in order to generate more sophisticated multivariate analyses. As far as I have been able to ascertain, the states with the most advanced data systems that are candidates for such reanalysis include California, Texas, Florida, and Washington; a brief investigation should be able to determine whether other states have well-developed UI data. One purpose of reanalyses is to learn more about the conditions under which programs are effective (see II.6), and another is to provide some guidance to states in the analysis and presentation of their data.

An outlier study would begin with an analysis of one state's data, identify especially effective colleges, and then carry out local case studies (parallel to those in IV.4) to identify possible reasons for their success. A sample of perhaps three to five colleges in such a state might generate enough information, as well as adding to case study results.

IV.11: Reanalyses of Local Data

Some community colleges have collected relatively sophisticated data on their own students and programs, and can use these data to examine the effectiveness of local practices including the integration of academic and occupational education, developmental education, tech-prep activities, and the like. The local case studies should probe for such local data, and NAVE could then either rely on local analyses or analyze local data further. These data suffer the obvious problem that they are inconsistent from college to college, and therefore cannot be used to develop state or national patterns, but they do illustrate what has been possible for local institutions.

An alternative is to conduct a more active search for such local data by using networks of institutional researchers in community colleges. In general, local data has been underutilized: many colleges collect more data than they ever examine, and even when they are analyzed the results are not widely known either inside or outside of individual colleges. One possibility would be for NAVE to fund the national organization of community college researchers, the National Council for Research and Planning, to identify local results and data sets appropriate for questions about PSOE and Perkins. This would have the added advantage of engaging institutional researchers and their organization in both federal policy and PSOE issues.

IV.12: Other Special Analyses

The analyses outlines above each involve several distinct question. In addition, there are several single-purpose studies that need to be carried out for specific issues raised in Sections I to III. These include the following:

1. The patterns of federal funding to PSOE through grants, loans, and tax credits is one part of the larger question of how federal funds support PSOE (see I.4). This requires data from different sources, some of which describe student aid and other (like IRS data) could help estimate the value of tax credits to PSOE.

2. The results of alternative approaches to the sub-state allocation of Perkins funds would require a simulation model (see I.6).

3. Identifying different types of postsecondary institutions providing PSOE would be helped by inquiring about such institutions from each state, and finding state information about purposes, enrollments, funding, and data on outcomes available.

4. The analysis of industry-generated credentials and standards can be accomplished in part through local case studies (IV.4), surveys of PSOE faculty (IV.6), and surveys or interviews with employers (IV.8). However, these are likely to miss many types of IGCs and IGSs. An alternative is first to identify several sectors that have IGCs and IOGSs, examine the structure of these standards, and then begin to investigate their effects at the local level. For example, the automotive program standards developed by NATEF could be selected; then the relevant questions include which community colleges have adopted NATEF standards and why, which employers require NATEF standards, and whether instructors have modified their teaching in response to these standards where they are widely used. Some of these questions can be investigated through local case studies and questionnaires of faculty and employers-though such responses require a focus on automotive trades, and there are many other sectors and IGCs to investigate. In this alternative, NAVE could identify a small number of industry-generated standards — perhaps 5 to 10 — and investigate their effects through special studies, rather than relying on other forms of data collection. It's also possible that this could be done in conjunction with the National Skill Standards Board. This would help understand the changing conditions of PSOE and help in defining who "completers" are. In addition, a better understanding of the conditions under which employer-driven skill standards emerge and thrive could government groups like the NSSB understand how to create other kinds of skill standards.

5. Each of the support services identified in Section II.3 merits its own analysis. While each of them can to some extent be addressed through questionnaires (IV.3) and local case studies (see IV.4), and while student surveys (IV.7) can generate other information on the use and usefulness of existing services, these approaches won't generate much information if, for example, the institutions selected for case studies don't happen to have interesting approaches to remediation or counseling. An alternative approach would be to supplement information from case studies and questionnaires — which yield information on the scope of conventional practices-with a more concerted effort to identify conceptions and instances of good practice for each of these support services.

6. Finally, the ultimate question posed in Section III.4 — the issue of what the federal role in PSOE could be-would certainly benefit from various studies including the analysis of what the effects of Perkins III have been (the topics in Section I), but it also requires several distinct activities. The development of an options paper based on consideration of fiscal federalism and the history of other federal programs is one, for example; a consensus-generating meeting would be another; international comparisons might also be valuable and they are quite different from any of the research activities presented so far. Thus NAVE should consider supporting several activities specifically to address this difficult normative question.

This page last modified August 11, 1999 (glc).

	Printable view Share this page