Designing Effective Development: Lessons from the Eisenhower Program - December 1999
Chapter 2
In this section, we examine whether the content covered by teachers in our sample emphasizes high standards. We begin our discussion by describing our data on the content taught and the approach we have taken to determine the degree to which the content reflects high standards. Then, we turn to our results.
To assess the consistency of the content taught with national standards, we have collected unusually fine-grained information on the content covered by our sample of teachers, and we have developed a unique strategy of measuring alignment, drawing on the full set of items administered as part of the National Assessment of Educational Progress. In the following sections, we provide a brief overview of these methods. (See Appendix C for more detail.)
Our main data on content come from the baseline wave of the longitudinal survey of teacher change. We characterize the content taught in terms of two major dimensions: the topics covered and the performance goals teachers hold for students.
In the content section of the survey, we asked teachers to describe the content they taught in the class they chose to describe, using a two-dimensional matrix. (Different forms of the matrix were used for elementary, middle, and high school mathematics and science. See Exhibit 2.1 for a sample section from the elementary mathematics form of the survey.) The matrix was initially developed by Porter et al. (1993) in a comprehensive study of mathematics reform and was revised for purposes of the Eisenhower evaluation. Since then, the matrix has been used in several other studies, including Gamoran et al. (1997).8
The rows of the matrix contain a comprehensive list of the topics and subtopics teachers might cover. Algebra, for example, is a topic in mathematics, and multi-step equations is a subtopic under algebra. Astronomy is a topic within science, and the Earth?s moon is a subtopic under astronomy. Each subject area (i.e., mathematics and science) and each school level (i.e., elementary, middle, and high school) has a unique set of topics and subtopics. The matrix for middle school mathematics, for example, has nine topics and 84 subtopics, while the matrix for high school science has 28 topics and 191 subtopics.
The columns of the matrix contain performance goals for students. Performance goals are teachers? expectations for what students should be able to do. There are six performance goals in the matrix: 1) memorize; 2) understand concepts; 3) perform procedures; 4) generate hypotheses; 5) collect, analyze, and interpret data; and 6) make connections. (See Exhibit 2.2 for definitions of the performance goals.) For example, when a teacher emphasizes memorizing, the teacher may expect students to be able to produce definitions or terms, facts, and formulas from memory. When a teacher emphasizes using information to make connections, the teacher may expect students to be able to use and integrate concepts, apply ideas to real-world situations, build or revise theory, and make generalizations.
EXHIBIT 2.1
Excerpt from Content Coverage Section of the Elementary School
Mathematics Teachers Survey
| Elementary School Topics | Coverage | Your Performance Goals for Students | |||||
|---|---|---|---|---|---|---|---|
| Whole Numbers | (none) | Memorize | Understand Concepts | Perform Procedures | Generate Hypotheses | Collect Analyze/Interpret | Make Connections |
| Addition | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Subtraction | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Multiplication | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Division | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Combinations of add, subtract, multiply and divide | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Equations (including missing addend, factor, etc.) | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Fraction | (none) | Memorize | Understand Concepts | Perform Procedures | Generate Hypotheses | Collect Analyze/Interpret | Make Connections |
| Identify equivalent fractions | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Add | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Subtract | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Multiply | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Divide | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
| Combinations of add, subtract, multiply, and divide | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 | 0 1 2 3 |
|
Source: Longitudinal Teacher Survey Elementary School Mathematics, Fall 1997(1996-97 school year). |
|
The following are various performance goals that teachers hold for their students. Refer to this list in describing your performance goals for each topic covered as part of instruction.
|
| Source: Longitudinal Teacher Survey, Instruction Booklet for section II. How to read this exhibit This exhibit illustrates the instructions given survey respondents for completing the matrix displayed in Exhibit 2.1. |
A content area can be defined as the intersection of the two dimensions, topics and performance goals. For example, if teachers emphasize memorizing facts about the Earth?s moon, the content area incorporates the subtopic (the Earth?s moon) and the performance goal (memorizing). Both elements—topics and performance goals—are integral to understanding the content of a lesson or course. For example, the student learning that would be likely to take place if the content were memorizing facts about the Earth?s moon (e.g., gravity, distance from the Earth) is very different from the student learning that would occur if the content were understanding the Earth?s moon (e.g., forces working to keep satellites in orbit).
Each teacher was asked to follow several steps in describing the teacher?s year-long course using the matrix. First, the teacher indicated the amount of time given to each subtopic, using a scale from 0=no time through 3=more than two lessons or class periods. Then, the teacher indicated the relative amount of emphasis given to each performance goal when teaching the subtopic, using a scale from 0=no emphasis to 3=sustained emphasis. We used the full matrix of data provided by each teacher to calculate the percentage of the teacher?s total year-long class time devoted to each topic and subtopic, each performance goal, and each content area (intersection of a subtopic and performance goal).
We compare teachers? instruction to the NAEP items in order to assess how well instruction meets high standards. To determine the relative amount of emphasis given by the NAEP to each subtopic, performance goal, and content area in our elementary, middle, and high school science and mathematics matrices, we reviewed the full set of NAEP mathematics and science items for the 1996 tests for fourth, eighth, and twelfth grade.9 We asked two curriculum experts in mathematics and two experts in science to review each NAEP item and to determine the specific subtopics and performance goals each item was designed to tap.10 Using this information for the full set of NAEP items, we computed the relative emphasis given by NAEP to each subtopic, performance goal, and content area.
In the paragraphs that follow, we draw on our data provided by teachers, as well as our information from the NAEP, to examine four aspects of the content taught. First, we examine the extent to which the topics covered by the teachers in our sample match the topics assessed by the NAEP. Then, we consider the extent to which the performance goals our teachers emphasize match the NAEP. Third, we examine the content areas (intersection of topics and performance goals). Finally, we develop an overall index of the alignment between the content covered by teachers and the content assessed by the NAEP.
In this section, we examine the emphasis given by teachers in our sample to specific topics, and we compare this with the relative emphasis given to the same topics in the NAEP. Research indicates that some topics, for example geometry and measurement in mathematics, are special weaknesses for students in the United States (Beaton et al., 1996). In the Third International Mathematics and Science Study, U.S. students in seventh grade scored 19th out of 27 countries in geometry and 23rd in measurement. U.S. students in eighth grade scored 21st out of 25 countries in geometry and 23rd in measurement (Beaton et al., 1996). Thus, in interpreting our results, we give special attention to these two topics.
Exhibit 2.3 presents our data on the emphasis given to particular topic areas, for the middle school mathematics teachers in our sample. (Similar results for the other teachers in our sample are included in Appendix D.) The results indicate that topics that traditionally have been weaknesses for U.S. students, especially for mathematics, do not receive much attention from teachers in our sample. Despite evidence that middle school students need to focus on measurement and geometry, middle school teachers surveyed for this study taught measurement on average for 12 percent of their course, compared to the NAEP?s emphasis of 20 percent. The average middle school teacher taught geometry for nine percent of their course, compared to the NAEP?s emphasis of 15 percent. On the other hand, middle school teachers tend to give more emphasis to low-level topics than does the NAEP. For example, middle school teachers emphasized number sense 37 percent of the time compared to 14 percent for the eighth-grade NAEP.11
EXHIBIT 2.3
Percentage of Emphasis on Topics in Middle School Math, Reported by NAEP and by
Teachers in the Longitudinal Teacher Survey
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). |
Questions about the topic focus for instruction can be extended from whether teachers cover critical topics to whether they cover any topics. Previous research has found that teachers sometimes focus so much on changing the process of instruction that they neglect the topics of the lesson. Roitman (1998), for example, described a case in which an observed teacher was so focused on active learning activities that her lesson was topic-free. To consider this possibility, using the Eisenhower data, we turn to the classroom observations. Several of the observed lessons did focus on process to the point of having little or no content emphasis. For example, in a sixth-grade science lesson in Boonetown, the class focused on using the scientific method. Students conducted an experiment to test the absorption of different brands of paper towels, as part of a consumer unit. In preparation for the experiment, student groups had written hypotheses and experimental designs the day before. The teacher introduced the class, reviewed relevant vocabulary, and directed the students to work in groups. The groups designed and conducted their own experiments. At the end of the lesson, the teacher asked students to consider ways in which the experiment could be improved. However, students did not discuss or present their findings. While the lesson could have developed understanding of material composition or properties, the focus throughout was on process rather than content.
The United States differs from other countries in how the content is organized across grades. In the U.S., topics are repeated in many grades, theoretically with increasingly complex subtopics. Other countries, such as France and Japan, focus on selected topics at each grade level (Matheson et al., 1996). For example, the eighth-grade mathematics teachers in Japan focus on four topics, with relatively little emphasis (less than four percent of instructional time) on other topics. In contrast, U.S. teachers spread instruction over 21 topics (Wilson and Blank, 1999).
According to national standards for science and mathematics, developed by the National Research Council (NRC) and the National Council of Teachers of Mathematics (NCTM), it is possible to cover the same broad topics across grades, while enhancing the depth of exposure. Revisiting topics in successive grades should build on understanding, allowing instruction to focus on more complex subtopics within broad topics.
The NRC science content standards cover the same broad topics (i.e., physical, life, earth and space science, science and technology, science in personal and social perspectives, and history and nature of science) at all grade levels. These standards stress developing a more sophisticated understanding of more advanced subtopics within each topic as students move up grade levels, to reflect developmental and learning abilities of students. With this approach, the subtopics become more abstract and students are expected to develop greater conceptual understanding as they progress through the grades. Thus, NRC?s science standards suggest that, at a gross level, there should be substantial overlap in topics across grade levels; however, students at each school level should be learning different subtopics within these broad topics.
Similar to the science standards, the NCTM standards emphasize five major topics (i.e., number and operations; patterns, functions, and algebra; geometry and spatial sense; measurement; and data analysis, statistics, and probability) in every grade. Again, like the science standards, the mathematics standards stress that specific subtopics within broad topics become more sophisticated as the student progresses through the grades.
Consistent with the international research, our data show that, on average, teachers across grades generally teach the same topics. There is little clear pattern of intensified coverage in broad topic areas as grade levels increase. For example, teachers at all grade levels teach measurement in mathematics, some grades more than others, but there is no pattern of consistent increase or decrease in focus on measurement as grade levels increase (see Exhibit 2.4). First-grade mathematics teachers emphasized measurement for 12 percent of the time, second-grade teachers for 23 percent, sixth-grade for 9 percent, and twelfth-grade for 13 percent.
Furthermore, the measurement subtopics emphasized do not consistently increase in complexity as the grade level increases (see Exhibit 2.5). Of the 16 measurement subtopics, only four show the expected pattern. Two of the more low-level subtopics (i.e., use of instruments and time and temperature) showed, on average, decreased emphasis as grade level increased (from 34 percent to nine percent and from 23 percent to one percent, respectively). Two of the more complex topics (i.e., Pythagorean theory and trigonometry) showed, on average, increased emphasis as the grade level increased (from zero to 6 percent and from zero to 48 percent, respectively). For the most part, however, there is little evidence of increasingly complex topics in successive grades.
EXHIBIT 2.4
Percentage of Emphasis Mathematics Teachers in the Longitudinal Teacher Survey
Give to Measurement, by Grade (n=181)
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). |
|
K |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
Trend |
|
|
Use of instruments |
33.6 |
20.0 |
8.0 |
20.4 |
23.1 |
23.2 |
12.8 |
9.7 |
6.6 |
7.6 |
3.6 |
3.6 |
9.4 |
Decrease |
|
Conversions |
1.5 |
0.5 |
0.6 |
2.2 |
5.9 |
5.6 |
8.5 |
7.9 |
9.0 |
4.1 |
4.9 |
2.8 |
2.9 |
None |
|
Metric system |
0 |
11.7 |
1.9 |
7.3 |
5.5 |
9.4 |
12.1 |
11.6 |
5.9 |
4.0 |
2.9 |
2.0 |
1.2 |
None |
|
Length, perimeter |
9.6 |
7.6 |
4.8 |
9.5 |
10.1 |
12.9 |
15.7 |
17.4 |
12.2 |
16.2 |
7.6 |
9.1 |
6.4 |
None |
|
Area, volume |
4.3 |
3.0 |
1.8 |
9.3 |
11.9 |
11.3 |
14.7 |
11.1 |
12.6 |
14.7 |
10.0 |
8.6 |
4.3 |
None |
|
Telling time |
18.1 |
31.3 |
30.9 |
18.9 |
15.9 |
7.3 |
na |
na |
na |
na |
na |
na |
na |
None |
|
Non-decimal money |
2.7 |
12.8 |
35.3 |
11.9 |
6.2 |
8.7 |
na |
na |
na |
na |
na |
na |
na |
None |
|
Circles |
1.7 |
1.4 |
1.2 |
0.6 |
3.9 |
7.2 |
8.6 |
9.7 |
13.6 |
10.6 |
6.8 |
5.8 |
7.0 |
None |
|
Mass |
5.5 |
4.0 |
2.1 |
4.2 |
6.7 |
5.8 |
4.4 |
3.9 |
3.4 |
2.6 |
2.7 |
1.2 |
0.7 |
None |
|
Time, temperature |
22.6 |
7.7 |
13.3 |
15.6 |
10.1 |
8.8 |
9.0 |
3.7 |
4.8 |
2.9 |
2.7 |
3.8 |
1.1 |
Decrease |
|
Theory |
na |
na |
na |
na |
na |
na |
2.3 |
2.6 |
1.6 |
2.2 |
1.9 |
1.2 |
0.6 |
None |
|
Surface area |
na |
na |
na |
na |
na |
na |
3.3 |
9.0 |
8.2 |
4.7 |
7.3 |
2.9 |
5.1 |
None |
|
Angles |
na |
na |
na |
na |
na |
na |
7.1 |
8.3 |
6.9 |
9.5 |
5.6 |
8.2 |
5.1 |
None |
|
Pythagorean theory |
na |
na |
na |
na |
na |
na |
0 |
2.7 |
10.0 |
9.7 |
7.2 |
6.8 |
6.4 |
Increase |
|
Simple trig, right triangles |
na |
na |
na |
na |
na |
na |
0 |
0.3 |
1.6 |
7.5 |
31.0 |
38.3 |
47.8 |
Increase |
|
Speed |
na |
na |
na |
na |
na |
na |
1.4 |
2.1 |
3.6 |
3.7 |
5.9 |
5.7 |
2.2 |
None |
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). |
In this section, we compare the performance goals emphasized by teachers in our sample with the performance goals emphasized in the NAEP. Prompted in part by the report A Nation at Risk (NCEE, 1983), there has been a movement toward teaching for understanding rather than memorizing (Roitman, 1998). National mathematics and science standards emphasize teaching for understanding. Teaching for understanding and teaching for memorizing are performance goals, or expectations teachers have for what students should be able to do. Ideally, teachers would have performance goals for students that are consistent with the performance goals advocated in the national standards. Of the six performance goals teachers could identify on the Longitudinal Teacher Survey (i.e., memorize, understand concepts, perform procedures, generate hypotheses, collect/analyze/interpret data, make connections), four are especially relevant for the abstract thinking involved in developing complex understanding: understanding concepts, generating hypotheses, collecting/analyzing/interpreting data, and making connections. Of the six performance goals, two are especially relevant for developing concrete skills and knowledge: memorizing and performing procedures. Ideally, teachers will balance their emphasis on the six performance goals. In this section, we examine whether teachers have balanced performance goals for their students, comparing teachers? performance goals to the emphases on the NAEP.
Exhibit 2.6 presents our results for elementary school mathematics. (Results for other teachers in our sample appear in Appendix D.) The results indicate that NAEP items tend to focus on the performance goals that involve less abstract thinking, such as memorizing, which does not exemplify the ideal pattern advocated by the national standards. Compared to the performance goal emphases in the NAEP, teachers in our sample give more equal emphases to all six goals. For example, elementary mathematics teachers report devoting 20 percent of their class time to performing procedures, compared to 44 percent for the NAEP, and they devote 17 percent of their class time to making connections, compared to 5 percent for the NAEP.
EXHIBIT 2.6
Comparison of NAEP and Teachers in the Longitudinal Teacher Survey on Emphasis
on Performance Goals (n=74)
| Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year) How to read this exhibit: The first bar shows that the NAEP has a relative emphasis of 16 percent on the performance goal Memorize. Each performance goal is included in the bar chart. The relative emphasis can be between 0 and 100 percent. |
To examine differences across types of teachers in the emphasis given to the six performance goals, we computed the mean percent emphasis for each goal by school level (elementary, middle, and high school), subject (mathematics and science), and school poverty (high and low). Results indicate that mathematics teachers emphasize the more concrete performance goal, performing procedures, significantly more than science teachers do, while science teachers emphasize the more abstract performance goals collecting, analyzing, and interpreting data, making connections, and generating hypotheses significantly more than mathematics teachers do. Teachers in high-poverty schools place significantly greater emphasis on memorizing and significantly less emphasis on understanding concepts, compared to teachers in low-poverty schools. High school teachers place significantly greater emphasis on performing procedures than either elementary or middle school teachers, and significantly less emphasis on generating hypotheses than middle school teachers; this reinforces the finding noted earlier, that instruction does not seem to be more complex or abstract at higher grade levels (see Exhibit 2.7).
EXHIBIT 2.7
Mean Percent Emphasis Given to Each Performance Goal (Standard Deviation),
by School Level, Subject, and Poverty Level (n=355)
|
Memorize |
Understand Concepts |
Perform Procedures |
Generate Hypothesis |
Collect/ Analyze/ Interpret |
Make Connections |
|
| School Level | ||||||
|
Elementary |
15.74 (0.07) |
23.94 (0.12) |
18.19 (0.06) |
11.72 (0.06) |
13.07 (0.05) |
17.22 (0.05) |
|
Middle |
15.55 (0.08) |
21.84 (0.05) |
18.32 (0.05) |
12.63 (0.05) |
13.98 (0.06) |
17.65 (0.05) |
|
High |
15.59 (0.06) |
23.47 (0.07) |
20.37 (0.07) |
10.81 (0.05) |
12.97 (0.06) |
16.78 (0.06) |
|
Subject |
||||||
|
Mathematics |
15.76 (0.06) |
23.15 (0.07) |
22.18 (0.06) |
10.66 (0.06) |
11.70 (0.06) |
16.46 (0.05) |
|
Science |
15.51 (0.08) |
23.45 (0.11) |
15.75 (0.04) |
12.54 (0.05) |
14.84 (0.05) |
17.87 (0.05) |
|
Poverty Level |
||||||
|
High Poverty |
16.60 (0.07) |
21.75 (0.07) |
18.06 (0.05) |
12.46 (0.05) |
13.86 (0.05) |
17.17 (0.05) |
|
Low Poverty |
15.09 (0.06) |
24.18 (0.10) |
19.59 (0.07) |
11.09 (0.06) |
12.88 (0.06) |
17.14 (0.06) |
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). |
The emphases teachers in this study give to the six performance goals found in this study are inconsistent with previous studies of teachers? instructional emphases. In the early 1980s, studies of 41 elementary school teachers found that 70 to 75 percent of mathematics instruction focused on teaching students skills, such as addition, with little attention to developing conceptual understanding or problem solving (Porter, 1989). In a ground breaking study of mathematics and science instruction in high school, Porter et al. (1993) found that in 1990 and 1991, teachers reported focusing most on solving routine problems (e.g., computation), in both mathematics and science. The performance goal of building and revising theory and developing proofs was emphasized very little. Observations confirmed that teachers did, in general, focus on the more rote performance goals such as memorizing facts, definitions, and equations, performing procedures, and solving routine problems. The TIMSS videotape classroom study also found an emphasis on routine procedures in U.S. mathematics classes (Stigler et al., undated). Teachers in the current study, however, report no greater emphasis on rote skills, such as memorization, than on conceptual development skills, such as making connections.
It could be that the low emphasis on rote skills reported here reflects teachers? beliefs about their instruction (Cohen, 1990). For example, Knapp et al. found that many teachers who were trying to use new instructional practices to improve student understanding "got the words but not the tune," or used new learning activities without understanding or capitalizing on their potential (Knapp et al., 1993: p. 23); and in a study of 25 mathematics teachers professing familiarity with and use of standards-based instructional strategies, only four truly demonstrated the intent of the standards (Spillane & Zeuli, 1999).
The Eisenhower evaluation observations found examples of similar differences between teachers? perceptions of their instruction and observers? analyses of the same classes. An elementary school in Richmond provides an example of the discrepancy between a teacher?s description of her instruction and a trained observer?s description of the same class. For the first half of this lesson, the teacher led a whole-class review of operations (e.g., addition, subtraction, exponents, square roots). She wrote pieces of equations on the chalkboard and asked students to fill in the missing information. While the lesson allowed students to "create" equations, the thinking tasks were defined within very narrow parameters, and the focus of the lesson was on arriving at the "correct answer." The teacher felt that she was emphasizing complex performance goals such as interpreting data and making connections for more than half of this part of the lesson (66 percent). The two observers, however, saw an emphasis on lower level performance goals, such as performing procedures and memorizing (60 percent).
A middle school in Rhinestone provides an example of a lesson that emphasizes more complex performance goals. In this middle school science lesson, students constructed a bridge out of spaghetti and marshmallows. The bridge had to meet certain specifications of length and width and have certain characteristics (e.g., it had to have two piers). According to the teacher, this project was intended to make more concrete the construction problems students had been discussing in previous classes. The class was part of a larger unit on bridges and other structures, and the unit was part of an inquiry-based curriculum designed by the observed teacher and a peer.
After the class, the teacher and the observer separately identified the performance goals and topics emphasized in the class. Both the teacher and the observer felt the lesson stressed performance goals that had to do with developing complex understanding. The teacher and observer gave favorable descriptions of the class. Both agreed on the percentage of emphasis the lesson gave to memorizing. But the teacher felt that she placed more emphasis on understanding content than the observer thought she did.
Clearly, in some cases, teachers are emphasizing performance goals of increased student understanding. However, in some cases, changes in instruction may not be at the level that would improve student understanding.
These observations raise the question of the prevalence of teacher exaggeration in their reported instruction. Do teachers, in general, tend to over-report performance goals that they see as positive? The second example shows that teachers do not consistently exaggerate. Yet the first example shows that some teachers may report a more favorable picture of their instruction than is observed. Although we cannot conclusively determine the accuracy of teachers? reporting from these data, the self-reporting bias, if any, should not have much influence on the analyses of change in teaching practice to be discussed in our third report, because we would expect any bias to be constant across the three waves of the survey.
As previous research suggests, there may be a gap between the teacher?s perception of her instruction and a more objective evaluation of the same lesson. Teachers may believe they are teaching in ways consistent with high standards; without feedback on their instruction, they may not recognize areas for improvement. Well-constructed professional development, which provides opportunities for such feedback, may help teachers continue to evaluate and improve their instruction.
In this section, we turn to the emphasis teachers give to specific content areas—that is, to the intersection of topics and performance goals. As noted above, the idea of "content" includes both the topic of instruction and the teacher?s goals for student performance.
The mathematics and science standards present a vision for instruction, in which each grade builds on the learning in the previous grade. However, research on teaching practice and findings reported earlier from the current study suggest that subtopics and performance goals do not become more challenging as students move through the grades. Rather, students visit and revisit the same topics and subtopics at a superficial level. This curriculum organization contributes to a phenomenon in the U.S. recognized as "teaching for exposure" (Porter, 1989). Because many topics and subtopics are taught at more than one grade level, teachers provide very limited instruction in a large number of topics and subtopics. This practice is unlikely to deepen students? understanding of any particular topic (Rollefson, 1996). Effective instruction must balance covering a variety of content areas (breadth) with developing deep understanding in each content area (depth), perhaps even emphasizing depth by limiting breadth somewhat (Raizen, 1998).
To assess the depth and breadth of content covered, we counted the total number of possible content areas (cells) in our matrix. We then determined the percentage of cells that teachers reported covering, and compared this percentage with content areas (cells) assessed by the NAEP. Our results, shown in Exhibit 2.8, indicate that teachers? instruction shows greater breadth than is reported on the NAEP. While the NAEP does not assess and the surveyed teachers do not cover all possible content areas, teachers consistently cover substantially more content areas than represented in the NAEP, in some cases as much as twice as many areas.13 For example, teachers cover 39 percent of the 144 content areas in middle school science, while the NAEP assesses only 16 percent.14 Teachers in high-poverty schools tend to cover more content areas than teachers in low-poverty schools.
EXHIBIT 2.8
Comparison of NAEP and Teachers in the Longitudinal Teacher Survey on Coverage of Content Areas (n=355)
| Mathematics | Science | |||||
|---|---|---|---|---|---|---|
|
Elementary School |
Middle School |
High School |
Elementary School |
Middle School |
High School |
|
|
Number of possible content areas |
60 |
54 |
96 |
96 |
144 |
168 |
| Percent assessed by NAEP |
35% |
34% |
20% |
26% |
16% |
15% |
|
Percent covered by teachers |
47% |
51% |
43% |
36% |
39% |
32% |
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). |
Covering a large number of content areas is not necessarily an instructional liability; however, if by covering more content areas teachers are unable to focus on each content area as thoroughly as needed, students may not have opportunities to develop deep understandings in each area. Our data suggest this is the case. In comparison to the NAEP items, teachers cover more content areas, giving many relatively little emphasis. For example, Exhibit 2.9a shows that in elementary school science, the NAEP gives strong emphasis (greater than 3 percent emphasis) to 11 content areas; Exhibit 2.9b shows that teachers report strong emphasis on only three content areas. On the other hand, the NAEP gives weak emphasis (1 to 2 percent emphasis) to only 15 content areas, shown in Exhibit 2.9a, while teachers report a weak emphasis on 27 areas. This pattern is found across subjects (i.e., mathematics, science) and school levels (i.e., elementary, middle, high school). As previous research suggests, and the current analysis reiterates, teachers tend to favor breadth over depth in their instruction.
EXHIBIT 2.9a
Emphasis on Content Areas in Fourth-Grade Science NAEP Items
![[data not available]](/inits/teachers/eisenhower/exhibits/exhibit2.9a.gif)
|
Source: NAEP, 1996. |
![[Data not available]](/inits/teachers/eisenhower/exhibits/exhibit2.9b.gif)
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). Kindergarten through fifth grade teachers. |
Finally, in this section, we report an overall measure of the alignment between the content areas taught by teachers in our sample and the content areas emphasized by the NAEP. We computed the measure based on the relative emphasis given to each content area in our matrix (each cell) by the teachers and by the NAEP.15 For each teacher, the index takes on a value ranging from zero (no agreement at all between the content areas the teacher emphasizes and those emphasized by the NAEP) to 100 percent alignment (complete agreement between the content areas emphasized by the teacher and the NAEP). High alignment indicates that teachers emphasize topics and performance goals that were similar to NAEP?s emphasis. For example, teachers might focus especially on understanding concepts (a performance goal) for motion and forces (a topic) by asking students to explain, in everyday terms, the relationship between motion and force. If the NAEP also emphasizes understanding concepts for motion and forces, there would be high agreement between instruction and the NAEP on that content area. If there were a pattern of such agreement across content areas, then the index of alignment would be high.
Depending on the subject (i.e., mathematics or science) and the school level (i.e., elementary, middle, or high school), the average alignment between the surveyed teachers? instruction and the NAEP ranges from 11 to 29 percent (see Exhibit 2.10). Considering the large number of content areas, alignment of 29 percent is quite high. There is no significant difference in alignment between high- and low-poverty schools.
EXHIBIT 2.10
Degree of Alignment between Teachers? Instructional Emphases
and NAEP Emphases (n=355)
![[Data not available]](/inits/teachers/eisenhower/exhibits/exhibit2.10.gif)
|
Source: Longitudinal Teacher Survey, Fall 1997 (1996-97 school year). |
As can be seen in Exhibit 2.10, the content of instruction for teachers in our sample is better aligned with the NAEP in mathematics than in science. There are a number of possible reasons for this. Mathematics often is seen as a core subject; everyone generally agrees that children should learn fractions or geometry. Science, however, is not always so central to the curriculum, especially in the early grades (Kennedy, 1998; Raizen, 1998). At all levels, students have less exposure to science than to mathematics: 30 minutes per day for science compared to 60 minutes per day for mathematics in elementary schools, and two to three years of mathematics compared to two years of science required in high school (Weiss, 1997). Further, there is greater national consensus on core topics in mathematics than in science: state mathematics curricula tend to focus on common topics, while there is little overlap between states in terms of science emphasized by the state curricula (Schmidt et al., 1996).
Finally, teachers may be less familiar with science, and this could affect the quality of instruction. Elementary school teachers generally have a basic understanding of reading and mathematics, and feel comfortable teaching these subjects; however, they may be less knowledgeable about and comfortable teaching science. Research suggests that teachers? content knowledge affects their instruction (Rollefson, 1996). A survey of elementary school mathematics and science teachers found that 60 percent felt qualified to teach mathematics, 28 percent felt qualified to teach life sciences, and fewer than 10 percent felt qualified to teach physical science (Weiss, 1997). Teachers teaching out-of-field may misrepresent key concepts or focus on trivial rather than central concepts and tend to rely on drill-and-practice activities rather than instruction oriented toward student inquiry. Further, because the science standards are relatively new, teachers and students have had limited time to become familiar with them.
Our data also indicate that content was more highly aligned with the NAEP in the elementary and middle schools than in the high schools. This phenomenon might be an artifact of the test. Although the NAEP is used as a standard for high expectations, the high school test is geared toward content covered prior to high school. Thus, the standard set by the high school NAEP test might not be as challenging as the instruction of teachers who participated in the Eisenhower evaluation. Although most of the high school teachers in the sample described average-level courses, such as algebra and biology, some did describe advanced courses such as calculus and physics, and honors courses (see Appendix D).
Our data indicate that the degree of alignment of the content taught with high standards seems to be related to the school in which instruction occurs. As much as 30 percent of the difference among surveyed teachers in terms of how well their instruction meets high standards can be attributed to the school in which the teacher teaches (see Appendix D). Our data indicate that the effect of the school on alignment with the NAEP is greater for science than for mathematics instruction, and greater for elementary and middle schools than high schools. These findings imply that, at least for this sample, strategies to help teachers improve instruction should be targeted to schools.
Finally, although we talk about average alignment across groups of teachers, teachers differ from each other in how closely their instruction aligns with the NAEP. This variation is quite visible in Exhibit 2.10, which shows that some teachers are nearly 50 percent aligned with the NAEP, while others have almost no alignment. For example, the instruction of one elementary school science teacher (called Teacher A) was minimally aligned with the NAEP: on an index of zero to 100, with 100 indicating perfect alignment, she had alignment of less than one percent. On the other hand, the instruction of another elementary school science teacher (called Teacher B) was relatively highly aligned with the NAEP: in the alignment index, she had alignment of 35 percent. The performance goals and topics that Teacher B emphasized resembled the NAEP much more closely than the performance goals and topics of Teacher A. Teacher B stressed memorizing and understanding, as the NAEP does, while Teacher A stressed performing procedures and collecting/analyzing/ interpreting data, unlike the NAEP. Teacher B emphasized components of living systems, ecology, properties of matter, and astronomy similarly to the NAEP, while Teacher A emphasized maintenance in animals, unlike the NAEP. A teacher such as Teacher B, whose instruction already meets high standards, may not need the same type of guidance as a teacher such as Teacher A, whose instruction is weak compared to the standards.
Data from the baseline Longitudinal Teacher Survey indicate that the content taught by teachers in our sample is moving toward but does not yet meet high national standards in several ways. Teachers do not focus on some advanced topics; rather, they emphasize low-level topics. Although teachers set more complex performance goals for their students than they have historically, the changes in their instruction are not always as deep as the teachers perceive them to be. Teachers appear to teach for exposure, and the content covered does not appear to become substantially more challenging in successive grades. These findings are generally consistent with previous research, which suggests that instruction in American schools does not emphasize challenging content.
Targeted professional development can help address these concerns. For example, professional development that focuses on content can help teachers develop a deeper understanding of the content they teach and develop lessons that are rich in challenging content (Kennedy, 1998). Professional development with in-classroom follow-up components could help teachers understand the level at which their instruction has changed and the areas in which the intended change is still superficial. (See, for example, Schifter, 1996). School-based professional development that includes the collective participation of groups of teachers from the same grade could help teachers organize instructional emphases across grades, so that each successive grade builds on the previous one. Later chapters examine the prevalence of these types of designs and characteristics in Eisenhower-assisted professional development activities and their relationship to teacher outcomes.
9 The NAEP fourth-grade test was compared to instruction for teachers in elementary school (grades K-5). The NAEP eighth-grade test was compared to instruction for teachers in middle school (grades 6-8). The NAEP twelfth-grade test was compared to instruction for teachers in high schools (grades 9-12).
10 Appendix D provides information on the reliability of the expert ratings of the NAEP items.
11 Results shown in Appendix D indicate that the emphasis on geometry is lower in high-poverty schools than in low-poverty schools.
13 Some performance goals that are especially difficult to measure on a timed, paper-and-pencil test, such as generating hypotheses, may be underrepresented on the NAEP.
14 This difference might be due, in part, to the fact that the teachers reported on all content areas covered over the course, while the NAEP only tests on a sample of content areas that students are expected to learn. However, the analysis included the full set of NAEP items, which is a sizable sample of items, so NAEP content coverage should be substantial.
15 The index of alignment is computed as the sum, across content areas, of the absolute value of the difference between the teacher's and the NAEP emphasis in each content area, divided by two, subtracted from one; the result is multiplied by 100. The absolute value is required because the index is designed to capture cells for which the teachers give more emphasis than the NAEP, as well as those for which they give less emphasis.
 [Effective Content and Pedagogy] |
![[Table of Contents]](/icons/up.gif){kind=icon} |
 [Pedagogy and High Standards] |