In comprehending a sentence, spoken or written sensory stimuli must be mapped onto meanings. This is a dynamic process involving determining not only the meanings of individual words but also the meanings of combinations of words that are usually embedded within a syntactic framework. Many studies investigating the neural organization of semantic processing have focused on the processing of single words and have implicated a distributed network of brain areas in this function (for reviews see Martin & Chao, 2001; Thompson-Schill, 2003; Damasio, Tranel, Grabowski, Adolphs, & Damasio, 2004). Less is known about how the semantic system deals with groups of words. For example, the nouns shipwreck and basketball each have their own individual meanings, but when they are combined, as in the sentence The shipwreck victim survived by clinging to a basketball, a new and much more complex semantic representation is created, in which the whole is greater than the simple sum of the parts and includes the now salient concept that basketballs float.¹ We refer to this process of combining the meanings of multiple words as “combinatorial semantic processing”.

Prior research on the neural correlates of combinatorial semantic processing has mainly focused on how single words are integrated within a sentence. An event related potential (ERP) component called the N400 varies with several lexical-semantic factors influencing sentence processing (Kutas & Hillyard, 1980; Kutas & Federmeier, 2000). For example, the N400 is larger when a word in the sentence is semantically unexpected but grammatically correct, as in the final word of the sentence, The man ate a window, compared to when a word is congruent with the semantic context (e.g., The man ate a sandwich). Similar semantically anomalous sentences have been used in fMRI experiments, which have shown greater activation for sentences with semantic anomalies in a variety of regions in the left inferior frontal, left inferior parietal, and left posterior temporal lobes (Friederici et al., 2003; Newman, Just, Keller, Roth, & Carpenter, 2003; Luke, Liu, Wai, Wan, & Tan, 2002; Kuperberg et al., 2000; Ni et al., 2000; Kang, Constable, Gore, & Avrutin, 1999). The prevailing interpretation of such results is that the anomalous word is more difficult to integrate into the established semantic context, thus the increase in activation is attributed to increased neural activity devoted to combinatorial semantic processing (Kutas & Federmeier, 2000).

A different approach to studying combinatorial semantic processing is to compare processing of normal sentences with processing of grammatically correct sentences in which the content words (i.e., nouns, verbs, adjectives) have been selected at random and are therefore thematically unrelated (Humphries, Binder, Medler, & Liebenthal, 2006; Mazoyer et al., 1993; Vandenberghe, Nobre, & Price, 2002). Because the words are no longer thematically related, the subject is unable to build a coherent semantic representation larger than that given by each individual word. Results of this experiment have varied, with one study finding greater activation in the anterior temporal pole for the semantically random stimuli over the normal stimuli (Vandenberghe et al., 2002), and other studies showing greater activation for the semantically normal over the semantically random stimuli in left middle temporal areas (Humphries et al., 2006; Mazoyer et al., 1993) and left inferior parietal lobe (Humphries et al., 2006). Some of these differences could be due to differences in the tasks that were performed by the participants, which made different demands on overt semantic interpretation (Humphries et al., 2006).

One difficulty in studying combinatorial semantic processing with functional imaging is that there are likely to be several component processes involved, each occurring on different time scales. For example, during comprehension of a sentence, access to the meanings of individual words starts as soon as the sentence begins. Building and representing an overall meaning, on the other hand, probably starts later in the sentence and may even continue after the stimulus has ended. In event-related fMRI studies, these different processes might be reflected by differences in the BOLD time course. The most common approach used in analyzing such studies is to average the activation across the entire stimulus trial. However, if the effect of interest changes over time (i.e., there is an interaction between time from stimulus onset and type of process activated), then averaging the response over time will not provide a complete and accurate picture of these component processes.

We examined these issues by acquiring fMRI data while subjects were presented with sentences and word lists with varying degrees of combinatorial and word-level semantic information. These materials included semantically congruent sentences and word lists, in which all of the content words (nouns, verbs, adjectives) were thematically related to a concrete event; semantically random sentences and word lists, in which the content words were thematically unrelated; and pseudoword sentences and word lists, in which the content words were replaced by meaningless pseudowords. Results from this study were published previously using contrasts that summed over a range of time points, showing large regions of temporal and parietal cortex to be involved in combinatorial semantic processing (Humphries et al., 2006). In the current study, we reanalyzed the data using a more sensitive, time-based analysis to better examine the time course of activation in these semantic processing areas. We hypothesized that combinatorial and word-level semantic processing would be active at different times during comprehension of the stimulus, with word-level processing occurring relatively early and combinatorial occurring relatively late. Analyzing individual time points should increase sensitivity to these semantic effects by avoiding averaging over a large number of possibly inactive time points. In addition, this approach will allow us to characterize differences in onset times of combinatorial and word-level related activity based on contrasts between conditions. Knowledge of the timing of these processes will allow us to better define the function of identified semantic regions, especially those regions that show activation both during word-level and during combinatorial processing.

There were 40 stimuli per condition. To remove any phrase-level prosodic contours that might distinguish the sentences from the word lists, the words and pseudoword used in the experiment were recorded individually in a random order, and the source talker (JRB) used the same neutral tone with slightly falling intonation contour for each item. To create the sentences and word lists, the individual word recordings were normalized by total energy and concatenated using Matlab (Mathworks Inc, Nattick, MA). Spacing was added between each word in the sentence or word list so that the total length of each stimulus was 6.1 seconds.

FMRI data were collected on a 1.5 Tesla GE scanner. For each subject a high-resolution T1-weighted anatomical scan was collected in the sagittal plane using an SPGR pulse sequence (FOV = 240 mm, matrix = 256 × 256, size = 0.9375 × 0.9375 mm, thickness = 1.2 mm, flip angle = 40 deg). Functional images consisted of 18 axial slices collected using an EPI pulse sequence (FOV = 240 mm, matrix = 64 × 64, size = 3.75 × 3.75 mm, thickness = 4 mm, TE = 40 ms, TR = 2000 ms, flip angle = 90 deg).

The experiment used an event-related design. There were 8 runs with 30 trials per run. Each trial consisted of the stimulus followed by a pseudo-randomly varied period (3.9, 5.9, 7.9, or 9.9 seconds) of rest.

The functional images were corrected for motion artifact with a rigid-body, 6-parameter model using AIR 5.0 (Woods et al., 1998). The images were then warped into a standard anatomical space defined by the MNI brain. The functional volumes were spatially smoothed with a Gaussian filter (6 mm FWHM), and the time course of each voxel was temporally filtered with a high-pass butterworth filter (0.01 Hz).

For individual subjects, a deconvolution-based regression analysis was applied to each voxel (Ward, 2002). This included a separate set of regressors for each condition coding 12 different temporal lags. The resulting analysis produced a 12-lag (24-second) time course for each condition. The time courses for each condition were then baseline corrected using the average of the first and last lag as a baseline. Contrasts were calculated between conditions for every time point. Significance of the contrasts was determined using a random effects analysis corrected for multiple comparisons across both time and space. Statistical maps for each contrast were initially thresholded at an uncorrected probability level of p<.01. A size-based cluster threshold was found for the individual maps using the AFNI program AlphaSim to provide a corrected p-value of .0042. This corresponds to a Bonferroni correction for multiple comparisons across time of p<.05. The final maps were displayed on an inflated surface brain based on the ICBM single subject template brain atlas. Time courses of regions of interest were produced by averaging the time course of each condition across multiple voxels.

Table 1 Behavioral ratings

Activation for the six semantic contrasts was more robust in the left hemisphere, with the right hemisphere showing very few distinct clusters. For this reason, and for the sake of achieving a manageable discussion of the results, we have focused our analysis only on the left hemisphere. The results of the contrasts for time points between 4 and 16 seconds are shown in figure 1. In the first row, a contrast between congruent sentences and rest is provided to illustrate the temporal evolution of the stimulus activation compared to a resting baseline. Initial activity was seen in auditory cortex, including the left superior temporal gyrus and sulcus (STG and STS), as well as the left inferior frontal gyrus (IFG). A larger spatial extent of activation in both temporal and frontal lobes was observed at later time points, peaking roughly at 10–12 seconds after stimulus onset. Deactivation was observed in inferior temporal and occipital areas.

Figure 1 Maps of group activation at time points 4, 6, 8, 10, 12, 14, and 16 seconds after stimulus onset. The activations are shown projected onto a left-hemisphere, surface-modeled, inflated brain based on the ICBM Single Subject MRI Template. Activations are (more ...)

For the semantic contrasts, two areas, the left angular gyrus (AG) and the anterior-inferior aspect of the left IFG showed initial semantic effects apparent at roughly 6 seconds after stimulus onset. These effects were seen in all the contrasts between words and pseudowords, including semantically congruent sentences vs. pseudoword sentences, semantically random sentences vs. pseudoword sentences, and both semantically congruent and semantically random word lists against the pseudoword lists. No differences were seen for this time period between the semantically congruent and semantically random stimuli. Additional activation for the semantically congruent and semantically random stimuli over the pseudoword stimuli appeared later, beginning at roughly 10 seconds and peaking at 12 seconds after stimulus onset, in the middle and posterior aspects of the left STS and middle temporal gyrus (MTG) as well as in more posterior and dorsal regions of the left IFG. Differences between the semantically congruent and semantically random sentences were present in the left AG at roughly 10 seconds after stimulus onset and peaked at 12 seconds, as well as in the left frontal operculum and left supramarginal gyrus.

Based on these activation patterns, we focused further analyses on four regions: the left AG, left STS/MTG, and two regions of the left IFG. Each region was defined functionally by combining maps of the semantic contrasts that showed differential activity within the region. Voxels were included that showed significant differences in activation in any of the chosen maps. Further divisions were made based on the underlying anatomy of the MNI brain. A depiction of these four regions projected onto a surface brain is shown in figure 2. The left AG was defined from the three semantic contrasts for sentences between time points 6 and 14. The boundary between AG and temporal lobe was defined by the upward deflection of the STS. The STS/MTG region was defined using the congruent over pseudoword and the random over pseudoword contrasts for both sentences and word lists between time points 6 and 14. Voxels in the AG were excluded from this ROI. IFG regions were defined using the congruent over pseudoword and the random over pseudoword contrasts for sentences and word lists between time points 6 and 14. These contrasts revealed two adjacent clusters, including a more anterior-inferior region (roughly BA 47 and anterior BA 45) that was differentially activated early (~ 6s) in the contrasts between word and pseudoword stimuli and a more posterior region (roughly BA 44 and posterior BA 45) that appeared later (~10 s) in the contrasts between word and pseudoword stimuli. A division between these two regions was identified by hand. BOLD time courses for these four areas are shown in figure 2. Significance values of the semantic contrasts at each time point and for each area are shown in table 2.

Figure 2 Graphs of mean BOLD activation within clusters across time for the six conditions. Congruent sentences are shown in blue, congruent word lists in red, random sentences in green, random word lists in yellow, pseudo word sentences in cyan, and pseudoword (more ...)

Table 2 Significance values for semantic contrasts within activation clusters

The AG was characterized by an initial deactivation (decrease in BOLD signal) peaking at about 8–10 seconds after stimulus onset compared to the resting baseline for both pseudoword sentences and pseudoword lists. BOLD signal for the other conditions showed relatively little difference from the resting baseline at this point in the time course. Later, peaking at about 10–12 seconds after stimulus onset, an increase in activation compared to rest was seen only for the semantically congruent sentences.

The left STS/MTG showed an initial increase in activation compared to rest for all stimulus types; however, at about 6 seconds the activation levels diverged. The pseudoword sentences and lists showed the smallest response, peaking at around 8–10 seconds. The semantically random sentences and words lists showed a larger response than the pseudoword stimuli, while the semantically congruent sentences and word lists showed the largest response (this difference was not apparent in the voxel wise maps but was reliable in the region-of-interest analysis; see table 2). No differences were seen between sentences and lists in any of the three semantic conditions (congruent, random, and pseudoword); in other words, this region showed no sensitivity to the presence or absence of syntactic structure.

Both of the regions of interest in the left IFG showed greater activation for the word conditions (sentences and words lists) compared to the pseudoword sentences and lists. The anterior-inferior region showed almost no activation for the pseudoword stimuli compared to rest. The more posterior region showed a small amount of activation for the pseudoword sentences and word lists. While the posterior region showed no clear effects of semantic congruency for either sentences or word lists, the anterior-inferior IFG region showed greater activation for congruent word lists compared to random word lists but no semantic congruency effect for sentences.

We identified a distributed network of brain regions that showed differential responses to combinatorial and word-level semantic information while subjects processed sentences and word lists. This network included the left AG, middle and posterior aspects of the left STS and MTG, and left IFG. In the following paragraphs we consider the possible role each brain region plays within the larger semantic processing system.

The AG showed the most striking differences between conditions over time. Depending on condition and time after stimulus onset, BOLD signal in the AG either increased or decreased compared to the resting baseline. Our results suggest that the time course of activity in the AG can be divided into three phases: an initial phase peaking at 8–10 seconds, a later phase peaking at 12–14 seconds, and a phase associated with the resting baseline. The first phase was characterized by a decrease in signal in the two pseudoword conditions compared to the previous resting phase, and relatively little change for the other conditions, which all used real words. Thus, during this time there was an effect of single-word meaning but not of combinatorial semantic information. The timing of this phase indicates that the underlying neural activity is associated with processes occurring during presentation of the stimulus. Considered together, these findings suggest that in this phase the AG is involved in processing semantic information associated with the incoming words. Previous studies examining semantic processing with single words found a similar pattern of activity in the AG. For example, the AG showed higher BOLD signals for word trials compared to nonword trials during lexical decision tasks (Binder et al., 2003; Binder, Westbury, McKiernan, Possing, & Medler, 2005; Ischebeck et al., 2004; Rissman et al., 2003; Xiao et al., 2005) and higher signals during naming of visual words compared to nonwords (Binder, Medler et al., 2005). Furthermore, for those studies that included a “resting” baseline, a deactivation for nonwords compared to the resting state was seen, with no difference observed between words and resting (Mechelli, Gorno-Tempini, & Price, 2003; Binder, Medler et al., 2005; Rissman et al., 2003; Xiao et al., 2005), suggesting that there is a reduction in tonic levels of activity during nonword processing but not during word processing (Binder et al., 1999).

In the second phase, there was an increase in activity above the resting baseline for the semantically congruent sentences and relatively no change from baseline for the other stimuli. This phase thus showed an effect of combinatorial semantic information but only for the sentences and not the word lists. The timing of this activation suggests that it is due to additional processing that occurs only at the end of a semantically congruent sentence. This pattern is similar to findings from several other studies implicating the AG in semantic integration during sentence processing. For example, the AG shows greater activity for sentences with a semantically anomalous word than for sentences without a semantic violation (Friederici et al., 2003; Newman, et al., 2003; Luke, et al., 2002; Kuperberg et al., 2000; Ni et al., 2000; Kang, et al., 1999).

Finally, during the resting phase, the AG returns to a state of tonic activity. This return to baseline begins immediately after the stimulus ends in the pseudoword conditions, and is delayed by another several seconds in the case of the congruent sentences. It has been suggested that the AG is one part of a network of brain regions active during the resting state (Binder et al., 1999; Mazoyer et al., 2001; Raichle et al., 2001; Greicius et al., 2003). For example, in a meta-analysis of 9 PET studies using different tasks contrasted against a resting baseline, the AG was consistently more active during the resting state compared to the task (Mazoyer et al., 2001). Another study found deactivation in the AG during a perceptual tone discrimination task compared to resting, but no difference in the AG when a semantic retrieval task was compared to resting, suggesting that activity in the AG during the resting state is related to semantic processing (Binder et al., 1999). In a recent study examining the subjectively reported phenomenon of “stream of consciousness”, BOLD signal in the left AG was correlated with the likelihood of experiencing “task-unrelated thoughts” across a range of perceptual tasks that varied in degree of difficulty (McKiernan et al., 2006).

Several functions have historically been attributed to the AG, including multisensory integration (Geschwind, 1965; Damasio, 1989), “body schema” representation (Critchley, 1953), mathematical processing (Dehaene, 2004), semantic processing, and recognition of visual words (Déjerine, 1892). On the basis of our current data, we propose that one important function of the AG is integration of semantic information into an ongoing context. During the “resting” state when there are no external stimuli that require attention, the subject is engaged in internally-generated cognitive processes, often unrelated to the ongoing experiment and often experienced as semantically-related sequences of “thoughts” and “images” (Antrobus, Singer, & Greenberg, 1966; Antrobus, 1968; James, 1890; McKiernan, et al., 2006; Singer, 1993). These ongoing thoughts are part of a larger inner narrative arising from the subject’s current goals and needs, and are manifestations of planning and problem-solving activities integral to the attainment of goals and survival (Binder et al., 1999; Gazzaniga, 2000). We propose that the AG is involved in producing this inner narrative by combining elemental lexical concepts into proposition-like thoughts and connecting these thoughts in a larger framework resembling discourse. When an external stimulus is presented, attention is diverted from processing this internal narrative to processing the incoming stimulus (Pope & Singer, 1976). For stimuli comprised of words, the AG shifts from integrating internal semantic information to integrating the semantic information associated with each word in the stimulus, thus the amount of neural activity devoted to this combinatorial semantic processing is relatively unchanged. In the case of pseudowords, not only does the incoming stimulus interrupt processing of the internal narrative, but the level of semantic input to the AG is impoverished, producing a reduction in AG activity compared to the previous resting state. By the end of the stimulus, the subject has built an overall meaning for those stimuli possessing a congruent semantic theme. For the semantically congruent sentences, which are also the most meaningful, this results in an increase in activity above baseline as this complex combinatorial representation is computed, while for the incongruent word stimuli, which are less meaningful, the activity level remains relatively constant.

In contrast to the AG, the left STS/MTG showed activation to all stimuli above the resting baseline. This region showed greater activation for the semantically congruent sentences and word lists than for the semantically random sentences and word lists, which in turn showed greater activation than the pseudoword sentences and word lists. Thus, activation in the left STS/MTG was proportional to the relative amount of semantic information in the stimulus (congruent > random word > pseudoword), with no sensitivity to whether the stimulus was a sentence or a word list. The STS and MTG have been proposed to be part of a hierarchical pathway, extending from auditory cortex on the dorsal temporal plane to the STS and more ventral regions, involved in processing and identifying speech (for reviews see Binder et al., 2000; Hickok & Poeppel, 2004; Liebenthal et al., 2005; Scott et al, 2004). We believe, based on several lines of evidence, that the STS/MTG activity observed in the current study reflects processing of semantic information associated with individual words rather than complex word combinations. For example, results from a previous study that used voxel-based lesion mapping in patients with left hemisphere lesions showed a strong correlation between middle MTG lesions and word comprehension deficits, as opposed to lesions in surrounding anterior, posterior, and ventral areas that were more strongly correlated with deficits in sentence comprehension (Dronkers et al., 2004).

The fact that the STS/MTG did not show any sensitivity to syntactic structure across conditions provides evidence that this region may not play a significant role in representing the overall meaning of the stimulus. Syntactic structure provides strong cues for determining overall meaning, for example by indicating thematic roles and case relationships. The importance of syntactic structure is clearly reflected in the current study by the subjects’ meaningfulness ratings. For example, the subjects rated the congruent sentences as much more meaningful than the congruent word lists, and the random sentences as more meaningful than the random lists. The pattern of activation in the AG was highly similar, across conditions, to these ratings. Activation in the STS/MTG, however, was less correlated with these meaningfulness judgments.

Nevertheless, the greater amount of BOLD signal observed for congruent over random stimuli suggests that the STS/MTG is involved in some way in combinatorial semantic processing, a discrepancy that can be explained in the following manner. During combinatorial semantic processing, words are grouped together to form new meanings, a dynamic process in which the individual word meanings are maintained and re-accessed throughout the course of the stimulus. Maintaining these semantic representations explains why the STS/MTG shows greater activation for the stimuli with real words than for pseudowords, especially later in the trial; however, it does not explain the greater activation for the congruent over the random stimuli, since presumably the words in the random stimuli would also be maintained and re-accessed. However, a further difference between processing the congruent and random stimuli has to do with what happens when a combinatorial representation is successfully formed. When two words are thematically related, their combination generally constrains and sometimes changes the original meanings of each word. For example, one of the stimuli used in the experiment was The man on vacation lost a bag and a wallet. By itself, the word bag has a broad semantic representation and could refer to many different types of objects: garbage bags, tea bags, hand bags, etc. In the context of the sentence, it becomes apparent that the bag being referred to is something that can be brought on vacation, thereby constraining the meaning of the word to something like suitcase. In neural network terms, this process can be thought of as resulting from competitive inhibition, in which inappropriate units are inhibited, further strengthening the signals in those units that form the correct representation (McClelland & Rumelhart, 1981). During combinatorial semantic processing, this tuning to the precise meaning is likely an ongoing, recurrent process, in which the semantic representations of individual words are constantly being altered to fit the overall theme of the other content words. In the brain, this process would manifest as a reactivation of the neural regions encoding word level representations. Thus, it appears that the STS/MTG processes the incoming words in the stimuli and then relays these representations to other areas, such as the AG, which work to build an overall semantic representation. During the course of this process, these other areas feed information back to the STS/MTG, allowing the meanings of the words to be updated and causing additional activation for the thematically congruent stimuli.

Both regions in the left IFG showed activation for semantically congruent and semantically random sentences and word lists, with reduced activation for the pseudoword sentences and pseudoword lists. The left IFG has been previously implicated in studies investigating semantic processing. For example, when activation for a semantic task is compared to a non-semantic task, such as a phonological task, greater activation during the semantic task is often seen in anterior parts of the IFG corresponding to Brodmann areas 47 and 45 (Binder et al., 2003; McDermott et al., 2003; Mechelli et al., 2006; Poldrack et al., 1999; Roskies et al., 2001).

It has been argued that the prefrontal cortex functions in an executive capacity by providing control for ongoing cognitive processes (Miller & Cohen, 2001). The current study suggests that one specific cognitive process the IFG regulates is semantic processing. This is supported by the fact that both areas of the IFG showed greater activation for stimuli with real words than for pseudoword stimuli. These differences were seen early on and continued throughout the trial, suggesting that the IFG is active throughout the processing of the stimulus. Of note, the IFG did not show an effect of combinatorial information. In fact the congruent word lists and random sentences showed slightly higher activation than the congruent sentences and random word lists. It has been proposed that the prefrontal cortex is involved in semantic processing by helping to select between competing semantic alternatives (Thompson-Schill, 1997) or in top-down retrieval of semantic information (Wagner, Pare-Blagoev, Clark, & Poldrack, 2001). The increased activation for the congruent word lists and the random sentences could be related to increased difficulty in selecting a response or in retrieving semantic information relevant to the meaningfulness task for these stimuli. The congruent word lists contain thematically related words but lack syntactic structure, and the random sentences possess syntactic structure but lack thematic relations between words. These conflicting characteristics may have made the criteria for selecting a response less clear for these conditions.

There were several differences observed between the more anterior-inferior region of the IFG and the more posterior-dorsal region. The posterior IFG showed an increase in signal above baseline for the two pseudoword conditions, whereas the anterior region showed no pseudoword activation. As in the temporal and parietal lobes, this difference in the response to pseudowords could reflect a distinction between early morphological processing and later processing associated with a more specifically semantic representation. Thus, a processing hierarchy might exist between posterior IFG, which focuses to a greater extent on structural features of the input, and anterior IFG, which plays a greater role in semantic processing. In addition, the anterior region showed less activation for the random word lists, which have neither combinatorial semantic information nor syntactic structure, than for the other word conditions, further suggesting that the anterior IFG may have a greater involvement in semantic processing than does the posterior IFG. An alternate explanation, which is also supported by our data, is that the anterior IFG may function exclusively in top-down retrieval of semantic information while the posterior IFG is involved in a more general process of selecting between competing alternatives (Badre, et al., 2005).

This work was supported by NINDS RO1 NS33576 and NIH General Clinical Research Center Grant M01 RR00058.