Element-arrangement texture patterns, introduced by Jacob Beck and his colleagues (Beck, 1982; Beck, Prazdny, and Rosenfeld, 1983; Beck, Sutter, and Ivry, 1987), have proven to be a powerful tool for exploring intermediate stages of visual processing. Element-arrangement textures are created by arranging two element types in a checkerboard or striped manner. An example pattern is shown in Figure 1. In this example the two element types are small and big squares, respectively, and the elements have a checkerboard arrangement in the center and a striped arrangement in the rest of the pattern.

Figure 1 Classic element-arrangement texture pattern

One of us (NG) started to work with Beck in the mid-1980’s, as an offshoot of a visit to his laboratory and an argument about the role of spatial-frequency and orientation-selective channels in texture perception. In a series of papers, starting with Sutter, Beck, and Graham (1989), these investigators (working together and later independently with other colleagues) used element-arrangement patterns to investigate the processes underlying intermediate level vision, and in particular, to learn about the kinds of rigorously-defined nonlinearities one could include in models containing spatial-frequency and orientation-selective channels to explain even more of texture segregation.

Here we review some of the major results of these studies, all of which owe a great deal to Jacob Beck even when he was not explicitly an author.

We also present some new experimental results. These continue to use element-arrangement patterns as stimuli but now use several objective perceptual tasks, involving detection and identification of different features of the patterns, rather than the subjective rating task of perceived segregation which was used in most of the earlier studies. These new results replicate important previous results using the subjective task as will be shown here. But we started using these tasks because we wish to go on to investigate properties of intermediate-level vision that required finer-grained distinctions than seemed possible using the subjective task. We present some preliminary new results here as well.

The form of the spatial nonlinearity is well described by complex (second-order) channels as was suggested to explain results with element-arrangement textures by Sutter, Beck, and Graham (1989), to explain some results with moving stimuli by Chubb and Sperling (1988, 1989), and to explain results in other contexts by a number of other people (see Graham and Sutter, 1998, for references). A complex channel consists of two layers of filtering – the first sensitive to higher spatial frequencies than the second – with an intermediate nonlinearity. Complex channels are sufficient to describe many texture segregation results that simple (first-order) channels –consisting of a single layer of filtering – are not. (See Graham, Beck and Sutter, 1992, for a detailed discussion of this; see Landy and Graham, 2003, and Schofield, 2000, for recent discussions of the importance of second-order processing in visual perception.)

A great deal has been learned about these complex channels through experiments with element-arrangement textures and with other similar patterns. (i) The spatial-frequency and orientation bandwidths of the first stage (Graham, Sutter, and Venkatesan, 1993) and of the second stage (Landy and Oruc, 2002; Ellemberg, Hess, and Allen, 2004) have been estimated. (ii) A slight correlation between the peak orientations or spatial frequencies at the first and second filters of individual channels has been found (Graham and Wolfson, 2001; Sutter, Sperling, and Chubb, 1995). (iii) The overall contrast sensitivity function of these channels, that is, the envelope of the individual channels’ sensitivity function has been measured (Sutter, Sperling, and Chubb, 1995; Landy and Oruc, 2002). (iv) Processing by complex channels has been found to be slower than that of the simple channels (Sutter and Graham, 1995; Sutter and Hwang, 1999). (v) The intermediate nonlinearity in the complex channels has been found to be expansive (Graham and Sutter, 1998), a point we will be returning to below.

Other texture segregation results suggested that, in addition to this spatial nonlinearity, a heavily compressive intensive nonlinearity was necessary (Graham, 1991; Graham, Beck, and Sutter, 1992). The possibility that the compressive effect was entirely an early, local nonlinearity occurring before the spatial-frequency and orientation-selective simple or complex channels (e.g., retinal or LGN light-adaptation) was considered extensively and – after a great deal of work –discarded (Beck, Graham, and Sutter, 1991; Graham and Sutter, 1996, 2000). There can, however, be effects of light adaptation if the luminances vary widely enough (e.g., Sutter, Beck, and Graham, 1989).

Instead, the highly compressive intensive nonlinearity is well described by inhibition among channels which can be modeled as a normalization network. Normalization is a contrast-gain-controlling process where the gain depends on the output of a collection of channels. In a normalization network the output of a channel is normalized by (is divided by, has its gain set by, is inhibited by) the output of a collection (pool, set) of channels. The collection of channels encompasses a wide range of spatial frequencies and orientations (but not necessarily a wide range of space).

Sketched in Figure 2 is a model that includes a normalization network. Such a model is consistent with all of our texture results. Normalization is also consistent with physiological results and others’ psychophysical results (see Graham and Sutter, 2000, for some references). It is understandable why the visual system has such a process. Normalization can prevent overload on higher levels by repositioning the limited dynamic range to be centered near the ambient contrast level and at the same time preserve selectivity along dimensions such as orientation and spatial frequency. (See discussions and references in, e.g., Bonds, 1993; Heeger, 1991; Lennie, 1998; Victor, Conte, and Purpura, 1997.) Normalization has also been suggested to have the right properties to help encode natural images efficiently (Simoncelli and Olshausen, 2001; Schwarz and Simoncelli, 2001).

Figure 2 Model with complex channels and normalization

Note that, while all the previously mentioned studies with element-arrangement patterns were done with monochromatic (gray-level patterns) as will be our studies below, Beck and collaborators have collected results with chromatically-varying element-arrangement patterns as well (Beck, 1994; Pessoa, Beck, and Mingolla, 1996; Oddo, Beck, and Mingolla, 1999).

Consider the space of element contrasts as sketched in Figure 3. The contrast of one element type is shown on one axis and the contrast of the other element type is shown on the other axis. A big black dot in the diagram represents a stimulus composed of elements at those two contrasts. The contrasts are in arbitrary (positive) step units (which will be defined for each particular task). Each of the thick gray right diagonal lines lays on top of a group of stimuli that are from a single constant-difference-series.

Figure 3 Diagram of element contrasts in our patterns

The predictions for constant-difference-series results – from several classes of models – are sketched in Figure 4. Each curve shows the results from one constant-difference-series (i.e., from the stimuli along one thick gray right diagonal line in Figure 3). The middle panel shows predictions from a model containing only (i) simple linear channels and (ii) complex channels containing a piecewise linear rectification at the intermediate stage (between the two layers of filters in the complex channel). A down-turn at the end of the constant-difference-series with a common envelope for all the curves (right panel) is indicative of a compressive nonlinearity (such as normalization). An up-turn of the curves (left panel) is indicative of an expansive nonlinearity

Figure 4 Nonlinearity characteristics and curve shapes

The predictions sketched in Figure 5 show an up-turn of the curves (characteristic of expansion) at low contrast-ratio-angles and a down-turn (characteristic of compression) at high contrast-ratio-angles. These predictions are from a model having an expansive intermediate nonlinearity in the complex channels and a normalization network. (See Graham and Sutter, 2000, for details and intuitions.) Results from the previously published work using subjective rating tasks are well accounted for by such a model (Graham, Beck, and Sutter, 1992; Graham and Sutter, 1996, 2000). See those publications for further explanation of the predictions from constant-difference-series experiments, including a discussion of area-contrast-tradeoff experiments, which are important but we will not discuss further here for lack of space (Sutter, Beck, and Graham, 1989; Graham and Sutter 1998, 2000).

Figure 5 Nonlinearity characteristics and more complicated curve shapes

Figure 6 Pieces of the textures used in this study

The five tasks were:

• Region Identification. For this task we used patterns with “embedded rectangles”. A caricature of a pattern with a vertically-elongated embedded rectangle is shown in the left panel of Figure 7. On each trial a pattern with an embedded rectangle region is shown. The rectangle is elongated in the horizontal or vertical direction and the observer indicates the elongated orientation of the embedded rectangle.

  Figure 7 Sketches of patterns
• Stripe Identification. For this task, we used patterns with only one region (that is, no embedded rectangle), as in the sketch in the right panel of Figure 7. On each trial a horizontally-striped pattern or a vertically-striped pattern is shown and the observer indicates the orientation of the stripes.
• Uncertain Detection. On each trial a horizontally-striped pattern or a vertically-striped patterns or a uniform pattern is shown and the observer indicates the presence or absence of stripes. The observer knows that there will be randomly intermixed horizontally- and vertically-striped patterns.
• Certain Horizontal Detection. On each trial a horizontally-striped pattern or a uniform pattern is shown and the observer indicates the presence or absence of stripes. The observer knows that all the striped patterns in the session will be horizontal.
• Certain Vertical Detection. The same as Certain Horizontal Detection but with vertical stripes.

We will also ask about the relationship of the three Detection and one Stripe Identification tasks to one another. Do models of complex (second-order) channels that are analogous to common models of simple (first-order) channels account for these results? What do these tasks have to say about several characteristics of complex channels: In particular, are the outputs of different channels probabilistically independent? Are the channel outputs labeled? Is an observer able to monitor some channels but not others (showing an ability to selectively attend)? Is the observer able to monitor all relevant channels (showing no attention capacity limitation)?

This section contains details of the methods that were not included in the Introduction. The reader should be able to skip these details without loss of continuity.

In the Region Identification task a rectangle was embedded in the background as shown schematically in the left panel of Figure 7. The rectangle was 7 x 11 elements in size, oriented vertically or horizontally, in three possible positions (see Graham and Wolfson, 2001, for details). One region was always vertically-striped and the other horizontally-striped.

In the other tasks, the whole pattern was one region of horizontally-striped, vertically-striped, or uniform texture. Small pieces of these textures are shown in Figure 6; a vertically-striped pattern is sketched in the right panel of Figure 7.

In the Stripe Identification task a vertically-striped or horizontally-striped patterns was shown on each trial.

In the Uncertain Detection task, a vertically-striped, a horizontally-striped, or an uniform pattern was shown on each trial. In the Certain Horizontal Detection task only horizontally-striped and uniform patterns were used. In the Certain Vertical Detection task only vertically-striped and uniform patterns were used.

The stimuli shown in Figure 3 here as big black dots do not accurately represent the stimuli used in this study. In particular, the number of constant-difference-series (3 are shown in this figure), the length of the series (shown as 5 in this figure), and the size of the contrast step (shown as 1 in arbitrary units) vary. Also, which element type is called element 1 and which is called element 2 in such a diagram is arbitrary.

For all three Detection tasks, if one were to diagram their stimuli like Figure 3, there would be 9 dots on each of 4 lines. The size of the contrast step for both elements was 5%. Thus the sizes of the constant differences were 0%, 5%, 10% and 15%.

For the Stripe Identification task there would be 9 dots on each of 3 lines. The size of the contrast step was 4%. The sizes of the constant differences were 4%, 8%, and 12%. (The contrasts are smallest here because pilot experiments had suggested that this is the easiest task.)

For the Region Identification task there would be 7 dots on each of 4 lines. The size of the contrast step was 9%. The sizes of the constant differences were 9%, 18%, 24%, and 36%. (The contrasts are largest here because pilot experiments had suggested that this is the hardest task.)

To clarify the constant differences and contrast-ratio-angles, here is an example. Consider the 9% constant-difference-series for the Region Identification task. This particular constant-difference-series is best represented in the diagram of Figure 3 by the middle thick gray line. This series in our Region Identification task has 7 contrast pairs (dots), so it would extend from (0,1) to (5,6) rather than from (0,1) to (4,5) as show in Figure 3. If we convert from these arbitrary contrast steps of Figure 3 to real contrast values, the contrast pairs in the Region Identification task are (0%, 9%), (9%, 18%), (18%, 27%), (27%, 36%), (36%, 45%), (45%, 54%), and (54%, 63%). These correspond to contrast-ratio-angles of about 45, 72, 79, 82, 84, 85, and 86 degrees.

For the three Detection tasks the observer’s response was to indicate (with the ‘y’ or ‘n’ key) whether a striped pattern had been seen or not. There were 432 trials per session (presented in random order). Half of these trials were dedicated to the zero constant-difference-series for which the correct answer was ‘n’; these trials were distributed equally across the 9 dots in this series. The other half were dedicated to the three other constant-difference-series (distributed equally across dots) for which the correct answer was ‘y’. Each observer completed 4 or 5 sessions in each Detection task.

For the Stripe Identification task the observer’s response was to indicate whether the pattern was vertically-striped (‘up arrow’ key) or horizontally-striped (‘right arrow’ key). There were 432 trials per session (presented in random order and equally distributed across the 3 series each of 9 dots). All observers completed 5 sessions except for UJT who completed 3.

For the Region Identification task the observer’s response was to indicate whether the embedded rectangle was oriented vertically (‘8’ key) or horizontally (‘4’ key). There were 672 trials per session (presented in random order and equally distributed across the 4 series each of 7 dots). All observers completed 5 sessions except for UJT who completed 3.

Figure 8 Normalization results from the objective tasks

We also ran two observers in a constant-difference experiment using patterns composed of Gaussian-blob elements and found the same general pattern of results.

The shapes of the curves in Figure 8 have implications for the underlying as shown in Figures 4 and 5. The down-turn of the ends of the curves (at high contrast-ratio-angles) indicates a compressive nonlinearity. The up-turn of the curves (at low contrast-ratio-angles) indicates an expansive nonlinearity. The predictions in Figure 5, which match the data in Figure 8 quite well, are predictions from a model (like that in Figure 2) with an expansive intermediate nonlinearity in the complex channels and a compressive intensive nonlinearity in the form of inhibition among channels in a normalization network.

Uncertainty effects for complex (second-order) channels are like those for simple (first-order) channels We are interested in the comparison of observer’s performance in conditions of certainty versus uncertainty, that is, the comparison of performance in the Certain Horizontal and Vertical Detection tasks with that in the Uncertain Detection task. This is a standard type of experiment that has been done on many dimensions (e.g., first-order spatial-frequency, in Davis, Kramer, and Graham, 1983; Kramer, Graham, and Yager, 1985; see also Graham, 1989, 1992, for models, and for results on other visual dimensions). These experiments provide evidence about the probabilistic independence and labeled outputs of underlying channels as well as about possible limitations to the observers’ ability to attend to particular subsets of channels. In the case of the experiments reported here, the channels are complex channels and the two subsets at issue are (i) the subset containing the complex channels having second-stage filters sensitive to our horizontally-striped patterns and (ii) the subset containing channels sensitive to our vertically-striped patterns.

The result we found is a common result on many dimensions: observers perform somewhat (but not dramatically) less well when they are uncertain than when they are certain. This result is seen for all of the observers except for GTM. (One could, in principle, see this result in Fig. 8, but the difference in performance is too small to be readily seen in this format. Further, to analyze this properly one needs to compute d’ values to avoid misleading effects of response biases.)

This kind of small difference in performance between conditions of certainty and uncertainty can be explained by a standard signal-detection model with the assumptions – precisely analogous to the assumptions for simple channels – that (i) these channels are noisy and probabilistically independent, and (ii) on each trial, the observer can attend in parallel to all the channels that may receive stimulation on that trial while NOT paying attention to the outputs of other channels. (In other words, the observer can “exclude” distracters or external noise with no capacity limitation. This kind of attention need not be conscious, of course.) Note that this assumption about attention requires that the outputs of the channels be sufficiently “labeled” that some process upstream can know which output comes from which channel so that selective attention can occur. We see no reason, therefore, to suspect that there is any more attention limitation for the complex channels processing static patterns than for the simple channels doing so. (Processing of motion may be different in this regard. See Ashida, Seiffert, and Osaka, 2001; Lu, Liu, Dosher, 2000.)

Although Jacob Beck was not directly involved in these uncertainty experiments, they nonetheless reflect his long-term influence on research in perception. He consistently pointed out –in publications, formal talks, and conversations – the importance of considering whether visual processing was occurring in parallel (without being limited by attention constraints) or not. See, for example, Beck (1972, 1975).

Relationship of identification and detection by complex (second-order) channels The comparison of detection and identification performances (the three Detection tasks compared to the Stripe Identification task) requires further results than those shown in Fig. 8 in order to compare detection and identification results appropriately. (In particular, the exact same contrast levels have to be used on all of the tasks to be compared.) We have collected the necessary results for two observers to date. These results are also consistent with the standard signal-detection model with the assumptions that (i) the complex channels sensitive to the vertically-striped and to the horizontally-striped patterns are probabilistically independent, and (ii) the outputs of the channels are labeled sufficiently that some process upstream knows which channel detected the stimulus. Ellemberg, Hess, and Allen (2004) have also reported preliminary results consistent with these assumptions. (See Graham, 1992, and Graham, 1989, chapter 10, for introduction to these models.)

Stripe Identification is the hardest task and Region Identification is the easiest If one considers an observer’s performance on a particular stimulus (at particular contrast levels) for all five tasks, then one finds that percent correct is highest for the Stripe Identification task and lowest for the Region Identification task. This may not be immediately evident from Figure 8 since we adjusted the contrast levels in order to keep the results for all tasks in the informative range. However, note that higher contrasts were used in the Region Identification task than in the other tasks but did not produce superior performance. (Similarly, lower contrasts were used in the Stripe Identification task than in the other tasks but did not produce inferior performance.)

The observant reader might have noticed that there were more trials per session in the Region Identification task than in the other tasks, so the low performance in the Region Identification might be thought to be due to exhaustion. However, we ran a control experiment in which the Stripe Identification and Region Identification tasks had the same number of trials per session and the same contrast levels, and it was still true that performance in the Region Identification task was much worse than in the Stripe Identification task.

As explained in the preceding subsections, the inter-relationship of Stripe Identification and the three Detection tasks is as would be expected from standard signal-detection models of multiple channels.

What is left to explain, therefore, is why the Region Identification task is so much harder than the four other tasks. The difference can be considerable: For example, it is possible to find situations where the Region Identification performance is barely above chance, say 65%, and Stripe Identification is essentially 100%. Or, Region Identification performance is at chance but Stripe Identification is still at 85 or 90%. At a qualitative level, this difference between the Region Segregation task and the others does not seem surprising. To do the Region Identification task requires – at the very least – processing of a wider area of the stimulus than do any of the other tasks. However, the processes that are involved in integrating over this wider area have not been explicated at anything like the level of rigor of the processes shown in Figure 2.

For the dynamics-of-normalization experiments described in the next subsection, it is important to note the following: The Region Identification task (like the subjective rating of region segregation we used in many earlier studies) may well tap into additional higher-level processes not revealed in the Stripe Identification and Detection tasks. However, the signature of normalization is just as well revealed in these tasks as in the Region Identification task. Thus, we can use the more sensitive Stripe Identification task for the further studies of normalization we are currently engaged in and that are briefly described in the next subsection.

Dynamics of normalization Very little is known psychophysically about the dynamics of any contrast-gain-controlling process in any perceptual task. And yet dynamic characteristics are a very good way of telling different candidate processes apart. And, more importantly, dynamic properties are extremely important for the functions of perception.

Previously we have investigated psychophysically the dynamics of retinal light adaptation by using a flickering background stimulus (a flickering large disc of light) to drive the light adaptation process and probing it with a brief-duration test stimulus (a small disc of light); this paradigm proved very powerful in discriminating among different models of light adaptation (Hood, Graham, von Weigand, and Chase, 1997; Hood and Graham, 1998; Wolfson and Graham, 2000, 2001a, b).

We have now generalized this paradigm in order to study the dynamics of the normalization contrast-gain control process in texture perception. We are using a uniform element-arrangement pattern – contrast modulating in time – as the flickering background stimulus to drive the normalization process. We then probe with a brief-duration introduction of a horizontally-striped or vertically-striped element-arrangement pattern at various phases with respect to the temporal modulation of the background pattern. The observer is required to identify the orientation of the stripes in the probe (i.e., a Stripe Identification task). This experiment is trickier than the light-adaptation experiments for several reasons, but one is the limited range of contrasts available for the probe. Using a Stripe Identification task, however, appears to offer us enough contrast range. For example, one preliminary result is that, for some patterns, the temporal modulation of the background stimulus appears to produce greater probe threshold elevation as the temporal frequency of the modulation increases (up to at least 4 or 8 Hz). To what extent this effect represents action only of the contrast-gain-control processes rather than of (retinal) light adaptation remains to be clarified, however.

A long series of published studies using element-arrangement textures (introduced by Beck and colleagues in the 1980s) has revealed a number of properties of visual pattern processing. Here we use element-arrangement patterns in five different objective perceptual tasks to investigate several more fine-grained questions about pattern processing. We find:

• The results of all five objective tasks replicate the results from subjective rating procedures in showing both the compressive effect of a normalization contrast-gain-controlling process and the expansive effect of the intermediate nonlinearity in the complex (second-order) channels.
• Comparisons of the Detection tasks (under conditions of certainty and uncertainty) and the Stripe Identification task with one another suggest that the complex channels are noisy and probabilistically independent, with labeled outputs so that processes upstream can selectively attend to subsets and can identify them. Further there is no apparent capacity limitation to this attention, at least when only two subsets of channels are involved. The complex channels are like the simple (first-order) spatial-frequency- and orientation-selective channels in this regard.
• We have begun to use the most sensitive of the tasks (Stripe Identification) to explore the dynamics of normalization.

Our current research benefits from, and our future research will continue to benefit from, the visual patterns Jacob Beck introduced, the perceptual phenomena he described, and the questions about visual processing that he raised.

This work was supported in part by National Eye Institute grant EY08459. Some of these results were presented at the Spring 2004 VSS meeting (Wolfson, Graham, and Slinin, 2004). We thank our observers for their hours of effort, and Oleg Slinin and Uma Thakur for performing many of the calculations that support the conclusions about uncertainty effects and the relationship of detection and identification.

A note from one of us (NG). During Jake Beck’s research lifetime, the scientific understanding of visual processes developed enormously. And it seemed to me that he enjoyed every step of the way. Each new discovery of others – as well as of his own – he greeted with enthusiasm and pleasure. Working together – either on studies we were doing together, or just talking with him about research we were doing with others – was exciting.