For over 100 years, scientists have observed and speculated on the often prolonged and spatially intimate relationships between different species of grouping animal (Morse 1977; Terborgh 1990; Krause & Ruxton 2002; Lukoschek & Mccormick 2002; Stensland et al. 2003; Caro 2005). Zebra, Equus burchelli, and wildebeest, Connochaetus taurinus, of the Serengeti, for example, remain in association during periods of long distance migration (Pennycuick 1975; Sinclair 1985), and species such as blue and redtail monkeys (Cercopithecus mitis and Cercopithecus ascanius, respectively) of the Kenyan Kakamega forest can stay in close proximity, taking exactly the same paths of movement through the forest, for periods of up to 12 hours (Cords 1990). As one can intuitively envisage, rates of encounter between individuals of different species in these and many other examples are statistically far more common than would be expected purely by chance (Stensland et al. 2003). Thus, mixed-species groupings cannot be explained as the trivial by-product of coincidence in the same habitat; rather, they form by the active behavioural choice of at least one species.

A key benefit of these mixed-species associations may be that they help in evading predators (Morse 1977; Terborgh 1990; Krause & Ruxton 2002; Lukoschek & Mccormick 2002; Stensland et al. 2003; Caro 2005), and mechanisms proposed include dilution of individual risk by increasing absolute group size, early detection of common predators, and confusion of predators through absolute numbers or mobbing (see Krause & Ruxton (2002) and Caro (2005) for further benefits and discussion). Most of these mechanisms apply equally to the benefits of single-species grouping, and in this article we present a mechanism that is specific to mixed-species groups because it depends critically on differences in the appearance of individuals of the different species. The model incorporates components that apply to a wide spectrum of predator–prey relationships and so is likely to be very general in application. It involves confoundment of image reconstruction in the retinotopic sensory maps (which have a similar topographic structure to the sensory surface) of predators, resulting from differences in the appearance of prey in mixed-species associations. Retinotopic maps occur universally in animals (Ashley & Katz 1994; Baier et al. 1996; Shipp 2004) and form a key component of many models of visual attention (Shipp 2004). The basic modelling framework for retinotopic mapping presented here has, moreover, proved extremely powerful in explaining the ‘confusion effect’ (decreasing predator targeting accuracy with size of single-species prey group) and most of its ameliorating factors (Tosh et al. 2006). If mixed-species grouping is promoted by a sensory mechanism, then there is a good reason to believe that this sensory motif could play a key role.

We present a simple (but generic) artificial neural network model of image reformation in retinotopic maps. Networks are trained to reproduce projections of single- and mixed-species prey groups in a map, and the ability of trained networks to subsequently reproduce projections of individual prey in mixed-species groups of various compositions is gauged. We demonstrate that, over a majority of parameter space, the difference in appearance of individuals of the two species in a mixed group could benefit individual prey by confusing the process of image reconstruction in the predator's sensory map. Mixed-species grouping is invariably predicted to benefit individuals of one species more than the other in the grouping relationship; a pattern seen widely in natural mixed-species groups (see §4). The ubiquitous occurrence of retinotopic maps and the agreement of fundamental model predictions with many published empirical studies indicate that we may have uncovered a source of natural selection in the evolution of mixed-species grouping in animals.

Our use of the word ‘targeting’ to describe these procedures is a convenience measure to assist intuitive understanding of the paper, and reconstruction accuracy in retinotopic maps is only one of many processes that determine the accuracy of object targeting by organisms (including the neural processes of visual attention and subsequent motor control). However, we can argue strongly that this process is very likely to closely correlate with targeting accuracy. Regions of retinotopic maps that are activated ‘falsely’ (in the absence of a projected object at the corresponding area of the retina) can attract attention in the cognitive apparatus of predators and promote strikes on empty regions of the environment. Conversely, regions of retinotopic maps that are not activated in the presence of an object in the corresponding retinal area (the ‘targeting accuracy’ criterion used in the present article) are unlikely to attract attention and strikes by the predator, representing a missed opportunity for the predator and a mechanism for predator avoidance by prey. We have also shown in a previous study (Tosh et al. 2006) that the output of the present model closely follows patterns of predator strike accuracy in a variety of organisms. Some other conventions used should be discussed here. The choice of a central map node to quantify reconstruction is arbitrary, because in networks with all nodes in one layer connected to all in the next, all output nodes are topologically equivalent. The practice of measuring reconstruction accuracy with a single node (rather than across the whole network) is carried out both because it has proved a powerful method previously, and also because analysis of the single prey organism is the most appropriate level with which to translate observed phenomena into arguments on behavioural evolution of predators and prey. Finally, the use of a binary (presence–absence) map arises principally from the training algorithm employed; however, numerous types of feature map are proposed in models of visual attention (Shipp 2004) and it is conceivable that such a map could play a role in visual attention.

We modelled four scenarios of predator–prey interaction. In the first, the predator was trained on single-species groups (of variable size) of conspicuous-looking prey. Network training can be considered analogous to a predator specializing on a particular type of prey group during the course of its development. ‘Conspicuous’ prey are very different in appearance from the background against which they are viewed; we call the contrasting case of prey that are closer to matching the background ‘cryptic’. The predator then targeted individuals in mixed-species groups (size=10) containing a conspicuous and a cryptic species. Various characteristics of the targeted groups (1: visual intensity of the cryptic species—the intensity of the conspicuous remained static at value=1; 2: relative numbers of the conspicuous and cryptic species; and 3: individual—conspicuous or cryptic—targeted by the network) were varied between groups to analyse the relationship between targeting accuracy and group characteristics. Scenario 2 was essentially as scenario 1 except that the predator trained on cryptic prey, and during targeting it was the intensity of the conspicuous species that, among other factors, varied. Scenario 3 involved training on mixed-species groups of a conspicuous and cryptic species whose visual intensity remained static between groups. Absolute numbers per group and relative numbers of the conspicuous and cryptic species per group varied. Targeting was then on a mixed-species group as scenario 1. Finally, scenario 4 involved training on mixed-species groups exactly as scenario 3. Targeting then involved mixed-species groups as scenario 2 (where the intensity of the conspicuous species, among other factors, varied between groups). Running of these different training/testing treatments allowed us to determine if the principal results from the study could be applied across various training and targeting regimes. Readers should refer to the electronic supplementary material of the article for detailed modelling procedures.

In our first investigation, denoted scenario 1, the predator (network) trained on multiple, single-species groups of conspicuous prey and subsequently targeted individuals in mixed-species groups. These targeted groups varied in several characteristics (visual intensity of the cryptic species, relative numbers of the conspicuous and cryptic species, individual—conspicuous or cryptic—targeted by the network) to allow analysis of the relationship between targeting accuracy and group characteristics (absolute group size was fixed at 10 individuals, to ease visualization and interpretation of results).

Figure 1 reveals that a number of grouping strategies, most of which can be positively identified as occurring in nature, confound accurate reconstruction of target images in the sensory map and hence could be favoured by natural selection on prey. The particular strategy favoured depends only on the relative visual intensity of prey items. When cryptic prey differ greatly in appearance from conspicuous prey (conspicuous species=1 and cryptic species=0.1), target prey are most poorly reconstructed in the spatial map of predators when all individuals in the group have the same appearance as the targets, i.e. when species separate into same-species groups. As the cryptic species becomes more similar in appearance to the conspicuous species, this continues to be the case for the conspicuous species, but the favoured strategy of the cryptic species gradually changes so that when the two species are relatively similar in appearance (conspicuous species=1 and cryptic species=0.7), individuals of the cryptic species are in fact targeted least accurately when they occur in small numbers within mixed-species groups. When the two species are very similar in appearance, predator performance is insensitive to the specific composition of the group, entirely as we would expect.

Figure 1 Representation of how various prey grouping strategies confound reconstruction of target prey in the predator's sensory map when predators have trained on single-species groups of conspicuous prey (scenario 1). Prey groups consisting of two species that (more ...)

The further three scenarios considered—(s2) single-species (cryptic) training with mixed-species targeting (cryptic training species always present), (s3) mixed-species training with mixed-species targeting (conspicuous training species always present), and (s4) mixed-species training with mixed-species targeting (cryptic training species always present)—all predicted a similar pattern across most of the parameter space to that for scenario 1 with species intensity values: conspicuous species=1 and cryptic species=0.7. Representative results for scenario 3 are shown in figure 2. Except when the prey species are very similar looking, when no grouping strategy strongly confounds predator targeting, the more conspicuous prey species is most poorly targeted when in a same-species group, and the cryptic species benefits from association with the more conspicuous species, especially when the former occurs in small numbers. Thus, over a majority of the parameter space considered in this study (most of the parameter space considered in scenarios 2–4 and some of the space in scenario 1), cryptic prey benefit from association with conspicuous prey (but not vice versa) through confusion of the image reconstruction process in predatory neural networks.

Figure 2 How various prey grouping strategies confound reconstruction of target prey in the predator's sensory map when predators have trained on mixed-species groups (scenario 3). Results shown are representative of all of the scenarios simulated, with the exception (more ...)

While the model presented here predicts single-species grouping under clearly delineated circumstances, the most significant prediction is that mixed-species grouping could be favoured by a number of plausible combinations of predator experience (training) and targeting strategies, and when mixed-species grouping occurs, it will be ‘asymmetric’ with regard to benefits of the relationship. It seems relevant, therefore, to ask how often in nature one partner in a mixed-species association is more active than the other in initiating and maintaining the relationship? It is known that zebras tend to follow wildebeest in their association, and not vice versa (Sinclair 1985), and redtail monkeys of the Kenyan Kakamega forest are also known to initiate and maintain relationships with blue monkeys more than the reverse (Cords 1990). Striped parrotfish, Scarus iserti, in mixed-species shoals with stoplight parrotfish, Sparisoma viride, and ocean surgeonfish, Acanthurus bahianus, when threatened by predators, have a tendency to associate with fish shoals, whereas the latter two species seek solitude (Wolf 1985). Similarly, in a choice between a group of fishes and an empty compartment, individual black mollies, Poecilia latipinna, are more likely to associate with whites, than whites with blacks (McRobert & Bradner 1998). The study of Mathis & Chivers (2003) is particularly interesting. Yellow perch, Perca flavescens, and fathead minnows, Pimephales promelas, when not threatened by predators, prefer same-species groups, but when threatened, minnows still prefer to associate with conspecifics, whereas perch associate with heterospecifics. This not only demonstrates the greater willingness of one partner to establish a relationship, but also embodies the ‘conflict’ inherent in our results, where the mixed-species association is only established when absolutely necessary. Another observation that is consistent with our model is the fact that Amazonian saddleback tamarins, Saguinus fuscicollis, form mixed-species associations only when species differ in body length by 8%, i.e. they somehow ‘look’ sufficiently different (Heymann 1997). The model could also explain the phenomenon of school-oriented mimicry in juvenile red sea blennies, Meiacanthus nigrolineatus, where young of this species have evolved similar coloration to cardinal fish (Apogonidae spp.) and join shoals of this species in small numbers (Dafni & Diamant 1984). Finally, while not directly related to the tendency to establish and maintain a mixed-species relationship, Thompson's gazelles, Gazella thomsoni, are less vigilant than Grant's gazelles, Gazella granti, in mixed groups, suggesting that they may be less at risk from predation (Fitzgibbon 1990). This asymmetric vigilance in mixed-species groups is also seen in some primate mixed-species associations (Chapman & Chapman 1996).

While conflict between different-species members of a mixed-species group is a major prediction of our model, many other conflicting and mutual interests are predicted. Conflict may also be expected between members of the same cryptic species who benefit from the mixed-species association. Presumably, additional cryptic individuals joining the mixed group can benefit from predator protection, but at the expense of increased predator detection of the cryptic individuals already in the mixed group. Conspicuous and cryptic individuals in the mixed group, on the other hand, might benefit from recruiting more conspicuous members into the group to increase the proportion of conspicuous to cryptic individuals (to the mutual predator avoidance benefit of both). These are complex interactions whose outcome could be determined by game theory analysis. Empiricists interested in testing our model may also find behavioural analysis of the various conflicts and mutualisms predicted useful. Alternatively, empirical analysis could proceed by testing the prediction that small numbers of cryptic individuals in a group of conspicuous prey should be targeted less accurately than when the groups consists exclusively of cryptic prey. However, this could be extremely challenging as attentional mechanisms in predatory animals may have evolved to avoid prey that are challenging to target in groups. It also difficult, as a human, to objectively state that a prey item is cryptic to a particular predator's sensory system relative to other prey. One way round these issues is to use humans as predators, who can be told to target certain prey items (whose appearance can be carefully controlled) generated on a computer screen (Tosh et al. 2006).

Modelling a ubiquitous motif in predator sensory systems, we have shown here that mixed-species grouping can exploit a major weakness in this system, and that the major and novel prediction of ‘asymmetry’ in the benefits of mixing is commonly seen in nature. Further weight is added to our results by the fact that the potency of the basic modelling approach has been demonstrated in a previous study on a different phenomenon (the confusion effect; Tosh et al. 2006). We suggest that the mechanism presented here may be an important, generic force in the evolution of mixed-species associations. Based solely on experimental observations, it has often been assumed that a small number of odd-looking individuals in a group will automatically be selected by predators (Krause & Ruxton 2002). However, this ‘oddity effect’ appears to occur only when prey groups are small and are highly asymmetric in composition (Krause & Ruxton 2002). Our study indicates that the oddity effect may not be as widespread as textbooks might suggest; a conclusion that is in fact supported by close examination of the available empirical data (Krause & Ruxton 2002). Prey species appearance interacts in the predatory sensory system in a complex manner that could not be foreseen from the small number of simple experiments that have been used to probe the oddity effect. This study emphasizes the powerful way that mechanistic modelling of ecological phenomenon can illuminate important questions in ecological research, suggesting entirely novel mechanisms and identifying appropriate empirical tests of these putative mechanisms.

This work was funded by the UK Biotechnology and Biological Sciences Research Council (grant no. BBS/B/01790).