Many aposematic species live gregariously. Fisher (12) suggested that for unpalatability to evolve the prey species would gain from living in family groups. In addition to kin benefits, any individual in an aggregation might derive statistical dilution benefits from grouping (see refs. 8 and 13–15) and from faster avoidance learning by predators (8, 16–18). For these reasons aggregation accompanied with learning have been considered to provide favorable conditions for the evolution of conspicuous signals emerging as an antipredatory defense in unpalatable prey.

Predators can create frequency-dependent selection on prey species (19), which can be either apostatic, favoring rare morphs, or antiapostatic, favoring more common forms. Antiapostatic selection, where rare morphs are more vulnerable to predation, is thought to be an additional barrier for the evolution of conspicuousness in unpalatable prey (20). This might explain why aposematism is a relatively uncommon defensive strategy in rare species (3) and might be beneficial only when it reaches some threshold abundance (ref. 21 and see ref. 22). Strong antiapostatic selection on aposematic prey at low densities has probably led to a phenotypic plasticity in desert locusts (22), which rely on crypsis at low densities and develop warning colors only when they are more abundant. Additionally, it has been suggested that by aggregating the rare morphs would overcome the costs of antiapostatic selection (23), perhaps because of statistical dilution (21). Alternative mechanisms to overcome the disadvantage of conspicuous rare morphs include predators' aversion to novel prey (ref. 24 but see refs. 25 and 26). Predators might be reluctant to eat novel prey because they do not recognize rare conspicuous species as potential prey or avoid them on the basis of nonfamiliarity. Then even conspicuous morphs may not suffer from the increased visibility risks (24) or perhaps not even from antiapostatic selection. Thus, novelty effects may locally create possibilities for aposematism to evolve.

Although there are several studies on the frequency-dependent nature of predator foraging on palatable polymorphic prey (see refs. 27 and 28) and mimicry systems (29), there are only few studies on the effects of frequency-dependent selection on conspicuous aposematic prey (30, 31). Most studies of aposematism (4–7, 9, 24, 32) have unrealistically used an equal ratio of unpalatable and palatable prey (but see refs. 8, 30, and 31), because such a test is statistically most effective and requires the least amount of work. However, then the focus is more on the learning function of the predator, rather than on the risks a conspicuous but rare prey would suffer, which in the end is the key question.

Further, avoidance learning by predators has been considered to depend on a fixed number of aposematic prey eaten leading to a frequency-dependent disadvantage to rare aposematic prey (26, 33). In a previous experiment, a pair of aposematic prey types was presented to a bird predator in different frequencies; birds killed fewer individuals of the rarer of the two aposematic prey than of the common one. However, birds had a tendency to kill a relatively larger fraction of aposematic prey when that type was rare (31). This finding suggests that perhaps a learning function by predators is not a fixed function of the number of aposematic prey eaten.

We set out to test the effects of frequency on the selection pressures on evolving rare conspicuous prey. Wild birds were captured and brought to the laboratory and tested on 2 successive days for their ability to learn to avoid unpalatable aposematic prey, which they have never seen before in a 3 × 2 design: three frequencies (4%, 12%, 32%) and two patterns of dispersion (solitary vs. aggregated).

Birds and Aviary. This study was carried out in the Konnevesi Research Station (Central Finland). Wild great tits (Parus major) were mistnetted from Konnevesi and kept individually in illuminated cages (65 × 80 × 65 cm³) on a diet that contained sunflower seeds, peanuts, and fresh water ad libitum. Birds were habituated to eat sunflower seeds and peanuts from the floor in an experimental aviary (57.7 m² × 3.5 m height). Twelve handling perches were equally distributed into the aviary. Eight perches (height 45 cm) were attached to wooden planks (see later) and one to each of the four walls. Observations of the birds' behavior were made through a one-way mirror from an observation room, without disturbing the birds.

Training. Birds were considered ready for the experiment after they learned to open artificial prey items, which they have never seen before. These prey items were made by gluing a tiny slice (ca. 8 mg) of almond between two 10 × 10 mm pieces of white paper. We taught birds to open the artificial prey items in five stages. First, birds were offered plain almond crumbs and after they had eaten almonds they were offered a piece of an almond that was glued to a piece of white paper. Then an almond crumb was glued between two pieces of white paper, with the almond clearly visible, and finally an almond was put between two pieces of paper that formed a “shell” that the birds had to open to get access to the almond. When a bird had eaten five prey items at each stage, it was considered ready for the experiment. Training to open the prey items took on average 2.44 days (n = 84, SE = 0.13).

Birds also were habituated to forage in an artificial environment before the learning experiment was performed. Two novel landscapes were created on smaller aviary floors. The landscapes were comprised of seven rows and 10 columns, each column containing 10 of A3 (European standard, size 42 × 29.6 cm²) sheets of paper with about 70 crosses as cryptic symbols on them (see ref. 24). Papers were covered with self-adhesive book-covering film (Pelloplast, Pello, Finland). Each piece of A3 paper also had 10 fake cryptic prey types, a background symbol glued to paperboard and on top of the film. This was done to make the landscape three-dimensional. If fake prey items had not been added, birds would have learned to search for food that appears on top of the landscape, without paying attention to the symbols. Between each row there was a space along which birds could move and handle the prey. During the test trial, birds were allowed to eat 12 white and three cryptic paper prey items from the landscape. A test trial was done to ensure that the birds were able to learn to eat artificial prey items from the novel landscape. Birds had to complete this test trial before the experiment.

Learning Experiment. The landscape, consisting of 15 rows and 22 columns, thus 330 sheets of A3 (size 42 × 29.6 cm²) papers covering an area of 41 m², was introduced into a large aviary (57.7 m² size, height 3.5 m). Four sheets under each of eight perches located on the landscape and two sheets under each wall-attached perch were kept empty. This was done to ensure that the handled prey items were not mixed with the unconsumed prey while making the observations. Thus there were 290 possible sheets onto which prey items could be placed.

Aposematic prey were made unpalatable by soaking the almonds in a chloroquinine solution (2 g of chloroquinine dissolved in 30 ml of water) for 1 h and then dried. According to a previous experiment (24), this dosage makes the prey items highly unpalatable, and birds are able to learn to avoid this conspicuous signal. Aposematic and unpalatable prey was signaled with squares (), which made the prey items highly conspicuous against the background consisting of crosses (×). This symbol is novel for the birds because they have unlikely encountered this prey in the wild before. Cryptic palatable prey items were marked with the same crosses as the background. These signals were chosen because wild birds do not show any preferences toward either of the two symbols (24). There is also a very clear visibility difference between these two symbol types. Squares were six times as conspicuous as crosses to birds when presented at equal frequencies against ×-background (24).

To test how the frequency and dispersion type of aposematic prey affects its survival, birds were divided into six groups (each n = 14) according to the frequency treatment (4%, 12% and 32%) and dispersion-type treatment (solitary or aggregated group of four) before the experiment. When the percentage of aposematic prey was 4% there were eight aposematic prey items, when 12% there were 24, and when 32% 64 aposematic prey items presented to the birds. There were two dispersion treatments, solitary and aggregation. In the aggregation treatment aposematic prey was presented as groups of four, thus there were two, eight, and 16 groups, respectively, in the frequency treatments. Palatable cryptic prey was solitary in both aposematic dispersion-type treatments.

During the trials birds could choose 50 prey items from 200 prey items that were scattered randomly in eight blocks to ensure interspersion. The order of the killed prey type consumption was recorded. The prey was considered “killed” if the birds ate, took bites, or tasted the prey. If birds took the prey and dropped it immediately, it was considered to have survived the encounter. However, birds rarely dropped the prey items. To assess learning, the same treatment as in day 1 with 200 prey was repeated on the consecutive day.

Statistical Analysis. The initial relative predation of unpalatable prey was counted from the first five prey items eaten in the first trial. The risk was calculated as number of unpalatable prey killed divided by expected number of unpalatable prey eaten within the first five prey. Expected numbers of unpalatable prey eaten differed between the frequency treatments and were assumed to be determined only by the frequency. This was therefore used as an estimation of visibility risk, because it is likely that learning had not yet taken place. Because the data structure did not meet the assumptions of parametric statistics, the variable was ranked and sum of squares and mean squares were estimated as in parametric analysis of variance. The test value H (= SS_source/MS_total) follows asymptotically χ² distribution with df_source (34). Total mortality for unpalatable prey within the two trials was calculated as percentages (unpalatable eaten/unpalatable presented), and both trials were ranked according to the mortality values. Parametric repeated measure ANOVA was performed on the ranked data, and H statistics were calculated as above. Statistics on the total amount of aposematic prey eaten in the whole experiment were ln-transformed to meet the assumptions of parametric statistics. The statistical analyses were made by using the spss statistical package. All P values are two-tailed.

The relative initial risk (before learning took place) was calculated from the first five prey items birds took in the experiment. Because there were different numbers of prey types present, the initial risk was calculated as the number of unpalatable prey items eaten divided by the expected numbers of unpalatable prey eaten within the first five prey. Thus, we estimated the relative visibility risk for aposematic prey (Fig. 1), which reflects the initial risk of being conspicuous. There was no interaction between the dispersion type and frequency (H = 0.60, df = 2, P = 0.741). Frequency did not have a main effect (H = 0.19, df = 2, P = 0.911) but the dispersion type affected the initial risk (H = 8.15, df = 1, P = 0.004), indicating that aggregated prey benefited in all frequencies at the beginning of the experiment (Fig. 1). This finding also suggested that there was no additional cost for being conspicuous in aggregations. However, it indicates that aggregated prey benefited from dilution.

Figure 1 The mean (± SE) initial relative risk of predation on aposematic prey within the first five encountered prey items, in the first trial. □ represent solitary treatments and ● aggregated treatments. The line of unity indicates that (more ...)

Mortality of aposematic prey on 2 consecutive days was used as a dependent variable to assess the risk of predation. There was no three-way interaction between learning (day 1, day 2), frequency, and dispersion type (repeated measure H = 1.04, df = 2, P = 0.593 for ranked data). There was a two-way interaction between learning and dispersion type (repeated measure H = 11.25, df = 1, P = 0.001). This result indicates that in the first trial the mortality was lower for an aggregated than solitary treatment. There was also a two-way interaction between learning and frequency (repeated measure H = 9.94, df = 2, P = 0.007). This result indicates that in the lowest frequencies aposematic prey suffered the highest mortality per prey in the first trial. Because learning was significant (repeated measure H = 196.42, df = 1, P < 0.001) this finding indicates that birds did learn to avoid unpalatable prey types in all treatments. Risk of predation therefore depended on the dispersion type of aposematic prey as well as its frequency (Fig. 2).

Figure 2 The mean (+ SE) relative risk of predation on 2 consecutive days in three different frequencies (a) in solitary and (b) in aggregated treatment. The random line indicates where aposematic prey is eaten at the same frequency it was presented.

If we analyze the total number of aposematic prey items birds ate within the whole experiment there was no interaction between the frequency and dispersion type (ANOVA, F_2,78 = 0.34, P = 0.712 for ln transformed data). However, in higher frequencies birds ate more unpalatable prey in absolute numbers (ANOVA, F_2,78 = 81.40, P < 0.001) (Fig. 3). Multiple comparisons revealed that all frequency treatments differed from each other in the number of unpalatable prey eaten (Fig. 3). There was also a main effect of the dispersion type of aposematic prey (ANOVA, F_1,78 = 24.63, P < 0.001), suggesting that birds ate in total numbers more unpalatable prey when the prey items were solitary.

Figure 3 The mean (± SE) cumulative sum of aposematic prey eaten within the experiment according to the dispersion type, (a) solitary and (b) aggregated. Equal number of prey (altogether 200) was presented on both day 1 and day 2. Birds were allowed to (more ...)

This experiment emphasizes the difficulties faced by a rare conspicuous morph when entering a cryptic population. The mortality of aposematic prey was antiapostatic, indicating that predators ate a higher fraction from less frequent morphs than from the common morphs (Fig. 2). Therefore, when considering the selective pressures on the different frequencies, the prey survived best at the highest frequency. There was also quite substantive visibility cost for a rare aposematic prey, which is expected by the theory (1–3, 11, 12, 20, 21) but not evident compared with previous experiments (4–7, 9, 24). This might be explained by differences in the experimental setups and questions. Previous experiments have concentrated on testing a predator's ability to learn to avoid conspicuous prey in an equal ratio between cryptic palatable and aposematic unpalatable prey (4–7, 9, 24), and perhaps therefore the initial costs for aposematic prey have not been as severe as in the present experiment.

Although mortality was highest in low frequencies, aggregated prey suffered less from the disadvantages of rarity. Additionally aggregations seemed to be more beneficial the lower the frequency of prey (Fig. 2). Solitary aposematic prey suffered an initial risk, which was significantly higher than expected (Fig. 1), but aggregated individuals benefited from aggregations from the beginning. Therefore aggregating would have provided more favorable conditions for aposematism to evolve as an antipredatory strategy. The benefits from aggregation are that a predator's avoidance learning is faster (8, 16–18), which was present also in this study. In addition to learning benefits, aggregated individuals might have gained from a dilution effect (35) or a predator's reluctance to eat novel aggregated prey (17).

It was fairly unexpected that predators' avoidance learning did not depend on a fixed number of unpalatable prey being eaten (Fig. 3). In low solitary frequencies birds ate only on average 4.07 (SE = 0.40), whereas for individuals in the highest frequency the same number was 14.86 (SE = 0.97) on the first day. All birds learned to avoid unpalatable prey, but in lower frequencies this resulted from far fewer aposematic individuals. This finding suggests that not only the negative reinforcement but also search image on cryptic prey seem to affect the avoidance learning of birds. A similar mechanism seemed to explain the results where two different aposematic prey items in variable frequencies were compared (31).

The fact that different numbers of aposematic individuals eaten resulted in an avoidance of conspicuous aposematic prey is surprising, because it is assumed to be the basis for Müllerian mimicry. The basic assumptions for the benefits for Müllerian mimicry have been derived from the hypothesis that predators eat an N amount of unpalatable prey and if this number is divided between two similar species at varying frequencies, their benefits can be calculated respectively (26, 33). Thus models of Müllerian mimicry primarily have concentrated on the model mimic ratio and have not treated the unpalatable cryptic prey ratio as an important factor. Our experiment suggests that this number-dependent learning might be a simplification, and we also should be careful when we incorporate frequencies of aposematic prey in reference to cryptic prey to our models of Müllerian mimicry.

The visibility risks from conspicuousness as well as the antiapostatic disadvantage sets barriers for aposematism to evolve. A predator's reluctance to eat novel prey could have balanced the costs from increased visibility (24). However, in this experiment, rare novel aposematic prey did not gain from the predator's reluctance to eat rare or novel prey. Our understanding of predator education is based on studies where equal ratios of cryptic and aposematic prey were presented to the predators, which is perhaps an unrealistic situation for the evolution of aposematism as a defensive strategy. Perhaps this study opens avenues for theoretical work, which usually has taken predators' learning as a fixed function of number of aposematic prey eaten (e.g., ref. 33).

We are grateful to H. Nisu and T. Puranen for the maintenance of the birds and assistance with the experiments. The Konnevesi Research Station is acknowledged for the help with practical matters. J. Kilpimaa helped us to catch birds while R.V.A. had pneumonia. Ryan Calsbeek kindly checked the language. Central Finland Regional Environment Centre gave us permission to mist-net and keep the birds (LS-44/99, Dnro 0998L0540/253), and the Ethical Committee of University of Jyväskylä gave us permission to do the experiment (25/23.11.1998). The Academy of Finland financed this study.