Avoiding predation is essential for most animals. For group-living species, effective predator avoidance relies on making fast and accurate collective decisions. However, the mechanisms underlying the ability to make adaptive collective decisions and to coordinate movements under predation threat remains unclear. Here, we used guppies artificially selected for divergence in the size of the telencephalon, the main brain region for advanced decision-making in vertebrates, to test the influence of telencephalon size on collective decision-making under predation threat. We measured the latency and accuracy of collective decision-making to avoid a model predator in guppy shoals. In addition, we used high-resolution tracking analysis to assess shoaling dynamics under predator threat between the telencephalon size selection lines. We found that collective decision-making latency was shorter in large telencephalon guppy shoals, indicating that variation in telencephalon size can cause variation in the ability to avoid predation. This result is unlikely to be driven by differences in boldness, as several standard tests suggest that there is no difference in boldness between the telencephalon size selection lines. General aspects of shoaling dynamics did not differ between the telencephalon size selected lines. Our study highlights that rapid mosaic changes in brain region size may be an important mechanism behind social behavioural variation with strong fitness implications.

Animals can employ various spatial learning strategies to navigate through their environment, and the proficiency in specific strategies varies greatly both intraspecifically and interspecifically. Currently, the neural basis of this variation is poorly understood. Here, we tested whether variation in performance in egocentric and allocentric spatial learning strategies is related to differential investment in distinct brain regions. To do so, we used guppies (Poecilia reticulata) from artificial selection lines expressing differences in relative telencephalon size, and tested their ability to learn a spatial task, based on either egocentric (left–right) or allocentric (environmental) cues. Surprisingly, fish with larger telencephalons showed enhanced performance in both tasks, regardless of cue type, suggesting a more complicated role of the fish telencephalon in spatial learning than previously thought. Our study provides the first direct evidence that evolutionary changes in relative telencephalon size lead to corresponding shifts in spatial cognition at the within-species level. Furthermore, our results offer critical and novel insights regarding the function of the telencephalon and its role in the evolution of spatial cognition.

Introduction: Variation in eye size is sometimes closely associated with brain morphology. Visual information, detected by the retina, is transferred to the optic tectum to coordinate eye and body movements towards stimuli and thereafter distributed into other brain regions for further processing. The telencephalon is an important visual processing region in many vertebrate species and a highly developed region in visually dependent species. Yet, the existence of a coevolutionary relationship between telencephalon size and eye size remains relatively unknown. Methods: Here, we use male and female guppies artificially selected for small- and large-relative-telencephalon-size to test if artificial selection on telencephalon size results in changes in eye size. In addition, we performed an optomotor test as a proxy for visual acuity. Results: We found no evidence that eye size changes with artificial selection on telencephalon size. Eye size was similar in both absolute and relative terms between the two selection regimes but was larger in females. This is most likely because of the larger body size in females, but it could also reflect their greater need for visual capacity due to sex-specific differences in foraging and mating behaviour. Although the optomotor response was stronger in guppies with a larger telencephalon, we found no evidence for differences in visual acuity between the selection regimes. Conclusion: Our study suggests that eye size and visual perception in guppies do not change rapidly with strong artificial selection on telencephalon size.

Collective motion is common across all animal taxa, from swarming insects to schools of fish. The collective motion requires intricate behavioral integration among individuals, yet little is known about how evolutionary changes in brain morphology influence the ability for individuals to coordinate behavior in groups. In this study, we utilized guppies that were selectively bred for relative telencephalon size, an aspect of brain morphology that is normally associated with advanced cognitive functions, to examine its role in collective motion using an open-field assay. We analyzed high-resolution tracking data of same-sex shoals consisting of 8 individuals to assess different aspects of collective motion, such as alignment, attraction to nearby shoal members, and swimming speed. Our findings indicate that variation in collective motion in guppy shoals might not be strongly affected by variation in relative telencephalon size. Our study suggests that group dynamics in collectively moving animals are likely not driven by advanced cognitive functions but rather by fundamental cognitive processes stemming from relatively simple rules among neighboring individuals.

Executive functions are a set of cognitive control processes required for optimizing goal-directed behavior. Despite more than two centuries of research on executive functions, mostly in humans and nonhuman primates, there is still a knowledge gap in what constitutes the mechanistic basis of evolutionary variation in executive function abilities. Here, we show experimentally that size changes in a forebrain structure (i.e. telencephalon) underlie individual variation in executive function capacities in a fish. For this, we used male guppies (Poecilia reticulata) issued from artificial selection lines with substantial differences in telencephalon size relative to the rest of the brain. We tested fish from the up- and down-selected lines not only in three tasks for the main core executive functions: cognitive flexibility, inhibitory control, and working memory, but also in a basic conditioning test that does not require executive functions. Individuals with relatively larger telencephalons outperformed individuals with smaller telencephalons in all three executive function assays but not in the conditioning assay. Based on our findings, we propose that the telencephalon is the executive brain in teleost fish. Together, it suggests that selective enlargement of key brain structures with distinct functions, like the fish telencephalon, is a potent evolutionary pathway toward evolutionary enhancement of advanced cognitive abilities in vertebrates. 

Over recent decades, substantial research has focused on fish cognitive evolution to increase our understanding of the evolution of the enormous diversity of cognitive abilities that exists in fishes. One important but understudied aspect of cognitive evolution is sexual dimorphism in cognitive abilities. Sex-specific variation in brain region morphology has been proposed to be an important mechanism in this context. However, it is also common to find sex-specific variation in behavior and cognition without associated differences in brain morphology among the sexes. The telencephalon is the major cognitive center in the vertebrate brain and variation in telencephalon size has been associated with variation in cognition. Here, we utilize recently developed guppy artificial selection lines with ca. 10% differences in relative telencephalon size to investigate whether similar responses to selection of the size of this region may affect cognitive abilities differently in males and females. To that end, we compared two ecologically relevant aspects of cognition, detour learning and binary spatial discrimination. We tested the significance of the interaction between telencephalon size and sex, and we found no sex-specific effects of evolutionary increases in telencephalon size in the cognitive abilities tested. This study indicates that no clear cognitive sex-specific effects occur in response to rapid selection of telencephalon size. We suggest that future research on sexual dimorphism in cognitive abilities in fish could use various cognitive tests and examine telencephalic sub-regions to gain a more comprehensive understanding of their evolution.

Mosaic brain evolution, the change in the size of separate brain regions in response to selection on cognitive performance, is an important idea in the field of cognitive evolution. However, untill now, most of the data on how separate brain regions respond to selection and their cognitive consequences stem from comparative studies. To experimentally investigate the influence of mosaic brain evolution on cognitive ability, we used male guppies artificially selected for large and small telencephalons relative to the rest of the brain. Here, we tested an important aspect of executive cognitive ability using a detour task. We found that males with larger telencephalons outperformed males with smaller telencephalons. Fish with larger telencephalons showed faster improvement in performance during detour training and were more successful in reaching the food reward without touching the transparent barrier (i.e., through correct detouring) during the test phase. Together, our findings provide the first experimental evidence showing that evolutionary enlargement of relative telencephalon size confers cognitive benefits, supporting an important role for mosaic brain evolution during cognitive evolution.

Determining how variation in brain morphology affects cognitive abilities is important to understand inter-individual variation in cognition and, ultimately, cognitive evolution. Yet, despite many decades of research in this area, there is surprisingly little experimental data available from assays that quantify cognitive abilities and brain morphology in the same individuals. Here, we tested female guppies (Poecilia reticulata) in two tasks, colour discrimination and reversal learning, to evaluate their learning abilities and cognitive flexibility. We then estimated the size of five brain regions (telencephalon, optic tectum, hypothalamus, cerebellum and dorsal medulla), in addition to relative brain size. We found that optic tectum relative size, in relation to the rest of the brain, correlated positively with discrimination learning performance, while relative telencephalon size correlated positively with reversal learning performance. The other brain measures were not associated with performance in either task. By evaluating how fast learning occurs and how fast an animal adjusts its learning rules to changing conditions, we find support for that different brain regions have distinct functional correlations at the individual level. Importantly, telencephalon size emerges as an important neural correlate of higher executive functions such as cognitive flexibility. This is rare evidence supporting the theory that more neural tissue in key brain regions confers cognitive benefits. 

The mosaic brain evolution hypothesis, stating that brain regions can evolve relatively independently during cognitive evolution, is an important idea to understand how brains evolve with potential implications even for human brain evolution. Here, we provide the first experimental evidence for this hypothesis through an artificial selection experiment in the guppy (Poecilia reticulata). After four generations of selection on relative telencephalon volume (relative to brain size), we found substantial changes in telencephalon size but no changes in other regions. Further comparisons revealed that up-selected lines had larger telencephalon, while down-selected lines had smaller telencephalon than wild Trinidadian populations. Our results support that independent evolutionary changes in specific brain regions through mosaic brain evolution can be important facilitators of cognitive evolution.

Understanding how brains have evolved and subsequently culminated in the huge variation in brain morphology among contemporary vertebrate species has fascinated researchers for many decades. It has been recognized that brain morphology is both genetically and environmentally determined. Adaptations to ecological challenges, for one, has been proposed to be a major force in brain diversification processes. Considering the large energetic costs of neural tissue, it is believed that brain evolution is a highly complex process, involving a delicate balance between the corresponding costs and benefits. 

Using the guppy (Poecilia reticulata) as the model organism, I first examined the conditions under which diversity in brain morphology is generated. This was done by investigating factors known to exert an influence on brain plasticity, namely environmental and cognitive effects (Paper I). Existing studies generally indicate that the provision of environmental enrichment lead to the enlargement of specific brain structures. While plastic alterations in brain morphology was found to respond to environmental complexity in my study, successful performance in two cognitive tasks did not produce any significant changes. 

I next assessed the feasibility of the mosaic brain evolution hypothesis by artificially selecting for an increase and decrease in the relative size of the telencephalon (Paper II). Telencephalon size was shown to respond rapidly to divergent selection pressures, with no substantial changes in any of the other brain regions. A comparison with wild fish revealed that fish from the unselected control treatment had telencephalon sizes most similar to that of wild populations, whereas both up-selected and down-selected fish had considerably larger and smaller telencephalon, respectively. 

I tested fish from the artificial selection lines in a test battery to determine if known differences in telencephalon size affects boldness (Paper III). Individuals were subjected to an emergence test, an open field test and a novel object test. I found no differences in boldness levels across selection treatments, but distinct sex differences were noted whereby males were more active and bolder. 

The cognitive benefits associated with a larger telencephalon were examined in males in a test of self-control (Paper IV). Guppies from the up-selected lines attained a steeper learning curve and made more correct detours compared to their down-selected conspecifics. 

In conclusion, I provide experimental evidence for the mosaic brain evolution hypothesis by showing that a specific brain region (telencephalon) can evolve rapidly and independently under directed selection. Future tests on other cognitive benefits as well as implicated costs, together with underlying neuronal changes would help to further unravel the factors governing brain evolution.