One of the most spectacular displays of social behavior is the synchronized movements that many animal groups perform to travel, forage and escape from predators. However, elucidating the neural mechanisms underlying the evolution of collective behaviors, as well as their fitness effects, remains challenging. Here, we study collective motion patterns with and without predation threat and predator inspection behavior in guppies experimentally selected for divergence in polarization, an important ecological driver of coordinated movement in fish. We find that groups from artificially selected lines remain more polarized than control groups in the presence of a threat. Neuroanatomical measurements of polarization-selected individuals indicate changes in brain regions previously suggested to be important regulators of perception, fear and attention, and motor response. Additional visual acuity and temporal resolution tests performed in polarization-selected and control individuals indicate that observed differences in predator inspection and schooling behavior should not be attributable to changes in visual perception, but rather are more likely the result of the more efficient relay of sensory input in the brain of polarization-selected fish. Our findings highlight that brain morphology may play a fundamental role in the evolution of coordinated movement and anti-predator behavior.

Understanding the basis of behavior requires dissecting the complex waves of gene expression that underlie how the brain processes stimuli and produces an appropriate response. In order to determine the dynamic nature of the neurogenomic network underlying mate choice, we use transcriptome sequencing to capture the female neurogenomic response in two brain regions involved in sensory processing and decision-making under different mating and social contexts. We use differential coexpression (DC) analysis to evaluate how gene networks in the brain are rewired when a female evaluates attractive and nonattractive males, greatly extending current single-gene approaches to assess changes in the broader gene regulatory network. We find the brain experiences a remarkable amount of network rewiring in the different mating and social contexts we tested. Further analysis indicates the network differences across contexts are associated with behaviorally relevant functions and pathways, particularly learning, memory and other cognitive functions. Finally, we identify the loci that display social context-dependent connections, revealing the basis of how relevant neurological and metabolic pathways are differentially recruited in distinct social contexts. More broadly, our findings contribute to our understanding of the genetics of mating and social behavior by identifying gene drivers behind behavioral neural processes, illustrating the utility of DC analysis in neurosciences and behavior.

The mosaic brain evolution hypothesis, stating that brain regions can evolve relatively independently during cognitive evolution, is an important idea to understand how brains evolve with potential implications even for human brain evolution. Here, we provide the first experimental evidence for this hypothesis through an artificial selection experiment in the guppy (Poecilia reticulata). After four generations of selection on relative telencephalon volume (relative to brain size), we found substantial changes in telencephalon size but no changes in other regions. Further comparisons revealed that up-selected lines had larger telencephalon, while down-selected lines had smaller telencephalon than wild Trinidadian populations. Our results support that independent evolutionary changes in specific brain regions through mosaic brain evolution can be important facilitators of cognitive evolution.

Cognitive ageing is the general process when certain mental skills gradually deteriorate with age. Across species, there is a pattern of a slower brain structure degradation rate in large-brained species. Hence, having a larger brain might buffer the impact of cognitive ageing and positively affect survival at older age. However, few studies have investigated the link between relative brain size and cognitive ageing at the intraspecific level. In particular, experimental data on how brain size affects brain function also into higher age is largely missing. We used 288 female guppies (Poecilia reticulata), artificially selected for large and small relative brain size, to investigate variation in colour discrimination and behavioural flexibility, at 4-6, 12 and 24 months of age. These ages are particularly interesting since they cover the life span from sexual maturation until maximal life length under natural conditions. We found no evidence for a slower cognitive ageing rate in large-brained females in neither initial colour discrimination nor reversal learning. Behavioural flexibility was predicted by large relative brain size in the youngest group, but the effect of brain size disappeared with increasing age. This result suggests that cognitive ageing rate is faster in large-brained female guppies, potentially due to the faster ageing and shorter lifespan in the large-brained selection lines. It also means that cognition levels align across different brain sizes with older age. We conclude that there are cognitive consequences of ageing that vary with relative brain size in advanced learning abilities, whereas fundamental aspects of learning can be maintained throughout the ecologically relevant life span.

The evolution of collective behaviour has been proposed to have important effects on individual cognitive abilities. Yet, in what way they are related remains enigmatic. In this context, the ‘distributed cognition’ hypothesis suggests that reliance on other group members relaxes selection for individual cognitive abilities. Here, we tested how cognitive processes respond to evolutionary changes in collective motion using replicate lines of guppies (Poecilia reticulata) artificially selected for the degree of schooling behaviour (group polarization) with >15% difference in schooling propensity. We assessed associative learning in females of these selection lines in a series of cognitive assays: colour associative learning, reversal learning, social associative learning, and individual and collective spatial associative learning. We found that control females were faster than polarization-selected females at fulfilling a learning criterion only in the colour associative learning assay, but they were also less likely to reach a learning criterion in the individual spatial associative learning assay. Hence, although testing several cognitive domains, we found weak support for the distributed cognition hypothesis. We propose that any cognitive implications of selection for collective behaviour lie outside of the cognitive abilities included in food-motivated associative learning for visual and spatial cues.

Despite ongoing advances in sexual selection theory, the evolution of mating decisions remains enigmatic. Cognitive processes often require simultaneous processing of multiple sources of information from environmental and social cues. However, little experimental data exist on how cognitive ability affects such fitness-associated aspects of behaviour. Using advanced tracking techniques, we studied mating behaviours of guppies artificially selected for divergence in relative brain size, with known differences in cognitive ability, when predation threat and sex ratio was varied. In females, we found a general increase in copulation behaviour in when the sex ratio was female biased, but only large-brained females responded with greater willingness to copulate under a low predation threat. In males, we found that small-brained individuals courted more intensively and displayed more aggressive behaviours than large-brained individuals. However, there were no differences in female response to males with different brain size. These results provide further evidence of a role for female brain size in optimal decision-making in a mating context. In addition, our results indicate that brain size may affect mating display skill in male guppies. We suggest that it is important to consider the association between brain size, cognitive ability and sexual behaviour when studying how morphological and behavioural traits evolve in wild populations.

Reversal learning assays are commonly used across a wide range of taxa to investigate associative learning and behavioural flexibility. In serial reversal learning, the reward contingency in a binary discrimination is reversed multiple times. Performance during serial reversal learning varies greatly at the interspecific level, as some animals adopt a rule-based strategy that enables them to switch quickly between reward contingencies. A larger relative brain size, generating enhanced learning ability and increased behavioural flexibility, has been proposed to be an important factor underlying this variation. Here, we experimentally tested this hypothesis at the intraspecific level. We used guppies (Poecilia reticulata) artificially selected for small and large relative brain size, with matching differences in neuron number, in a serial reversal learning assay. We tested 96 individuals over 10 serial reversals and found that learning performance and memory were predicted by brain size, whereas differences in efficient learning strategies were not. We conclude that variation in brain size and neuron number is important for variation in learning performance and memory, but these differences are not great enough to cause the larger differences in efficient learning strategies observed at higher taxonomic levels.

Despite the common assumption that the brain is malleable to surrounding conditions mainly during ontogeny, plastic neural changes can occur also in adulthood. One of the driving forces responsible for alterations in brain morphology is increasing environmental complexity that may demand enhanced cognitive abilities (e.g. attention, memory and learning). However, studies looking at the relationship between brain morphology and learning are scarce. Here, we tested the effects of both learning and environmental enrichment on neural plasticity in guppies (Poecilia reticulata), by means of either a reversal-learning test or a spatial-learning test. Given considerable evidence supporting environmentally induced plastic alterations, two separate control groups that were not subjected to any cognitive test were included to account for potential changes induced by the experimental setup alone. We did not find any effect of learning on any of our brain measurements. However, we found strong evidence for an environmental effect, where fish given access to the spatial-learning environment had larger relative brain size and optic tectum size in relation to those exposed to the reversal-learning environment. Our results demonstrate the plasticity of the adult brain to respond adaptively mainly to environmental conditions, providing support for the environmental enhancement theory.

Snapshot analyses have demonstrated dramatic intraspecific variation in the degree of brain sexual size dimorphism (SSD). Although brain SSDis believed to be generated by the sex-specific cognitive demands of reproduction, the relative roles of developmental and population-specific contributions to variation in brain SSD remain little studied. Using a common garden experiment, we tested for sex-specific changes in brain anatomy over the breeding cycle in three-spined stickleback (Gasterosteus aculeatus) sampled from four locations in northern Europe. We found that the male brain increased in size (ca. 24%) significantly more than the female brain towards breeding, and that the resulting brain SSD was similar (ca. 20%) for all populations over the breeding cycle. Our findings support the notion that the stickleback brain is highly plastic and changes over the breeding cycle, especially in males, likely as an adaptive response to the cognitive demands of reproduction (e.g. nest construction and parental care). The results also provide evidence to suggest that breeding-related changes in brain size may be the reason for the widely varying estimates of brain SSD across studies of this species, cautioning against interpreting brain size measurements from a single time point as fixed/static.

Animals living in groups can show substantial variation in social traits and this affects their social organization. However, as the specific mechanisms driving this organization are difficult to identify in already organized groups typically found in the wild, the contribution of interindividual variation to group level behaviour remains enigmatic. Here, we present results of an experiment to create and compare groups that vary in social organization, and study how individual behaviour varies between these groups. We iteratively sorted individuals between groups of guppies, Poecilia reticulata, by ranking the groups according to their directional alignment and then mixing similar groups. Over the rounds of sorting the consistency of the group rankings increased, producing groups that varied significantly in key social behaviours such as collective activity and group cohesion. The repeatability of the underlying individual behaviour was then estimated by comparing the experimental data to simulations. At the level of basic locomotion, individuals in more coordinated groups displayed stronger interactions with the centre of the group, and weaker interactions with their nearest neighbours. We propose that this provides the basis for a passive phenotypic assortment mechanism that may explain the structures of social networks in the wild.