The organization and coordination of fish schools provide a valuable model to investigate the genetic architecture of affiliative behaviours and dissect the mechanisms underlying social behaviours and personalities. Here we used replicate guppy selection lines that vary in schooling propensity and combine quantitative genetics with genomic and transcriptomic analyses to investigate the genetic basis of sociability phenotypes. We show that consistent with findings in collective motion patterns, experimental evolution of schooling propensity increased the sociability of female, but not male, guppies when swimming with unfamiliar conspecifics. This finding highlights a relevant link between coordinated motion and sociability for species forming fission–fusion societies in which both group size and the type of social interactions are dynamic across space and time. We further show that alignment and attraction, the two major traits forming the sociability personality axis in this species, showed heritability estimates at the upper end of the range previously described for social behaviours, with important variation across sexes. The results from both Pool-seq and RNA-seq data indicated that genes involved in neuron migration and synaptic function were instrumental in the evolution of sociability, highlighting a crucial role of glutamatergic synaptic function and calcium-dependent signalling processes in the evolution of schooling.

One of the most spectacular displays of social behavior is the synchronized movements that many animal groups perform to travel, forage and escape from predators. However, elucidating the neural mechanisms underlying the evolution of collective behaviors, as well as their fitness effects, remains challenging. Here, we study collective motion patterns with and without predation threat and predator inspection behavior in guppies experimentally selected for divergence in polarization, an important ecological driver of coordinated movement in fish. We find that groups from artificially selected lines remain more polarized than control groups in the presence of a threat. Neuroanatomical measurements of polarization-selected individuals indicate changes in brain regions previously suggested to be important regulators of perception, fear and attention, and motor response. Additional visual acuity and temporal resolution tests performed in polarization-selected and control individuals indicate that observed differences in predator inspection and schooling behavior should not be attributable to changes in visual perception, but rather are more likely the result of the more efficient relay of sensory input in the brain of polarization-selected fish. Our findings highlight that brain morphology may play a fundamental role in the evolution of coordinated movement and anti-predator behavior.

Understanding the basis of behavior requires dissecting the complex waves of gene expression that underlie how the brain processes stimuli and produces an appropriate response. In order to determine the dynamic nature of the neurogenomic network underlying mate choice, we use transcriptome sequencing to capture the female neurogenomic response in two brain regions involved in sensory processing and decision-making under different mating and social contexts. We use differential coexpression (DC) analysis to evaluate how gene networks in the brain are rewired when a female evaluates attractive and nonattractive males, greatly extending current single-gene approaches to assess changes in the broader gene regulatory network. We find the brain experiences a remarkable amount of network rewiring in the different mating and social contexts we tested. Further analysis indicates the network differences across contexts are associated with behaviorally relevant functions and pathways, particularly learning, memory and other cognitive functions. Finally, we identify the loci that display social context-dependent connections, revealing the basis of how relevant neurological and metabolic pathways are differentially recruited in distinct social contexts. More broadly, our findings contribute to our understanding of the genetics of mating and social behavior by identifying gene drivers behind behavioral neural processes, illustrating the utility of DC analysis in neurosciences and behavior.

Despite ongoing advances in sexual selection theory, the evolution of mating decisions remains enigmatic. Cognitive processes often require simultaneous processing of multiple sources of information from environmental and social cues. However, little experimental data exist on how cognitive ability affects such fitness-associated aspects of behaviour. Using advanced tracking techniques, we studied mating behaviours of guppies artificially selected for divergence in relative brain size, with known differences in cognitive ability, when predation threat and sex ratio was varied. In females, we found a general increase in copulation behaviour in when the sex ratio was female biased, but only large-brained females responded with greater willingness to copulate under a low predation threat. In males, we found that small-brained individuals courted more intensively and displayed more aggressive behaviours than large-brained individuals. However, there were no differences in female response to males with different brain size. These results provide further evidence of a role for female brain size in optimal decision-making in a mating context. In addition, our results indicate that brain size may affect mating display skill in male guppies. We suggest that it is important to consider the association between brain size, cognitive ability and sexual behaviour when studying how morphological and behavioural traits evolve in wild populations.

The relationship between brain size and ageing is a paradox. The cognitive benefits of large brains should protect from extrinsic mortality and thus indirectly select for slower ageing. However, the substantial energetic cost of neural tissue may also impact the energetic budget of large-brained organisms, causing less investment in somatic maintenance and thereby faster ageing. While the positive association between brain size and survival in the wild is well established, no studies exist on the direct effects of brain size on ageing. Here we test how brain size influences intrinsic ageing in guppy (Poecilia reticulata) brain size selection lines with 12% difference in relative brain size. Measuring survival under benign conditions, we find that large-brained animals live 22% shorter than small-brained animals and the effect is similar in both males and females. Our results suggest a trade-off between investment into brain size and somatic maintenance. This implies that the link between brain size and ageing is contingent on the mechanism of mortality, and selection for positive correlations between brain size and ageing should occur mainly under cognition-driven survival benefits from increased brain size. We show that accelerated ageing can be a cost of evolving a larger brain.

Understanding the evolution of mate choice requires dissecting the mechanisms of female preference, particularly how these differ among social contexts and preference phenotypes. Here, we studied the female neurogenomic response after only 10 min of mate exposure in both a sensory component (optic tectum) and a decision-making component (telencephalon) of the brain. By comparing the transcriptional response between females with and without preferences for colourful males, we identified unique neurogenomic elements associated with the female preference phenotype that are not present in females without preference. A network analysis revealed different properties for this response at the sensory-processing and the decision-making levels, and we show that this response is highly centralized in the telencephalon. Furthermore, we identified an additional set of genes that vary in expression across social contexts, beyond mate evaluation. We show that transcription factors among these loci are predicted to regulate the transcriptional response of the genes we found to be associated with female preference.

Understanding what drives animal decisions is fundamental in evolutionary biology, and mate choice decisions are arguably some of the most important in any individual's life. As cognitive ability can impact decision making, elucidating the link between mate choice and cognitive ability is necessary to fully understand mate choice. To experimentally study this link, we used guppies (Poecilia reticulata) artificially selected for divergence in relative brain size and with previously demonstrated differences in cognitive ability. A previous test in our female guppy selection lines demonstrated the impact of brain size and cognitive ability on information processing during female mate choice decisions. Here, we evaluated the effect of brain size and cognitive ability on male mate choice decisions. Specifically, we investigated the preference of large-brained, small-brained and non-selected guppy males for female body size, a key indicator of female fecundity in this species. For this, male preference was quantified in dichotomous choice tests when presented with dyads of females with small, medium and large body size differences. All types of males showed a preference for larger females but no effect of brain size was found in the ability to discriminate between differently sized females. However, we found that non-selected and large-brained males, but not small-brained males, showed a context-dependent preference for larger females depending on the difference in female size. Our results have two important implications. First, they provide further evidence that male mate choice also occurs in a species in which secondary sexual omamentation is present only in males. Second, they show that brain size and cognitive ability have important effects on individual variation in mating preference and sexually selected traits.

Mate choice decisions are central in sexual selection theory aimed to understand how sexual traits evolve and their role in evolutionary diversification. We test the hypothesis that brain size and cognitive ability are important for accurate assessment of partner quality and that variation in brain size and cognitive ability underlies variation in mate choice. We compared sexual preference in guppy female lines selected for divergence in relative brain size, which we have previously shown to have substantial differences in cognitive ability. In a dichotomous choice test, large-brained and wild-type females showed strong preference for males with color traits that predict attractiveness in this species. In contrast, small-brained females showed no preference for males with these traits. In-depth analysis of optomotor response to color cues and gene expression of key opsins in the eye revealed that the observed differences were not due to differences in visual perception of color, indicating that differences in the ability to process indicators of attractiveness are responsible. We thus provide the first experimental support that individual variation in brain size affects mate choice decisions and conclude that differences in cognitive ability may be an important underlying mechanism behind variation in female mate choice.

The visual system is highly variable across species, and such variability is a key factor influencing animal behavior. Variation in the visual system, for instance, can influence the outcome of learning tasks when visual stimuli are used. We illustrate this issue in guppies (Poecilia reticulata) artificially selected for large and small relative brain size with pronounced behavioral differences in learning experiments and mate choice tests. We performed a study of the visual system by quantifying eye size and optomotor response of large-brained and small-brained guppies. This represents the first experimental test of the link between brain size evolution and visual acuity. We found that female guppies have larger eyes than male guppies, both in absolute terms and in relation to their body size. Likewise, individuals selected for larger brains had slightly larger eyes but not better visual acuity than small-brained guppies. However, body size was positively associated with visual acuity. We discuss our findings in relation to previous macroevolutionary studies on the evolution of brain morphology, eye morphology, visual acuity, and ecological variables, while stressing the importance of accounting for sensory abilities in behavioral studies.

Competition over access for mates has led to the evolution of many striking examples of morphological traits and behaviour in animals. The rapid development of the sexual selection field in recent decades have dramatically advanced our understanding of what traits make individuals more successful in attracting mates and how preferences for mates evolve over time. However, till now, research in this field has put less emphasis on the mechanisms that underlie variation in mate choice and sexual behaviour. Cognitive processes could potentially be key drivers of individual variation in mating preferences and sexual behaviours and therefore deserve further investigation. In this thesis, I used guppies artificially selected for relative brain size as the model system to study the association between brain size, cognitive ability and various aspects of mate choice. Previous studies in this model system showed that large-brained individuals of both sexes outperformed small-brained individuals in cognitive tests. Here I quantified their sexual behaviours and mating preferences to provide novel empirical data concerning the association between brain size, cognitive ability and sexual selection. In dichotomous choice preference tests based on visual cues, comparisons between large-brained and small-brained guppies showed important differences in their assessment of mate quality. These results are not driven by pre-existing visual biases caused by the artificial selection since further investigation of the visual capacity of these fish detected no differences between large-brained and small-brained individuals in their sensitivity to colour or in their capacity to resolve spatial detail. I also quantified sexual behaviour in male guppies artificially selected for relative brain size and found no difference in the behaviours of large-brained and small-brained males in a single male-single female non-competitive scenario. On the contrary, in a more complex social setting I found a reduction in large-brained males in the rate of courtship towards females and dominance displays towards other males when exposed to different degrees of predation threat and different numbers of male competitors. However, this reduction in behavioural intensity did not result in a lower access to copulation with females for large-brained males. I likewise evaluated female sexual behaviour and found that large-brained females had higher behavioural flexibility such that they decreased their receptiveness towards males more strongly under higher levels of predation threat. Together, these results provide novel empirical evidence that brain size and cognitive ability are tightly linked to mating preferences and sexual behaviours. These findings suggest that brain size and cognitive ability might be important mechanisms behind variation in mating preferences and in sexually selected traits across and within species.