The mosaic brain evolution hypothesis, stating that brain regions can evolve relatively independently during cognitive evolution, is an important idea to understand how brains evolve with potential implications even for human brain evolution. Here, we provide the first experimental evidence for this hypothesis through an artificial selection experiment in the guppy (Poecilia reticulata). After four generations of selection on relative telencephalon volume (relative to brain size), we found substantial changes in telencephalon size but no changes in other regions. Further comparisons revealed that up-selected lines had larger telencephalon, while down-selected lines had smaller telencephalon than wild Trinidadian populations. Our results support that independent evolutionary changes in specific brain regions through mosaic brain evolution can be important facilitators of cognitive evolution.

Domesticated animals display suites of altered morphological, behavioral, and physiological traits compared to their wild ancestors, a phenomenon known as the domestication syndrome (DS). Because these alterations are observed to co-occur across a wide range of present day domesticates, the traits within the DS are assumed to covary within species and a single developmental mechanism has been hypothesized to cause the observed co-occurrence. However, due to the lack of formal testing it is currently not well-resolved if the traits within DS actually covary. Here, we test the hypothesis that the presence of the classic morphological domestication traits white pigmentation, floppy ears, and curly tails predict the strength of behavioral correlations in support of the DS in 78 dog breeds. Contrary to the expectations of covariation among DS traits, we found that morphological traits did not covary among themselves, nor did they predict the strength of behavioral correlations among dog breeds. Further, the number of morphological traits in a breed did not predict the strength of behavioral correlations. Our results thus contrast with the hypothesis that the DS arises due to a shared underlying mechanism, but more importantly, questions if the morphological traits embedded in the DS are actual domestication traits or postdomestication improvement traits. For dogs, it seems highly likely that strong selection for breed specific morphological traits only happened recently and in relation to breed formation. Present day dogs therefore have limited bearing of the initial selection pressures applied during domestication and we should reevaluate our expectations of the DS accordingly.

Selection of behavioral traits holds a prominent role in the domestication of animals, and domesticated species are generally assumed to express reduced fear and reactivity toward novel stimuli compared to their ancestral species. However, very few studies have explicitly tested this proposed link between domestication and reduced fear responses. Of the limited number of studies experimentally addressing the alterations of fear during domestication, the majority has been done on canids. These studies on foxes, wolves, and dogs suggest that decreased expression of fear in domesticated animals is linked to a domestication-driven delay in the first onset of fearful behavior during early ontogeny. Thus, wolves are expected to express exaggerated fearfulness earlier during ontogeny compared to dogs. However, while adult dogs are less fearful toward novelty than adult wolves and wolf-dog hybrids, consensus is lacking on when differences in fear expression arise in wolves and dogs. Here we present the first extended examination of fear development in hand-raised dogs and European gray wolves, using repeated novel object tests from 6 to 26 weeks of age. Contrary to expectations, we found no evidence in support of an increase in fearfulness in wolves with age or a delayed onset of fear response in dogs compared to wolves. Instead, we found that dogs strongly reduced their fear response in the period between 6 and 26 weeks of age, resulting in a significant species difference in fear expression toward novelty from the age of 18 weeks. Critically, as wolves did not differ in their fear response toward novelty over time, the detected species difference was caused solely by a progressive reduced fear response in dogs. Our results thereby suggest that species differences in fear of novelty between wolves and dogs are not caused by a domestication-driven shift in the first onset of fear response. Instead, we suggest that a loss of sensitivity toward novelty with age in dogs causes the difference in fear expression toward novelty in wolves and dogs.

Species with fast life-histories typically prioritize current over future reproductive events, compared to species with slow life-histories. These species therefore require greater energetic input into reproduction, and also likely have less time to realize their reproductive potential. Hence, behaviors that increase access to both resources and mating opportunities, at a cost of increased mortality risk, could coevolve with the pace of life-history. However, whether this prediction holds across species, remains untested under standardized conditions. Here, we test how risky behaviors, which facilitate access to resources and mating opportunities (i.e., activity, boldness, and aggression), along with metabolic rate, coevolve with the pace of life-history across 20 species of killifish that present remarkable divergences in the pace of life-history. We found a positive association between the pace of life-history and aggression, but interestingly not with other behavioral traits or metabolic rate. Aggression is linked to interference competition, and in killifishes is often employed to secure mates, while activity and boldness are more relevant for exploiting energetic resources. Our results suggest that the trade-off between current and future reproduction plays a more prominent role in shaping mating behavior, while behaviors related to energy acquisition may be influenced by ecological factors.

Animals living in groups can show substantial variation in social traits and this affects their social organization. However, as the specific mechanisms driving this organization are difficult to identify in already organized groups typically found in the wild, the contribution of interindividual variation to group level behaviour remains enigmatic. Here, we present results of an experiment to create and compare groups that vary in social organization, and study how individual behaviour varies between these groups. We iteratively sorted individuals between groups of guppies, Poecilia reticulata, by ranking the groups according to their directional alignment and then mixing similar groups. Over the rounds of sorting the consistency of the group rankings increased, producing groups that varied significantly in key social behaviours such as collective activity and group cohesion. The repeatability of the underlying individual behaviour was then estimated by comparing the experimental data to simulations. At the level of basic locomotion, individuals in more coordinated groups displayed stronger interactions with the centre of the group, and weaker interactions with their nearest neighbours. We propose that this provides the basis for a passive phenotypic assortment mechanism that may explain the structures of social networks in the wild.

It has become increasingly clear that a larger brain can confer cognitive benefits. Yet not all of the numerous aspects of cognition seem to be affected by brain size. Recent evidence suggests that some more basic forms of cognition, for instance colour vision, are not influenced by brain size. We therefore hypothesize that a larger brain is especially beneficial for distinct and gradually more complex aspects of cognition. To test this hypothesis, we assessed the performance of brain size selected female guppies (Poecilia reticulata) in two distinct aspects of cognition that differ in cognitive complexity. In a standard reversal-learning test we first investigated basic learning ability with a colour discrimination test, then reversed the reward contingency to specifically test for cognitive flexibility. We found that large-brained females outperformed small-brained females in the reversed-learning part of the test but not in the colour discrimination part of the test. Large-brained individuals are hence cognitively more flexible, which probably yields fitness benefits, as they may adapt more quickly to social and/or ecological cognitive challenges. Our results also suggest that a larger brain becomes especially advantageous with increasing cognitive complexity. These findings corroborate the significance of brain size for cognitive evolution.

Confirmatory path analysis allows researchers to evaluate and compare causal models using observational data. This tool has great value for comparative biologists since they are often unable to gather experimental data on macro-evolutionary hypotheses, but is cumbersome and error-prone to perform. I introduce phylopath, an R package that implements phylogenetic path analysis (PPA) as described by von Hardenberg Gonzalez-Vayer (2113). In addition to the published method, I provide support for the inclusion of binary variables. I illustrate PPA and phylopath by recreating part of a study on the relationship between brain size and vulnerability to extinction. The package aims to make the analysis straight-forward, providing convenience functions, and several plotting methods, which I hope will encourage the spread of the method.

The evolution of the vertebrate brain has remained a topic of intense interest from biologists over many decades. Evolutionary biologists have seen it as an intriguing example of how the size and structure of a trait evolves across large phylogenies and under body size constraints, with both large shifts in deep evolutionary time and continuous smaller scale adaptation. Behavioral ecologists, on the other hand, have put great effort in trying to understand the costs and benefits of brain size and structural variation, usually assuming that the brain morphology of species is the result of a balance between energetic costs and cognitive benefits.

I discuss two hypotheses that aim to explain under what circumstances a higher cognitive ability yields fitness benefits. The predation avoidance hypothesis states that large brains help to avoid predators. The social brain hypothesis predicts that cognition is especially beneficial for animals living in complex social environments. In practice these hypotheses are difficult to differentiate (paper I), as sociality often evolves in response to predation pressure. Comparative studies on either hypothesis should therefore aim to control for effects of the other hypothesis, and experiments may be especially useful in testing more explicit mechanistic explanations.

I put the predation hypothesis to the test using two approaches, a comparative analysis and a within-species experiment. The comparative analysis (paper II) used published data on hawk predation and related it to both relative brain size and relative telencephalon size. While sparrowhawk predation was unrelated to brain morphology, birds that experience more goshawk predation had larger brains and telencephali. Next, I performed an experiment (paper III) on guppies that had been artificially selected for relative brain size. The selection lines have demonstrated differences in cognitive ability, as well as a marked survival difference under predation in females. I exposed fish to either a predator model or a novel object control, varying both sex and group size. Large-brained females performed fewer and shorter predator inspections than small-brained females, while keeping a larger distance from the predator model.

I performed another experiment (paper IV) to investigate differences in social competence. I calculated the duration of contests between random pairs of small- and large-brained males, using movement data. When the loser was large-brained, contests were decided almost 40 minutes earlier than when the loser was small-brained, indicating that the decision for the loser to give up is made quicker with a larger brain.

This thesis ends with an exploration of variation in the scaling relationship between brain and body size across vertebrates (paper V). The observed scaling between brain and body depends on what taxonomic level is under investigation. This effect, however, exclusively occurs in the two classes with the largest brains, mammals and birds. This indicates that strong developmental constraints have been alleviated in the two highly encephalized classes, but not elsewhere.

In conclusion, I find evidence that both predator avoidance and social factors may contribute to the evolution of brain size. Further work on explicit behavioral frameworks for cognitive benefit hypotheses is likely to yield significant insight. Constraints in brain size may be hard to overcome and play an especially large role at a larger taxonomic scale.

The vertebrate brain shows an extremely conserved layout across taxa. Still, the relative sizes of separate brain regions vary markedly between species. One interesting pattern is that larger brains seem associated with increased relative sizes only of certain brain regions, for instance telencephalon and cerebellum. Till now, the evolutionary association between separate brain regions and overall brain size is based on comparative evidence and remains experimentally untested. Here, we test the evolutionary response of brain regions to directional selection on brain size in guppies (Poecilia reticulata) selected for large and small relative brain size. In these animals, artificial selection led to a fast response in relative brain size, while body size remained unchanged. We use microcomputer tomography to investigate how the volumes of 11 main brain regions respond to selection for larger versus smaller brains. We found no differences in relative brain region volumes between large- and small-brained animals and only minor sex-specific variation. Also, selection did not change allometric scaling between brain and brain region sizes. Our results suggest that brain regions respond similarly to strong directional selection on relative brain size, which indicates that brain anatomy variation in contemporary species most likely stem from direct selection on key regions.

Mate choice decisions are central in sexual selection theory aimed to understand how sexual traits evolve and their role in evolutionary diversification. We test the hypothesis that brain size and cognitive ability are important for accurate assessment of partner quality and that variation in brain size and cognitive ability underlies variation in mate choice. We compared sexual preference in guppy female lines selected for divergence in relative brain size, which we have previously shown to have substantial differences in cognitive ability. In a dichotomous choice test, large-brained and wild-type females showed strong preference for males with color traits that predict attractiveness in this species. In contrast, small-brained females showed no preference for males with these traits. In-depth analysis of optomotor response to color cues and gene expression of key opsins in the eye revealed that the observed differences were not due to differences in visual perception of color, indicating that differences in the ability to process indicators of attractiveness are responsible. We thus provide the first experimental support that individual variation in brain size affects mate choice decisions and conclude that differences in cognitive ability may be an important underlying mechanism behind variation in female mate choice.