The evolution of the vertebrate brain has remained a topic of intense interest from biologists over many decades. Evolutionary biologists have seen it as an intriguing example of how the size and structure of a trait evolves across large phylogenies and under body size constraints, with both large shifts in deep evolutionary time and continuous smaller scale adaptation. Behavioral ecologists, on the other hand, have put great effort in trying to understand the costs and benefits of brain size and structural variation, usually assuming that the brain morphology of species is the result of a balance between energetic costs and cognitive benefits.

I discuss two hypotheses that aim to explain under what circumstances a higher cognitive ability yields fitness benefits. The predation avoidance hypothesis states that large brains help to avoid predators. The social brain hypothesis predicts that cognition is especially beneficial for animals living in complex social environments. In practice these hypotheses are difficult to differentiate (paper I), as sociality often evolves in response to predation pressure. Comparative studies on either hypothesis should therefore aim to control for effects of the other hypothesis, and experiments may be especially useful in testing more explicit mechanistic explanations.

I put the predation hypothesis to the test using two approaches, a comparative analysis and a within-species experiment. The comparative analysis (paper II) used published data on hawk predation and related it to both relative brain size and relative telencephalon size. While sparrowhawk predation was unrelated to brain morphology, birds that experience more goshawk predation had larger brains and telencephali. Next, I performed an experiment (paper III) on guppies that had been artificially selected for relative brain size. The selection lines have demonstrated differences in cognitive ability, as well as a marked survival difference under predation in females. I exposed fish to either a predator model or a novel object control, varying both sex and group size. Large-brained females performed fewer and shorter predator inspections than small-brained females, while keeping a larger distance from the predator model.

I performed another experiment (paper IV) to investigate differences in social competence. I calculated the duration of contests between random pairs of small- and large-brained males, using movement data. When the loser was large-brained, contests were decided almost 40 minutes earlier than when the loser was small-brained, indicating that the decision for the loser to give up is made quicker with a larger brain.

This thesis ends with an exploration of variation in the scaling relationship between brain and body size across vertebrates (paper V). The observed scaling between brain and body depends on what taxonomic level is under investigation. This effect, however, exclusively occurs in the two classes with the largest brains, mammals and birds. This indicates that strong developmental constraints have been alleviated in the two highly encephalized classes, but not elsewhere.

In conclusion, I find evidence that both predator avoidance and social factors may contribute to the evolution of brain size. Further work on explicit behavioral frameworks for cognitive benefit hypotheses is likely to yield significant insight. Constraints in brain size may be hard to overcome and play an especially large role at a larger taxonomic scale.