Animals are typically expected to avoid mating with relatives due to the costs associated with incestuous matings. Yet for more than 4 decades, theoretical models have consistently suggested that animals may tolerate, or even prefer, mating with relatives under a broad range of conditions. However, empirical studies that evaluate inbreeding strategy under alternative social and sexual contexts remain scarce. Here, we investigate how experimental variation in sexual and social complexity influence precopulatory inbreeding avoidance behaviors in the guppy Poecilia reticulata, a species known to experience inbreeding depression. In an integrated set of experiments, we examined if sexual and affiliative behaviors of virgin and experienced females and males were differentially directed towards either related or unrelated individuals. In simple social context, neither virgin nor experienced females or males showed a preference for related or unrelated partners. In more sexually and socially complex free-swimming arenas, females appeared to respond slightly more to related males, but this effect was not statistically significant once corrected for multiple comparisons; male preference remained unchanged. Overall, these findings challenge previous reports of preference shifts between virgin and experienced female guppies and suggest that inbreeding avoidance behaviors may be less prevalent in complex social environments than previously thought. Although animals are often expected to avoid inbreeding, theory suggests mating with relatives can sometimes be beneficial. We tested how social and sexual complexity affects guppies' mating decisions. In simple setups, neither females nor males preferred relatives or nonrelatives. In more naturalistic, free-swimming arenas, females showed a weak tendency to respond more to related males, while males showed no preference. These results suggest that inbreeding avoidance may be limited in guppies.

Group living species are constantly facing decisions about which conspecifics to associate with. These decisions are likely guided by the benefits and costs of associations. Associating in larger groups can minimize predation risk, while also providing individuals with beneficial social information from conspecifics. By contrast, associating with multiple individuals could also increase potentially costly conflicts over resources and/or mates. Here, we examine male association strategies in a shoaling fish, the pygmy halfbeak (Dermogenys collettei), by confronting males with four different social scenarios. We found that males preferentially associated with an opposite sex pair (i.e., a female and a male) over rival males, but showed no preference when choosing between a pair and females. By contrast, the number of presented fish (one or two) did not influence male association preferences, indicating that the observed male behavior was not driven by shoaling behavior. Finally, male association preference correlated with the duration of courtship behavior that the male observed, but only under specific social scenarios. Overall, our data show that males followed informed association strategies that are primarily driven by mating opportunities. Using a simplified social environment, we illuminate which basic rules might drive association behavior in complex social groups.

Animals deploy a wide range of anti-predator adaptations, often combining multiple strategies into a ‘defence portfolio’. However, why certain defences evolve together whereas others are mutually exclusive remains unclear. Here we use a phylogenetic approach to examine the evolution of conspicuous visual markings—including eyespots and other conspicuous markings—alongside alternative anti-predator defences in skates and rays (Batoidea), a diverse group of cartilaginous fishes. We compiled data on the presence, type and number of conspicuous markings in 580 species of skates and rays, together with information on robust mechanical and electrical defences (venomous caudal stings and electric organs), adult body size and habitat depth as a proxy for light environment. We show that eyespots evolve stepwise from other conspicuous markings and that evolutionary gains of multiple markings are typically followed by reductions to fewer (often paired) markings. We further find that defence strategies follow alternative evolutionary trajectories: species possessing robust defences rarely evolve eyespots, whereas eyespots are favoured in smaller-bodied species lacking such defences and inhabiting well-lit waters where visual signals are most effective. These results show that accounting for multiple anti-predator defences can resolve why iconic anti-predator defences, such as eyespots, appear in some taxonomic groups but are conspicuously absent in others.

Species-specific sexual traits facilitate species-assortative mating by reducing mating across species and reducing hybrid sexual attractiveness. For learned sexual traits, such as song in oscine birds, species distinctiveness can be eroded when species co-occur. Transcriptional regulatory divergence in brain regions involved in sensory learning is hypothesized to maintain species distinctiveness, but relatively few studies have compared gene expression in relevant brain regions between closely related species. Species differences in song are an important premating reproductive barrier between the collared (Ficedula albicollis) and pied flycatcher (F. hypoleuca). Here, we compare brain gene expression in adult males from each species and their naturally occurring F1 hybrids. We report overall conserved expression across species in a portion of the brain containing regions and nuclei known to be involved in song responses and learning. Further, among those genes that were differentially expressed between species, we find largely intermediate expression in hybrids. A single gene, SYT4 (synaptotagmin 4), known to be singing-associated, both was differentially expressed and has a putative upstream transcriptional regulatory factor containing fixed differences between the 2 species. Although a finer-scale investigation limited to song-specific regions may reveal further species differences, our findings provide insight into regulatory divergence in the brain between closely related species.

Animals can employ various spatial learning strategies to navigate through their environment, and the proficiency in specific strategies varies greatly both intraspecifically and interspecifically. Currently, the neural basis of this variation is poorly understood. Here, we tested whether variation in performance in egocentric and allocentric spatial learning strategies is related to differential investment in distinct brain regions. To do so, we used guppies (Poecilia reticulata) from artificial selection lines expressing differences in relative telencephalon size, and tested their ability to learn a spatial task, based on either egocentric (left–right) or allocentric (environmental) cues. Surprisingly, fish with larger telencephalons showed enhanced performance in both tasks, regardless of cue type, suggesting a more complicated role of the fish telencephalon in spatial learning than previously thought. Our study provides the first direct evidence that evolutionary changes in relative telencephalon size lead to corresponding shifts in spatial cognition at the within-species level. Furthermore, our results offer critical and novel insights regarding the function of the telencephalon and its role in the evolution of spatial cognition.

In species expressing parental care, juveniles often utilize cues from their parents to distinguish between safe and dangerous situations. For example, nestling songbirds beg in response to parental feeding cues but stop vocalizing in response to parental alarm calls. Previous studies have shown that nestling birds discriminate between calls of their own and other species. However, it is unknown whether and how nestlings discriminate call variation across populations within their own species. Adult pied flycatchers, Ficedula hypoleuca, produce acoustically identical calls at a fast rate when a predator is nearby, and slowly during feeding visits, potentially signalling the level of danger near the nest. Taking advantage of this functional duality, our study examines whether nestling discrimination of call variation across populations depends on the context in which they are produced. In response to rapidly repeated predator-associated calls, we found that nestlings reduced begging activity similarly, whether calls were from local or foreign populations. Conversely, in response to slowly repeated feeding-associated calls, nestlings begged more in response to local than foreign conspecific calls. Our results show that nestlings discriminate within-species call variation but adjust their response threshold depending on the call context, which we argue may optimize their begging activity in order to get fed, while simultaneously avoiding predation. This study underscores the importance of studying the effects of within-species variation in nonmating signals, such as calls, on songbird behaviour.

Animals often reproduce in complex environments, which should generate selection for both enhanced detectability in signaling traits and improved cognitive processing abilities. However, the extent to which signaling and cognitive traits have evolved to overcome the challenges of interacting in complex habitats remains understudied. We examined whether habitat complexity influences sexual selection in the pygmy halfbeak, Dermogenys collettei, a small livebearing freshwater fish. Using free-swimming arenas, we created low- and high-complexity environments and observed mating behaviors in mixed-sex groups. While the opportunity for sexual selection did not differ significantly between environments for either sex, we observed positive selection gradients for female brain size in open arenas, but not in complex habitats. Selection on morphological traits associated with visual signaling was also primarily detected in open environments, particularly in females. These results suggest that habitat complexity may reduce selection pressures on both cognitive traits, such as brain size, and signaling traits relevant to mate choice. Together, our findings highlight the importance of integrating cognitive traits into sexual selection theory and considering sex-specific selection across ecologically relevant contexts.

Signals in vocal communication systems range from innate to learned. Although innate and learned signals are often assumed to be independent, Darwin speculated that they could be evolutionarily related, with the former being the foundation of the latter even in our own communication system, language. Here we test this hypothesis by studying the vocal communication systems of avian hosts of brood parasites. First, we show that 21 bird species separated by approximately 53 million years of evolution produce structurally similar ‘whining’ vocalizations towards their respective brood parasites. Exploring the social correlates of whining vocalization production, we find that species that produce this vocalization often exist in areas with dense parasite–host networks, suggesting that its production facilitates interactions among host species. Experiments across three continents show that this vocalization is referential towards brood parasites in multiple host species, that hearing them elicits an innate rapid recruiting response, and that host species from different continents respond equally to the whining vocalizations of each other, indicating that convergent use facilitates cooperative defences across species. Our results provide an example of a referential animal vocalization for which sound production in the correct context is learned but for which hearing it elicits an innate response, representing an intermediate between innate and learned signals.

Song divergence driven by social learning has been proposed to be a key factor driving allopatric speciation in oscine birds. Songbirds often respond more to songs deriving from their local population, suggesting the potential for acoustic divergence across populations to shape both intra- and intersexual interactions. However, many of these studies were conducted on species with simple songs and, as a result, we know comparatively little about the emergence of population differences and song discrimination in species with complex songs. We addressed this question in the pied flycatcher (Ficedula hypoleuca) by calculating the dissimilarity of songs from 2 foreign populations as well as from our study site to the local centroid. We then conducted a paired-design playback experiment where both local and foreign songs were played simultaneously. We found that pied flycatcher males showed significantly stronger responses to those songs that sounded more similar to the local population. This suggests that despite the high complexity of the pied flycatcher song, individuals are still able to discriminate across populations. Our results support the hypothesis that learned song divergence can act as a mechanism for assortative mating and allopatric speciation.

Animals are known to adjust their acoustic signals in the presence of anthropogenic noise. These changes may affect fitness by altering susceptibility to predators or changing signal efficiency in intra- and intersexual interactions. Male field crickets, Gryllus bimaculatus, chirp to attract females, with chirp rate being an important success factor. Males reduce the chirp rate when exposed to predators or traffic noise and increase it in response to male competitors. However, the combined effects of these pressures on signalling are unknown. This study examined whether antipredator responses are influenced by male-male competition and variations in traffic noise exposure. We used substrate-borne vibrations to simulate a predator approaching and varied perceived male-male competition using playbacks to calling males. We found that responses to increased competition were affected by variation in traffic noise exposure, with males chirping faster as noise levels increased. Additionally, antipredator responses depended on an interaction between traffic noise and competition. Under high competition, males reduced the chirp rate as traffic noise increased. Our results demonstrate that adjustments in signal production in response to noise pollution may negatively impact communication in both antipredator and competitive contexts, indicating more pervasive effects of anthropogenic noise than previously recognized.