Globally, fisheries are challenged by the combined impacts of overfishing, degradation of ecosystems and impacts of climate change, while fisheries livelihoods are further pressured by conservation policy imperatives. Fishers' adaptive responses to these pressures, such as exiting from a fishery to pursue alternative livelihoods, determine their own vulnerability, as well as the potential for reducing fishing effort and sustaining fisheries. The willingness and ability to make particular adaptations in response to change, such as exiting from a declining fishery, is influenced by economic, cultural and institutional factors operating at scales from individual fishers to national economies. Previous studies of exit from fisheries at single or few sites, offer limited insight into the relative importance of individual and larger-scale social and economic factors. We asked 599 fishers how they would respond to hypothetical scenarios of catch declines in 28 sites in five western Indian Ocean countries. We investigated how socioeconomic variables at the individual-, household- and site-scale affected whether they would exit fisheries. Site-level factors had the greatest influence on readiness to exit, but these relationships were contrary to common predictions. Specifically, higher levels of infrastructure development and economic vitality - expected to promote exit from fisheries - were associated with less readiness to exit. This may be due to site level histories of exit from fisheries, greater specialisation of fishing households, or higher rewards from fishing in more economically developed sites due to technology, market access, catch value and government subsidies. At the individual and household scale, fishers from households with more livelihood activities, and fishers with lower catch value were more willing to exit. These results demonstrate empirically how adaptive responses to change are influenced by factors at multiple scales, and highlight the importance of understanding natural resource-based livelihoods in the context of the wider economy and society.