The semiarid Mahafaly region in southwestern Madagascar is not only a unique biodiversity hotspot, but also one of the poorest regions in the world. Crop failures occur frequently, and despite a great number of rural development programs, no effective progress in terms of improved yields, agricultural income, or well-being among farming households has been observed. In addition to the severe development challenges in the region, environmental degradation and the loss of biodiversity are prevailing issues. This paper takes a social-ecological systems perspective to analyze why the region appears locked in poverty. Specifically, we address the social-ecological interaction between environmental factors such as low and variable precipitation, the lack of sustainable intensification in agriculture resulting in recalcitrant hunger, and several environmental degradation trends. The study is based on (i) longitudinal data from 150 farming households interviewed at high temporal resolution during the course of 2014, and (ii) extensive recall surveys from the southwestern Madagascar project region. The analysis reveals a complex interplay of pronounced seasonality in income generation due to recurrent droughts and crop failures making local farmers highly risk averse. This interplay results in a gradual depletion of environmental assets and hinders the accumulation of capital in the hands of smallholder farmers, and improvements in agricultural production even where environmental conditions would allow for it. As a result, households are insufficiently buffered and insured against repetitive income and food security shocks. This can be understood as a set of interacting, partly nested social-ecological traps, which entrench the Mahafalian smallholder population in deep poverty while the productivity of the environment declines. We provide new insights on the interplay between hunger, poverty, and loss of environmental assets in a global biodiversity hotspot. Finally, we propose a set of key issues that need to be considered to unlock this severe lock-in and enable transformation toward a more sustainable development in southwestern Madagascar.