Statistical physics and dynamical systems theory are key tools to study high-impact geophysical events such as temperature extremes, cyclones, thunderstorms, geomagnetic storms, and many others. Despite the intrinsic differences between these events, they all originate as temporary deviations from the typical trajectories of a geophysical system, resulting in well-organized, coherent structures at characteristic spatial and temporal scales. While statistical extreme value analysis techniques are capable of providing return times and probabilities of occurrence of certain geophysical events, they are not apt to account for their underlying physics. Their focus is to compute the probability of occurrence of events that are large or small with respect to some specific observable (e.g., temperature, precipitation, solar wind), rather than to relate rare or extreme phenomena to the underlying anomalous geophysical regimes. This paper outlines this knowledge gap, presenting some related challenges, new formalisms and briefly commenting on how stochastic approaches tailored to the study of extreme geophysical events can help to advance their understanding.