High-energy nuclear collisions create a quark-gluon plasma, whose initial condition and subsequent expansion vary from event to event, impacting the distribution of the eventwise average transverse momentum [𝑃⁡([𝑝_T])]. Disentangling the contributions from fluctuations in the nuclear overlap size (geometrical component) and other sources at a fixed size (intrinsic component) remains a challenge. This problem is addressed by measuring the mean, variance, and skewness of 𝑃⁡([𝑝_T]) in ²⁰⁸Pb+²⁰⁸Pb and ¹²⁹Xe+¹²⁹Xe collisions at √𝑠_NN=5.02 and 5.44 TeV, respectively, using the ATLAS detector at the LHC. All observables show distinct features in ultracentral collisions, which are explained by a suppression of the geometrical component as the overlap area reaches its maximum. These results demonstrate a new technique to separate geometrical and intrinsic fluctuations, providing constraints on initial conditions and properties of the quark-gluon plasma, such as the speed of sound.