Black hole X-ray binaries display significant stochastic variability on short time-scales (0.01–100 s), with a complex pattern of lags in correlated variability seen in different energy bands. This behaviour is generally interpreted in a model where slow fluctuations stirred up at large radii propagate down through the accretion flow, modulating faster fluctuations generated at smaller radii. Coupling this scenario with radially stratified emission opens the way to measure the propagation time-scale from data, allowing direct tests of the accretion flow structure. We previously developed a model based on this picture and showed that it could fit the Neutron star Interior Composition Explorer (NICER; 0.5–10 keV) data from the brightest recent black hole transient, MAXI J1820+070. However, here we show it fails when extrapolated to higher energy variability data from the Insight-Hard X-ray Modulation Telescope(HXMT). We extend our model so that the spectrum emitted at each radius changes shape in response to fluctuations (pivoting) rather than just changing normalization. This gives the strong suppression of fractional variability as a function of energy seen in the data. The derived propagation time-scale is slower than predicted by a magnetically arrested disc (MAD), despite this system showing a strong jet. Our new model jointly fits the spectrum and variability up to 50 keV, though still cannot match all the data above this. Nonetheless, the good fit from 3 to 40 keV means the quasi-periodic oscillation (QPO) can most easily be explained as an extrinsic modulation of the flow, such as produced in the Lense–Thirring precession, rather than arising in an additional spectral-timing component such as the jet.