We extract the spectra of the fastest variability (above 10 Hz) from the black hole XTE J1550-564 during a transition from hard to soft state on the rise to outburst. We confirm previous results that the rapid variability contains no significant disc component despite this being strongly present in the total spectrum of the softer observations. We model ionized reflection significantly better than previous work, and show that this is also suppressed in the rapid variability spectrum compared to the total emission. This is consistent with the fast variability having its origin in a hot inner flow close to the black hole rather than in the accretion disc or in a corona above it. However, the rapid variability spectrum is not simply the same as the total Comptonized emission. It is always significantly harder, by an amount which increases as the spectrum softens during the outburst. This adds to evidence from time lags that the Comptonization region is inhomogeneous, with harder spectra produced closest to the black hole, the same region which produces the fastest variability.