We use transient absorption spectroscopy with circularly polarized x rays to detect laser-excited hole states below the Fermi level and compare their dynamics with that of unoccupied states above the Fermi level in ferromagnetic [Co/Pd] multilayers. While below the Fermi level, an instantaneous and significantly stronger demagnetization is observed, above the Fermi level, the demagnetization is delayed by 35 ± 10 fs. This provides a direct visualization of how ultrafast demagnetization proceeds via initial spin-flip scattering of laser-excited holes to the subsequent formation of spin waves.