Ultrafast processes in matter, such as the electron emission after light absorption, can now be studied using ultrashort light pulses of attosecond duration (10-18 seconds) in the extreme ultraviolet spectral range. The lack of spectral resolution due to the use of short light pulses has raised issues in the interpretation of the experimental results and the comparison with theoretical calculations. We determine photoionization time delays in neon atoms over a 40-electron volt energy range with an interferometric technique combining high temporal and spectral resolution. We spectrally disentangle direct ionization from ionization with shake-up, in which a second electron is left in an excited state, and obtain excellent agreement with theoretical calculations, thereby solving a puzzle raised by 7-year-old measurements.