The performance of several standard and popular approaches for calculating X-ray absorption spectra at the carbon, nitrogen, and oxygen K-edges of 40 primarily organic molecules up to the size of guanine has been evaluated, focusing on the low-energy and intense 1s -> pi* transitions. Using results obtained with CVS-ADC(2)-x and fc-CVS-EOM-CCSD as benchmark references, we investigate the performance of CC2, ADC(2), ADC(3/2), and commonly adopted density functional theory (DFT)-based approaches. Here, focus is on precision rather than on accuracy of transition energies and intensities-in other words, we target relative energies and intensities and the spread thereof, rather than absolute values. The use of exchange-correlation functionals tailored for time-dependent DFT calculations of core excitations leads to error spreads similar to those seen for more standard functionals, despite yielding superior absolute energies. Long-range corrected functionals are shown to perform particularly well compared to our reference data, showing error spreads in energy and intensity of 0.2-0.3 eV and similar to 10%, respectively, as compared to 0.3-0.6 eV and similar to 20% for a typical pure hybrid. In comparing intensities, state mixing can complicate matters, and techniques to avoid this issue are discussed. Furthermore, the influence of basis sets in high-level ab initio calculations is investigated, showing that reasonably accurate results are obtained with the use of 6-311++G**. We name this benchmark suite as XABOOM (X-ray absorption benchmark of organic molecules) and provide molecular structures and ground-state self-consistent field energies and spectroscopic data. We believe that it provides a good assessment of electronic structure theory methods for calculating X-ray absorption spectra and will become useful for future developments in this field.