Context. Large area catalogs of galaxy clusters constructed from ROSAT All-Sky Survey provide the basis for our knowledge of the population of clusters thanks to long-term multiwavelength efforts to follow up observations of these clusters.

Aims. The advent of large area photometric surveys superseding previous, in-depth all-sky data allows us to revisit the construction of X-ray cluster catalogs, extending the study to lower cluster masses and higher redshifts and providing modeling of the selection function.

Methods. We performed a wavelet detection of X-ray sources and made extensive simulations of the detection of clusters in the RASS data. We assigned an optical richness to each of the 24 788 detected X-ray sources in the 10 382 square degrees of the Baryon Oscillation Spectroscopic Survey area using red sequence cluster finder redMaPPer version 5.2 run on Sloan Digital Sky Survey photometry. We named this survey COnstrain Dark Energy with X-ray (CODEX) clusters.

Results. We show that there is no obvious separation of sources on galaxy clusters and active galactic nuclei (AGN) based on the distribution of systems on their richness. This is a combination of an increasing number of galaxy groups and their selection via the identification of X-ray sources either by chance or by groups hosting an AGN. To clean the sample, we use a cut on the optical richness at the level corresponding to the 10% completeness of the survey and include it in the modeling of the cluster selection function. We present the X-ray catalog extending to a redshift of 0.6.

Conclusions. The CODEX suvey is the first large area X-ray selected catalog of northern clusters reaching fluxes of 10⁻¹³ ergs s⁻¹ cm⁻². We provide modeling of the sample selection and discuss the redshift evolution of the high end of the X-ray luminosity function (XLF). Our results on z <  0.3 XLF agree with previous studies, while we provide new constraints on the 0.3 <  z <  0.6 XLF. We find a lack of strong redshift evolution of the XLF, provide exact modeling of the effect of low number statistics and AGN contamination, and present the resulting constraints on the flat ΛCDM.