We aimed to explore sex differences in the association of childhood socioeconomic position (SEP) with the level of cognitive performance and the rate of cognitive decline. We studied 84,059 individuals (55% women; mean age 64 years) from the Survey on Health, Ageing and Retirement in Europe. Sex differences in the association of childhood SEP (household characteristics at age 10) with the level of cognitive performance (verbal fluency, immediate recall, delayed recall) were analysed using multilevel linear regression. Structural equation modelling tested education, depressive symptoms and physical state as mediators. The relationship between childhood socioeconomic advantage and disadvantage and the rate of cognitive decline was assessed using linear mixed-effects models. Higher childhood SEP was associated with a higher level of cognitive performance to a greater extent in women (B = 0.122; 95% CI 0.092–0.151) than in men (B = 0.109; 95% CI 0.084–0.135). The strongest mediator was education. Childhood socioeconomic disadvantage was related to a higher rate of decline in delayed recall in both sexes, with a greater association in women. Strategies to prevent impaired late-life cognitive functioning, such as reducing childhood socioeconomic disadvantages and improving education, might have a greater benefit for women.