Objective To explore the differential associations of neurodegeneration and microvascular lesion load with cognitive decline and dementia in older people and the modifying effect of the APOE genotype on these associations. Methods A sample of 436 participants (age >= 60 years) was derived from the population-based Swedish National study on Aging and Care in Kungsholmen, Stockholm, and clinically examined at baseline (2001-2003) and 3 occasions during the 9-year follow-up. At baseline, we assessed microvascular lesion load using a summary score for MRI markers of lacunes, white matter hyperintensities (WMHs), and perivascular spaces and neurodegeneration load for markers of enlarged ventricles, smaller hippocampus, and smaller gray matter. We assessed cognitive function using the Mini-Mental State Examination (MMSE) test and diagnosed dementia following the Diagnostic and Statistical Manual of Mental Disorders, 4th edition criteria. We analyzed data using linear mixed-effects, mediation, and random-effects Cox models. Results During the follow-up, 46 participants were diagnosed with dementia. Per 1-point increase in microvascular lesion and neurodegeneration score (range 0-3) was associated with multiple adjusted beta-coefficients of -0.35 (95% confidence interval, -0.51 to -0.20) and -0.44 (-0.56 to -0.32), respectively, for the MMSE score and multiple adjusted hazard ratios of 1.68 (1.12-2.51) and 2.35 (1.58-3.52), respectively, for dementia; carrying APOE epsilon 4 reinforced the associations with MMSE decline. WMH volume changes during the follow-up mediated 66.9% and 12.7% of the total association of MMSE decline with the baseline microvascular score and neurodegeneration score, respectively. Conclusions Both cerebral microvascular lesion and neurodegeneration loads are strongly associated with cognitive decline and dementia. The cognitive decline due to microvascular lesions is exacerbated by APOE epsilon 4 and is largely attributed to progression and development of microvascular lesions.