Diabetes has been linked to dementia risk; however, the cognitive trajectories in older adults with diabetes remain unclear. We aimed to investigate the effect of prediabetes and diabetes on cognitive trajectories among cognitively intact older adults in a long-term follow-up study. Within the Swedish Adoption/Twin Study of Aging, 793 cognitively intact older adults aged >= y50 were identified at baseline and followed for up to 23 years. Based on standardized scores from 11 cognitive tests, administered at baseline and up to seven follow-ups, four cognitive domains (verbal abilities, spatial/fluid, memory, perceptual speed) were identified by principal-component analysis. Prediabetes was defined according to blood glucose levels in diabetes-free participants. Diabetes was ascertained based on self-report, hypoglycemic medication use and blood glucose levels. Data were analyzed with linear mixed-effect models adjusting for potential confounders. At baseline, 68 participants (8.6%) had prediabetes and 45 (5.7%) had diabetes. Compared to diabetes-free individuals, people with diabetes had a steeper decline over time in perceptual speed and verbal abilities. The annual declines in these domains were greater than the annual decline in memory. Prediabetes was associated with lower performance in memory in middle-age, but also associated with a less steep memory decline over the follow-up. Diabetes is associated with a faster decline in perceptual speed and verbal abilities, while prediabetes is associated with lower memory performance in middle-age. However, the detrimental effects of hyperglycemia seem to not affect memory over time.