Background: Olfactory impairment has been associated with both cognitive impairment and dementia and Alzheimer’s disease (AD). This study aimed to investigate the association between olfactory dysfunction (OD) and change trajectories in different cognitive domains in aging. Method: Participants (n = 2473, mean age = 72 years, 61% female) from the population-based Swedish National study on Aging and Care-Kungsholmen (SNAC-K) were repeatedly assessed with tasks measuring episodic memory, semantic memory, verbal fluency, and perceptual speed across 12 years. OD was measured at baseline and participants were categorized as normosmic, hyposmic, or anosmic based on the Sniffin’ Sticks odor identification task. Linear mixed-effects models were used to assess the associations between baseline OD and rates of cognitive decline. Result: OD was related to poorer baseline performance and faster rates of decline during follow-up in all examined domains, as well as in global cognition. Associations were generally more pronounced for anosmia compared to hyposmia. Conclusion: Olfactory impairment is associated with accelerated decline in aging across a wide range of cognitive domains.