Characterizing aging-related decline trajectories in mental abilities, and relationships of the epsilon 4 allele of the Apolipoprotein gene, helps to identify individuals at high risk for dementia. However, longitudinal changes in olfactory and non-olfactory cognitive abilities have not been investigated in relation to the epsilon 4 allele. In the present study, participants from a large population-based study (657 middle-aged and 556 old) were tested over 10 years on their performance on an odor identification task and three non-olfactory cognitive tasks; MMSE, episodic memory, and semantic memory. Our key finding is that in middle-aged participants, odor identification declined twice as fast for epsilon 4/4 homozygotes, compared to non-carriers. However, in old participants, the epsilon 4/4 homozygotes showed an impaired odor identification ability, but they declined at a similar rate as the non-carriers. Furthermore, in old participants all assessments displayed aging-related declines, but exaggerated declines in epsilon 4-carriers were found only in MMSE and episodic memory assessments. In sum, we present evidence that odor identification ability starts to decline already in middle-aged, and that carriers of epsilon 4/4, who are at highest risk of developing dementia, decline twice as fast. Our results may have implications for use of odor identification assessment in detection of early-stage dementia.