Aging is typically associated with declining mental abilities, most prominent for some forms of memory. There are, however, large inter-individual differences within the older population. Some people experience rapid decline whereas others seem almost spared from any adverse effects of aging. This thesis examined the neural underpinnings of such individual differences by using longitudinal observations of episodic memory change across 15-20 years, combined with structural and functional magnetic resonance imaging of the brain. Study I found significant correlations between volume and activity of the hippocampus (HC), and memory change over a 6-year period. That is, individuals with decline in HC function also had declining memory. In contrast, Study II showed that successfully aged individuals, who maintained high memory scores over 15-20 years, had preserved HC function compared to age-matched elderly with average memory change. The successful agers had HC activity levels comparable to those of young individuals, as well as higher frontal activity. Study III revealed that individual differences in memory ability and brain activity of elderly reflect both differential age-related changes, and individual differences in memory ability that are present already in midlife, when age effects are minimal. Specifically, memory scores obtained 15-20 years earlier reliably predicted brain activity in memory-relevant regions such as the frontal cortex and HC. This observation challenges results from previous cross-sectional aging studies that did not consider individual differences in cognitive ability from youth. Collectively the three studies implicate HC and frontal cortex function behind heterogeneity in cognitive aging, both substantiating and qualifying previous results from cross-sectional studies. More generally, the findings highlight the importance of longitudinal estimates of cognitive change for fully understanding the mechanisms of neurocognitive aging.