Olfactory function, and specifically semantic olfactory memory (i.e., odor identification), has frequently been shown to predict cognitive functioning across multiple domains in old age. This observation suggests that olfactory function can serve as a marker for the integrity of temporolimbic cortical networks, but a clear delineation of this association is still missing. To address this issue, the present study employed voxel-based morphometry in a region of interest-based design to determine the extent to which gray matter volumes of core olfactory and memory areas are associated with olfactory memory performance in an aging population free from neurodegenerative disease. We further aimed to determine potential overlap in structural anatomical correlates, and differences in association strength, for semantic and episodic olfactory memory. Structural magnetic resonance imaging (MRI), episodic and semantic odor memory and episodic and semantic verbal memory data were collected in 422 participants from the Swedish National Study on Aging and Care in Kungsholmen (SNAC-K), all aged >= 60 years. Controlling for age and education, semantic, but not episodic, olfactory memory was positively related to gray matter volume in a cluster extending from the anterior hippocampus and amygdala into the posterior piriform cortex. The observed associations remained even when verbal memory performance was controlled for, supporting a link between the olfactory memory domain and cortical volume over and above more generalized memory abilities. As such, our data provide evidence for distinct functionalstructural associations for semantic odor memory, supporting the idea of temporolimbic integrity as a neurobiological substrate linking olfactory function to cognitive health in old age.