The direct access of olfactory afferents to memory-related cortical systems has inspired theories about the role of the olfactory pathways in the development of cortical neurodegeneration in Alzheimer’s disease (AD). In this study, we used baseline olfactory identification measures with longitudinal flortaucipir and PiB PET, diffusion MRI of 89 cognitively normal older adults (73.82 ± 8.44 years; 56% females), and a transcriptomic data atlas to investigate the spatiotemporal spreading and genetic vulnerabilities of AD-related pathology aggregates in the olfactory system. We find that odor identification deficits are predominantly associated with tau accumulation in key areas of the olfactory pathway, with a particularly strong predictive power for longitudinal tau progression. We observe that tau spreads from the medial temporal lobe structures toward the olfactory system, not the reverse. Moreover, we observed a genetic background of odor perception-related genes that might confer vulnerability to tau accumulation along the olfactory system.