Loss of olfactory function is common in old age, but evidence regarding qualitative olfactory dysfunction in the general older population is scarce. The current study investigates the prevalence and correlates of phantom smell experiences (phantosmia) in a population-based study (Swedish National Study on Aging and Care in Kungsholmen [SNAC-K]) of Swedish adults (n = 2569) aged between 60 and 90 years. Phantosmia was assessed through a standardized interview and defined as reporting having experienced an odor percept in the absence of any stimuli in the surrounding environment that could emit the odor. The relationships between phantosmia and demographic, genetic, health-related, and behavioral variables were analyzed with hierarchical logistic regression analyses. The overall prevalence of phantom smells was 4.9%, and was associated with female gender, carrying the met allele of the BDNF gene, higher vascular risk burden, and reporting distorted smell sensations (parosmia). Olfactory dysfunction was, however, not related to phantosmia. The most frequently reported phantom smell was smoky/burnt. A novel finding was that some individuals reported phantom smells with an autobiographical connotation. The results from this study indicate that the prevalence of phantosmia in the general older population is not negligible and that some factors that are beneficial for preserved olfactory function, such as female gender and the BDNF met allele, are also associated with the occurrence of phantom smells.