The association of poor pulmonary function (PF) with cognitive trajectories and structural brain differences remains unclear. Within the Rush Memory and Aging Project, 1377 dementia-free subjects were followed up to 21 years. PF was assessed with a composite score measured at baseline. Global and domain-specific cognitive function was assessed annually constructed from 19 cognitive tests. A subsample of 351 participants underwent brain magnetic resonance imaging to investigate the cross-sectional association between PF and structural brain volumes. We found that low PF was related to faster decline in global cognition, and domain-specific function including episodic memory, semantic memory, working memory, visuospatial ability, and perceptual speed. In addition, low PF was associated with smaller volumes of total brain, white matter and gray matter, and larger white matter hyperintensities volume. Our results suggest that low PF is associated with faster cognitive decline, and both neurodegeneration and vascular brain lesions may underlie the association.