Olfactory loss can be acquired (patients with a history of olfactory experiences), or inborn (patients without olfactory experiences/life-long inability to smell). Inborn olfactory loss, or congenital anosmia (CA), is relatively rare and there is a knowledge gap regarding the compensatory neural mechanisms involved in this condition. The study aimed to investigate the top-down olfactory processing in patients with CA or idiopathic acquired anosmia (IA) in comparison to normosmia controls (NC) during expectancy and reading of odor-associated words. Functional magnetic resonance imaging was used to assess brain activations in 14 patients with CA, 8 patients with IA, and 16 NC healthy participants during an expectancy and reading task. Words with strong olfactory associations (OW) (e.g. “banana”) or with little or no olfactory associations (CW) (e.g. “chair”) were used as stimuli and were presented with a block design Analyses were conducted to explore the brain activation in response to OW expectancy or OW reading between groups (CW as baseline). During the expectancy condition of OW, IA and NC groups showed stronger activation in posterior OFC extending to right insula, caudate region and frontal medial OFC respectively. Whereas during the reading condition of OW, CA patients showed stronger activation in posterior OFC extending to the insula. Increased activation of higher-order brain regions related to multisensory integration among CA patients suggests a compensatory mechanism for processing semantic olfactory cues.