Individuals with congenital sensory deprivation usually demonstrate altered brain morphology in areas associated with early processing of the absent sense. Here, we aimed to establish whether this also applies to individuals born without a sense of smell (congenital anosmia) by comparing cerebral morphology between 33 individuals with isolated congenital anosmia and matched controls. We detected no morphological alterations in the primary olfactory (piriform) cortex. However, individuals with anosmia demonstrated gray matter volume atrophy in bilateral olfactory sulci, explained by decreased cortical area, curvature, and sulcus depth. They further demonstrated increased gray matter volume and cortical thickness in the medial orbital gyri; regions closely associated with olfactory processing, sensory integration, and value-coding. Our results suggest that a lifelong absence of sensory input does not necessarily lead to morphological alterations in primary sensory cortex and extend previous findings with divergent morphological alterations in bilateral orbitofrontal cortex, indicating influences of different developmental processes.