Navigating the world and remembering events require processing information from various sensory modalities. However, research on human spatial memory and navigation is dominated by vision and little is known about the nonvisual senses. According to a recent theoretical framework, spatial memory and navigation evolved together with olfaction to aid navigation. We tasked participants with memorizing locations of odors and sounds in a virtual reality setting, hypothesizing that odor memory representations would disproportionately interfere with sound memory representations. In three recall times (immediate, 15-min delay, and 1-week delay), we tested whether spatial memory of odors would retroactively interfere with sounds. We predicted that performance should decline more in the sensory modality encoded first than in the modality encoded second, due to retroactive interference, especially when sounds are encoded first. Results indicated a decline in spatial memory performance over time, and similar retroactive interference effects from smells and sounds, but no olfactory-specific interference. However, exploratory results indicated that olfactory retroactive interference is apparent in error trials where participants tended to misplace sounds in the vicinity of smells related to the same concepts (e.g., misplacing the sound of a coffee maker at the location of the smell of coffee). There was no similar interference from sounds on odor misplacements, suggesting asymmetry or “olfactory dominance,” at the conceptual level. Our exploratory results suggest that the hypothesized unique role of olfactory retroactive interference might manifest in the type of errors, rather than in overall performance.