In this population-based prospective study, we examined the association of job demand-control combinations with dementia, and explored the roles of Apolipoprotein E epsilon4 (APOE epsilon 4) and work duration in this association. A total of 2,579 dementia-free individuals aged 60+ years from Sweden were followed over 12 years. Dementia diagnosis was made by physicians. Lifelong occupational experience was collected, and job demands and control were assessed using a psychosocial job-exposure matrix. Data were analyzed using multivariate Cox proportional hazard models. During the follow-up, 282 people developed dementia. Passive jobs (low control/low demands) were related to a higher risk of dementia compared with active jobs (high control/high demands) among the younger-old (aged <= 72 years), but not among the older-old (aged >= 78 years). Among the younger-old, compared to those with no passive job experience, those with 11+ years in passive jobs had a higher dementia risk. The joint-effect analyses showed that APOE epsilon 4 carriers with passive jobs had an even higher risk of dementia compared to APOE epsilon 4 non-carriers with active jobs. These findings suggest that passive jobs are related to a higher dementia risk among the younger-old. APOE epsilon 4 and long work duration may amplify the impact of passive jobs on dementia.