Both short (< 6 hr) and long (> 8 hr) sleep are associated with increased mortality. We here investigated whether the association between sleep duration and all-cause, cardiovascular disease and cancer mortality differs between men and women. A cohort of 34,311 participants (mean age and standard deviation = 50.5 ± 15.5 years, 65% women), with detailed assessment of sleep at baseline and up to 20.5 years of follow-up (18 years for cause-specific mortality), was analysed using Cox proportional hazards model to estimate HRs with 95% confidence intervals. After adjustment for covariates, all-cause, cardiovascular disease and cancer mortalities were increased for both < 5 hr and ≥ 9 hr sleep durations (with 6 hr as reference). For all-cause mortality, women who slept < 5 hr had a hazard ratio = 1.54 (95% confidence interval = 1.32–1.80), while the corresponding hazard ratio was 1.05 (95% confidence interval = 0.88–1.27) for men, the interaction being significant (p < 0.05). For cardiovascular disease mortality, exclusion of the first 2 years of exposure, as well as competing risk analysis eliminated the originally significant interaction. Cancer mortality did not show any significant interaction. Survival analysis of the difference between the reference duration (6 hr) and the short duration (< 5 hr) during follow-up showed a gradually steeper reduction of survival time for women than for men for all-cause mortality. We also observed that the lowest cancer mortality appeared for the 5-hr sleep duration. In conclusion, the pattern of association between short sleep duration and all-cause mortality differed between women and men, and the difference between men and women increased with follow-up time.