Objectives The aim of the study was to examine the associations between heavy physical workload among middle-aged and older workers and disability pension due to any diagnosis, as well as musculoskeletal, psychiatric, cardiovascular or respiratory diagnoses. The population-based design made it possible to examine dose-response and potential gender differences in the associations. Methods About 1.8 million men and women aged 44-63 years and registered as living in Sweden in 2005 were followed regarding disability pension during 2006-2016, until ages 55-65 years. Mean values of physical workload and job control, estimated through gender-specific job-exposure matrices (JEMs), were assigned to individuals through their occupational titles in 2005. Exposure values were ranked separately for women and men and divided into quintiles. Associations were analyzed with Cox proportional-hazards regression. Results The analyses showed robust, dose-response associations between physical workload and disability pension with a musculoskeletal diagnosis in both genders: the adjusted hazard ratio and 95% confidence interval for those with the heaviest exposure was 2.58 (2.37-2.81) in women and 3.34 (2.83-3.94) in men. Dose-response associations were also seen in relation to disability pension with a cardiovascular or a respiratory diagnosis, though the hazard ratios were smaller. Physical workload was not associated with disability pension with a psychiatric diagnosis after adjustment for job control. Conclusion This study of the entire Swedish population of middle-aged and older workers suggests that higher degrees of physical workload may increase the risk of disability pension overall, and specifically with musculoskeletal, cardiovascular or respiratory diagnosis, in both women and men.