Background: Estimating excess mortality and years of life lost (YLL) attributed to coronavirus disease 19 (COVID-19) infection provides a comprehensive picture of the mortality burden on society. We aimed to estimate the impact of the COVID-19 pandemic on age- and sex-specific excess mortality and YLL in Sweden during the first 17 months of the pandemic. Methods: In this population-based observational study, we calculated age- and sex-specific excess all-cause mortality and excess YLL during 2020 and the first 5 months of 2021 and cause-specific death [deaths from cardiovascular disease (CVD), cancer, other causes and deaths excluding COVID-19] in 2020 compared with an average baseline for 2017–19 in the whole Swedish population. Results: COVID-19 deaths contributed 9.9% of total deaths (98 441 deaths, 960 305 YLL) in 2020, accounting for 75 151 YLL (7.7 YLL/death). There were 2672 (5.7%) and 1408 (3.0%) excess deaths, and 19 141 (3.8%) and 3596 (0.8%) excess YLL in men and women, respectively. Men aged 65–110 years and women aged 75–110 years were the greatest contributors. Fewer deaths and YLL from CVD, cancer and other causes were observed in 2020 compared with the baseline adjusted to the population size in 2020. Conclusions: Compared with the baseline, excess mortality and YLL from all causes were experienced in Sweden during 2020, with a higher excess observed in men than in women, indicating that more men died at a younger age while more women died at older ages than expected. A notable reduction in deaths and YLL due to CVD suggests a displacement effect from CVD to COVID-19.