To address the challenges of demographic aging, governments and organizations encourage extended working lives. This study investigates how individual health- and age-related workplace factors contribute to preferred, expected and actual retirement timing, as well as to the congruency between preferences vs. expectations, and preferences vs. actual retirement. We used data from a representative Swedish longitudinal sample comprising 4058 workers aged 50–64, with follow-up data regarding actual retirement timing available for 1164 respondents. Multinomial logistic regression analyses suggest that later preferred, expected, and actual retirement timing were, to different extent, influenced by better health, an age-friendly workplace and feeling positive regarding the future at work. Emotional exhaustion, age-related inequalities at work and experiencing aging as an obstacle increased the likelihood of preferring to retire earlier than one expected to, over retiring at the time one expected to. Those with better health and positive work prospects were less likely to prefer retiring earlier than they expected to, and more likely to being “pulled toward working until 65 and beyond”, compared to being “pulled toward early retirement”. Experiencing aging as an obstacle decreased the chances of being “pulled toward working until 65 and beyond”. The results provide insights on how to facilitate extended working lives.