Preferred retirement age (PRA) is one key dimension when studying retirement decision-making. However, little is known concerning how PRA develops over the late career years. This study used a person-centered approach to longitudinally investigate trajectories of PRA and how they differ in self-rated health, perceived work ability, and effort–reward imbalance (ERI) at baseline levels and over 6 years. The study used data from four waves (2010, 2012, 2014, and 2016) of the Swedish Longitudinal Occupational Survey of Health. The sample consisted of 1,510 individuals aged 50–55 in 2010, who answered to the questionnaire for those in paid work (including self-employment) at the baseline and at least one of the following waves. Results from the latent class growth curve modeling show both within- and between-person variability in PRA over the 6-year span. We found four distinct trajectories, which differed both at the baseline levels and in the patterns of change in PRA: “C1: normative, relatively stable PRA” (42% of all participants); “C2: considerably early, increasing PRA” (6% of the participants); “C3: late, relatively stable PRA” (4% of the participants); and “C4: early, increasing PRA” (49% of the participants). Participants revealed a clear preference for retirement before the age of 65. Trajectories comprising earlier PRA showed poorer self-rated health, poorer work ability, and higher levels of ERI at the baseline and over time. The findings reinforce the importance of healthy work environments that promote work ability and facilitate a balance between efforts and rewards for encouraging longer working lives.