Providing unpaid informal care to someone who is ill or disabled is a common experience in later life. While a supportive and potentially rewarding role, informal care can become a time and emotionally demanding activity, which may hinder older adults’ quality of life. In a context of rising demand for informal carers, we investigated how caregiving states and transitions are linked to overall levels and changes in quality of life, and how the relationship varies according to care intensity and burden. We used fixed effects and change analyses to examine six-wave panel data (2008–2018) from the Swedish Longitudinal Occupational Survey of Health (SLOSH, n = 5076; ages 50–74). The CASP-19 scale is used to assess both positive and negative aspects of older adults’ quality of life. Caregiving was related with lower levels of quality of life in a graded manner, with those providing more weekly hours and reporting greater burden experiencing larger declines. Two-year transitions corresponding to starting, ceasing and continuing care provision were associated with lower levels of quality of life, compared to continuously not caregiving. Starting and ceasing caregiving were associated with negative and positive changes in quality of life score, respectively, suggesting that cessation of care leads to improvements despite persistent lower overall levels of quality of life. Measures to reduce care burden or time spent providing informal care are likely to improve the quality of life of older people.