Negative effects of habitat fragmentation on individual performance have been widely documented, but relatively little is known about how simultaneous effects on multiple vital rates translate into effects on population viability in long-lived species. In this study, we examined relationships between population size, individual growth, survival and reproduction, and population growth rate in the perennial plant Phyteuma spicatum. Population size positively affected the growth of seedlings, the survival of juveniles, the proportion of adults flowering, and potential seed production. Analyses with integral projection models, however, showed no relationship between population size and population growth rate. This was due to the fact that herbivores and pathogens eliminated the relationship between population size and seed production, and that population growth rate was not sensitive to changes in the vital rates that varied with population size. We conclude that effects of population size on vital rates must not translate into effects on population growth rate, and that populations of long-lived organisms may partly be able to buffer negative effects of small population size on vital rates that have a relatively small influence on population growth rate. Our study illustrates that we need to be cautious when assessing the consequences of habitat fragmentation for population viability based on effects on only one or a few vital rates.