Infant-directed speech (IDS) is characterized by a range of register-typical characteristics. Many of those can be objectively measured, such as acoustic-prosodic and structural-linguistic modifications. Perceived vocal affect, however, is a socio-emotional IDS characteristic and is subjectively assessed. Vocal affect goes beyond acoustic-prosodic and structural-linguistic IDS features and includes a perceptive-subjective component in the listener. This study describes vocal affect valence in Swedish IDS during the first year of life, and compares vocal affect between mothers’ and fathers’ IDS and their adult-directed speech. Adult native speakers of Swedish (N = 16) rated affect valence in low-pass filtered IDS samples from free play interactions of mothers with infants at 3, 6, 9, and 12 months (N = 10) and fathers with their 12-month-olds (N = 6). Across the first year, the trajectory of mothers’ positive affect in Swedish IDS shows the highest affect scores toward the youngest infants and the lowest at 9 months of age. This follows a pattern comparable to that in Australian English IDS, showing that mothers express vocal affect similarly across different languages and cultures. Both mothers’ and fathers’ IDS to 12-month-olds have higher positive vocal affect valence than their adult-directed speech. There was no difference in positive affect valence between mothers’ and fathers’ IDS, that is, mothers and fathers express vocal affect to the same extent when talking to their infants. In conclusion, the findings of this study indicate that high positive vocal affect characterizes IDS across different languages and speakers.