A central problem in evolutionary biology is to identify the forces that maintain genetic variation for fitness in natural populations. Sexual antagonism, in which selection favours different variants in males and females, can slow the transit of a polymorphism through a population or can actively maintain fitness variation. The amount of sexually antagonistic variation to be expected depends in part on the genetic architecture of sexual dimorphism, about which we know relatively little. Here, we used a multivariate quantitative genetic approach to examine the genetic architecture of sexual dimorphism in a scent-based fertilization syndrome of the moss Ceratodon purpureus. We found sexual dimorphism in numerous traits, consistent with a history of sexually antagonistic selection. The cross-sex genetic correlations (r(mf)) were generally heterogeneous with many values indistinguishable from zero, which typically suggests that genetic constraints do not limit the response to sexually antagonistic selection. However, we detected no differentiation between the female- and male-specific trait (co)variance matrices (G(f) and G(m), respectively), meaning the evolution of sexual dimorphism may be constrained. The cross-sex cross-trait covariance matrix B contained both symmetric and asymmetric elements, indicating that the response to sexually antagonistic or sexually concordant selection, and the constraint to sexual dimorphism, are highly dependent on the traits experiencing selection. The patterns of genetic variances and covariances among these fitness components is consistent with partly sex-specific genetic architectures having evolved in order to partially resolve multivariate genetic constraints (i.e. sexual conflict), enabling the sexes to evolve towards their sex-specific multivariate trait optima.