Chromosomal rearrangements, such as inversions, have received considerable attention in the speciation literature due to their hampering effects on recombination. Less is known about how other rearrangements, such as chromosome fissions and fusions, can affect the evolution of reproductive isolation. Here, we use crosses between populations of the wood white butterfly (Leptidea sinapis) with different karyotypes to identify genomic regions associated with hybrid inviability. We map hybrid inviability candidate loci by contrasting allele frequencies between F2 hybrids that survived until the adult stage with individuals of the same cohort that succumbed to hybrid incompatibilities. Hybrid inviability candidate regions have high genetic differentiation between parental populations, reduced recombination rates, and are enriched near chromosome fusions. By analysing sequencing coverage, we exclude aneuploidies as a direct link between hybrid inviability and chromosome fusions. Instead, our results point to an indirect relationship between hybrid inviability and chromosome fusions, possibly related to reduced recombination in fused chromosomes. Thus, we map postzygotic isolation to chromosomal rearrangements, providing crucial empirical evidence for the idea that chromosome number differences between taxa can contribute to speciation.