We discuss the relation between particle number conservation and topological phases. In four spatial dimensions, we find that systems belonging to different topological phases in the presence of a U(1) charge conservation can be connected adiabatically, i.e., without closing the gap, upon intermediately breaking this local symmetry by a superconducting term. The time reversal preserving topological insulator states in two and three dimensions which can be obtained from the four-dimensional parent state by dimensional reduction inherit this protection by charge conservation. Hence, all topological insulators can be adiabatically connected to a trivial insulating state without breaking time reversal symmetry, provided an intermediate superconducting term is allowed during the adiabatic deformation. Conversely, in one spatial dimension, non-symmetry-protected topological phases occur only in systems that break U(1) charge conservation. These results can intuitively be understood by considering a natural embedding of the classifying spaces of charge conserving Hamiltonians into the corresponding Bogoliubov-de Gennes classes.