This paper presents both the result of a search for fossil systems (FSs) within the XMM Cluster Survey and the Sloan Digital Sky Survey and the results of a study of the stellar mass assembly and stellar populations of their fossil galaxies. In total, 17 groups and clusters are identified at z < 0.25 with large magnitude gaps between the first and fourth brightest galaxies. All the information necessary to classify these systems as fossils is provided. For both groups and clusters, the total and fractional luminosity of the brightest galaxy is positively correlated with the magnitude gap. The brightest galaxies in FSs (called fossil galaxies) have stellar populations and star formation histories which are similar to normal brightest cluster galaxies (BCGs). However, at fixed group/cluster mass, the stellar masses of the fossil galaxies are larger compared to normal BCGs, a fact that holds true over a wide range of group/cluster masses. Moreover, the fossil galaxies are found to contain a significant fraction of the total optical luminosity of the group/cluster within 0.5 R-200, as much as 85%, compared to the non-fossils, which can have as little as 10%. Our results suggest that FSs formed early and in the highest density regions of the universe and that fossil galaxies represent the end products of galaxy mergers in groups and clusters.