We use the Ultraviolet Imaging of the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey fields (UVCANDELS) to measure half-light radii in the rest-frame far-UV for ∼16,000 disk-like galaxies over 0.5 ≼ z ≼ 3. We compare these results to rest-frame optical sizes that we measure in a self-consistent way and find that the stellar mass–size relation of disk galaxies is steeper in the rest-frame UV than in the optical across our entire redshift range. We show that this is mainly driven by massive galaxies (≳1010Me), which we find to also be among the most dusty. Our results are consistent with the literature and have commonly been interpreted as evidence of inside-out growth wherein galaxies form their central structures first. However, they could also suggest that the centers of massive galaxies are more heavily attenuated than their outskirts. We distinguish between these scenarios by modeling and selecting galaxies at z = 2 from the VELA simulation suite in a way that is consistent with UVCANDELS. We show that the effects of dust alone can account for the size differences we measure at z = 2. This indicates that, at different wavelengths, size differences and the different slopes of the stellar mass–size relation do not constitute evidence for inside-out growth.