We present calculations for attosecond atomic delays in the photoionization of noble-gas atoms based on the full two-color two-photon random-phase approximation with exchange in both the length and velocity gauge. Gauge-invariant atomic delays are demonstrated for the complete set of diagrams. The results are used to investigate the validity of the common assumption that the measured atomic delays can be interpreted as a one-photon Wigner delay and a universal continuum-continuum contribution that depends only on the kinetic energy of the photoelectron, the laser frequency, and the charge of the remaining ion, but not on the specific atom or the orbital from which the electron is ionized. Here, we find that although effects beyond the universal IR-photoelectron continuum-continuum transitions are rare, they do occur in special cases such as around the 3s Cooper minimum in argon. We conclude also that in general the convergence in terms of many-body diagrams is considerably faster in the length gauge than in the velocity gauge.