Photoionization of atoms and molecules is one of the fastest processes in nature. The understanding of the ultrafast temporal dynamics of this process often requires the characterization of the different angular momentum channels over a broad energy range. Using a two-photon interferometry technique based on extreme ultraviolet and infrared ultrashort pulses, we measure the phase and amplitude of the individual angular momentum channels as a function of kinetic energy in the outer-shell photoionization of neon. This allows us to unravel the influence of channel interference as well as the effect of the short-range, Coulomb and centrifugal potentials, on the dynamics of the photoionization process.