Due to their cosmological distances high-energy astrophysical sources allow for unprecedented tests of fundamental physics. Gamma-ray bursts (GRBs) comprise among the most sensitive laboratories for exploring the violation of the central physics principle of Lorentz invariance (LIV), by exploiting the spectral time lag of arriving photons. It has been believed that GRB spectral lags are inherently related with their luminosities, and intrinsic source contributions, which remain poorly understood, could significantly impact the LIV results. Using a combined sample of 49 long and short GRBs observed by the Swift telescope, we perform a stacked spectral lag search for LIV effects. We set novel limits on LIV, including limits on quadratic effects, and systematically explore for the first time the impacts of the intrinsic GRB lag-luminosity relation. We find that source contributions can strongly impact resulting LIV tests, modifying their limits by up to a factor of a few. We discuss constraints coming from GRB 221009A.