We report two epochs of simultaneous near-infrared (IR) and X-ray observations of the low-mass X-ray binary black hole candidate Swift J1753.5-0127 with a subsecond time resolution during its long 2005-2016 outburst. Data were collected strictly simultaneously with VLT/ISAAC (KS band, 2.2 μm) and RXTE (2-15 keV) or XMM-Newton (0.7-10 keV). A clear correlation between the X-ray and the IR variable emission is found during both epochs but with very different properties. In the first epoch, the near-IR variability leads the X-ray by ∼130 ms, which is the opposite of what is usually observed in similar systems. The correlation is more complex in the second epoch, with both anti-correlation and correlations at negative and positive lags. Frequency-resolved Fourier analysis allows us to identify two main components in the complex structure of the phase lags: the first component, characterised by a near-IR lag of a few seconds at low frequencies, is consistent with a combination of disc reprocessing and a magnetised hot flow; the second component is identified at high frequencies by a near-IR lag of ≈0.7 s. Given the similarities of this second component with the well-known constant optical/near-IR jet lag observed in other black hole transients, we tentatively interpret this feature as a signature of a longer-than-usual jet lag. We discuss the possible implications of measuring such a long jet lag in a radio-quiet black hole transient.