In this article, we develop a method for removing the phase drift induced by physically distorted object in terahertz time-domain reflection imaging (THz-TDRI). The proposed approach is defined as self-referenced, as it does not rely on any numerical parameter optimization nor extra instrumental components, and it is based on the unique manipulation of time-domain imaging data. In fact, we demonstrate that the problem can be solved assuming a linear contribution of the temporal shift induced by surface curvature. We illustrate how the self-referenced method is modeled and implemented, and we report the results obtained on two objects with different characteristics: a tilted and highly reflective surface, and a warped and heterogeneous surface. The proposed method demonstrates how to successfully remove the phase alterations induced on the reflected electric field, and how to repair the heavily corrupted images in the frequency-domain.