Since the release of the Gravitational Wave Transient Catalogue GWTC-2.1 by the LIGO-Virgo collaboration, sub-threshold gravitational wave (GW) candidates are publicly available. They are expected to be released in real-time as well, in the upcoming O4 run. Using these GW candidates for multi-messenger studies complement the ongoing efforts to identify neutrino counterparts to GW events. This in turn, allows us to schedule electromagnetic follow-up searches more efficiently. However, the definition and criteria for sub-threshold candidates are pretty flexible. Finding a multi-messenger counterpart via archival studies for these candidates will help to set up strong bounds on the GW parameters which are useful for defining a GW signal as sub-threshold, thereby increasing their significance for scheduling follow-up searches. Here, we present the current status of this ongoing work with the IceCube Neutrino Observatory. We perform a selection of the sub-threshold GW candidates from GWTC-2.1 and conduct an archival search for sub-TeV neutrino counterparts detected by the dense infill array of the IceCube Neutrino Observatory, known as "DeepCore". For this, an Unbinned Maximum Likelihood (UML) method is used. We report the 90% C.L. sensitivities of this sub-TeV neutrino dataset for each selected sub-threshold GW candidate, considering the spatial and temporal correlation between the GW and neutrino events within a 1000 s time window.