Groundwater is an important resource for a large share of the global population and economies. Although groundwater dependence in most sub-Saharan African countries is relatively low at the national level, localized overexploitation is occurring, leading to a decline in groundwater levels and quality deterioration. Currently, the sustainable and equitable governance of groundwater, both through promotion and regulation, is turning out to be a key challenge in many sub-Saharan African countries. This paper uses case studies of urban groundwater governance in Arusha, and rural groundwater development in the Pangani basin, to analyse how the current policy and regulation inadvertently creates spaces for asymmetric access to (good quality) groundwater resources in Tanzania. It shows how the groundwater landscape is evolving into a situation where small users (farmers and households) rely on springs and shallow wells, while large users (commercial users and urban water authorities) are encouraged to sink deep boreholes. Amidst a lack of knowledge and enforcing capacity, exacerbated by different priorities among government actors, the water access rights of shallow well and spring users are being threatened by increased groundwater exploitation. Hence, the current groundwater policy and institutional setup do not only empower larger actors to gain disproportionate access to the groundwater resources, but presents this as a benefit for small users whose water security will supposedly increase.