SN 2002cx-like Type Ia supernovae (also known as SNe Iax) represent one of the most numerous peculiar SN classes. They differ from normal SNe Ia by having fainter peak magnitudes, faster decline rates and lower photospheric velocities, displaying a wide diversity in these properties. We present both integral-field and long-slit visual-wavelength spectroscopy of the host galaxies and explosion sites of SNe Iax to provide constraints on their progenitor formation scenarios. The SN Iax explosion-site metallicity distribution is similar to that of core-collapse SNe and metal poor compared to either normal SNe Ia or SN 1991T-like events. Fainter members, speculated to form distinctly from brighter SN Iax, are found at a range of metallicities, extending to very metal poor environments. Although the SN Iax explosion-sites' ages and star formation rates are comparatively older and less intense than the distribution of star-forming regions across their host galaxies, we confirm the presence of young stellar populations (SPs) at explosion environments for most SNe Iax, expanded here to a larger sample. Ages of the young SPs (several x 10(7) to 10(8) yr) are consistent with predictions for young thermonuclear and electron-capture SN progenitors. The lack of extremely young SPs at the explosion sites disfavours very massive progenitors such as Wolf-Rayet explosions with significant fallback. We find weak ionized gas in the only SN Iax host without obvious signs of star formation. The source of the ionization remains ambiguous but appears unlikely to be mainly due to young, massive stars.