We investigate the impact of each ingredient in the employed physical data model on the Bayesian forward inference of initial conditions from biased tracers at the field level. Specifically, we use dark matter halos in a given cosmological simulation volume as tracers of the underlying matter density field. We study the effect of tracer density, grid resolution, gravity model, bias model and likelihood on the inferred initial conditions. We find that the cross-correlation coefficient between true and inferred phases reacts weakly to all ingredients above, and is well predicted by the theoretical expectation derived from a Gaussian model on a broad range of scales. The bias in the amplitude of the inferred initial conditions, on the other hand, depends strongly on the bias model and the likelihood. We conclude that the bias model and likelihood hold the key to an unbiased cosmological inference. Together they must keep the systematics - which arise from the sub-grid physics that are marginalized over - under control in order to obtain an unbiased inference.