We consider a quantum system such as a qubit, interacting with a bath of fermions as in the Frohlich polaron model. The interaction Hamiltonian is thus linear in the system variable and quadratic in the fermions. Using the recently developed extension of Feynman-Vernon theory to nonharmonic baths we evaluate quadratic and the quartic terms in the influence action. We find that for this model the quartic term vanish by symmetry arguments. Although the influence of the bath on the system is of the same form as from bosonic harmonic oscillators up to effects to sixth order in the system-bath interaction, the temperature dependence is nevertheless rather different, unless rather contrived models are considered.