Context. A strong toroidal field can exist in form of a magnetic layer in the overshoot region below the solar convection zone. This motivates a more detailed study of the magnetic buoyancy instability with rotation. Aims: We calculate the α effect due to helical motions caused by an unstable magnetic layer in a rotating density-stratified system with angular velocity Ω making an angle θ with the vertical. We also study the dependence of the α effect on θ and the strength of the initial magnetic field. Methods: We carry out three-dimensional hydromagnetic simulations in Cartesian geometry. A turbulent electromotive force (EMF) due to the correlations of the small scale velocity and magnetic field is generated. We use the test-field method to calculate the transport coefficients of the inhomogeneous turbulence produced by the layer. Results: We show that the growth rate of the instability and the twist of the magnetic field vary monotonically with the ratio of thermal conductivity to magnetic diffusivity. The resulting α effect is non-uniform and increases with the strength of the initial magnetic field. It is thus an example of an "anti-quenched" α effect. The α effect is also nonlocal, i.e. scale dependent, requiring around 8-16 Fourier modes to reconstruct the actual mean EMF based on the actual mean field.