What is the dimension of a smooth family of complex Hadamard matrices including the Fourier matrix? We address this problem with a power series expansion. Studying all dimensions up to 100 we find that the first order result is misleading unless the dimension is 6, or a power of a prime. In general the answer depends critically on the prime number decomposition of the dimension. Our results suggest that a general theory is possible. We discuss the case of dimension 12 in detail, and argue that the solution consists of two 13-dimensional families intersecting in a previously known 9-dimensional family. A precise conjecture for all dimensions equal to a prime times another prime squared is formulated.