We study the non-relativistic expansion of general relativity coupled to matter. This is done by expanding the metric and matter fields analytically in powers of 1/c2 where c is the speed of light. In order to perform this expansion it is shown to be very convenient to rewrite general relativity in terms of a timelike vielbein and a spatial metric. This expansion can be performed covariantly and off shell. We study the expansion of the Einstein-Hilbert action up to next-to-next-to-leading order. We couple this to different forms of matter: point particles, perfect fluids, scalar fields (including an off-shell derivation of the Schrödinger-Newton equation) and electrodynamics (both its electric and magnetic limits). We find that the role of matter is crucial in order to understand the properties of the Newton-Cartan geometry that emerges from the expansion of the metric. It turns out to be the matter that decides what type of clock form is allowed, i.e. whether we have absolute time or a global foliation of constant time hypersurfaces. We end by studying a variety of solutions of non-relativistic gravity coupled to perfect fluids. This includes the Schwarzschild geometry, the Tolman-Oppenheimer-Volkoff solution for a fluid star, the FLRW cosmological solutions and anti-de Sitter spacetimes.