We study the cosmology of bimetric theory with a composite matter coupling. We find two possible branches of background evolution. We investigate the question of stability of cosmological perturbations. For the tensor and vector perturbations, we derive conditions on the absence of ghost and gradient instabilities. For the scalar modes, we obtain conditions for avoiding ghost degrees. In the first branch, we find that one of the scalar modes becomes a ghost at the late stages of the evolution. Conversely, this problem can be avoided in the second branch. However, we also find that the constraint for the second branch prevents the doubly coupled matter fields from being the standard ingredients of cosmology. We thus conclude that a realistic and stable cosmological model requires additional minimally coupled matter fields.