We discuss the cosmological implications of nonlocal modifications of general relativity containing tensorial structures. Assuming the presence of standard radiation-and matter-dominated eras, we show that, except in very particular cases, the nonlocal terms contribute a rapidly growing energy density. These models therefore generically do not have a stable cosmological evolution.