Disease,Corruption, idiots being in charge, natural disasters, Roman citizens not wanting to serve in the Legions leading to a lot of mercenaries being hired to take their place, the Huns, and a crapload of civil wars is just the beginning of a very very long list