The factories are, beyond mercenary work (which also requires money to run), the source of the money available to spend. Not working on them doesn't mean you lose some time, it means you lose.
Given the primary defense is obscurity (admittedly that's not the best defense method, but it has been...