Violence would be playing into their hands, yeah, but what are the alternatives? Plus, they themselves are not above violence -- you can't take the high road if they're actively trying to destroy said high road and you.
The very system you're arguing for --organizing communities, ballots, et...