There was no rule against shouts, it might be orthodox but no one disputes that the duel was unfair. The Stormcloaks say it was a perfectly just duel that showed Toryg weak. The Empire says any duels to usurp authority are invalid no matter what.
I said it was against the spirit of the rules, not the letter. Yes, the rules did not prohibit the Shouts. But, AFAIK, they did prohibit magic as the idea was to be a duel of arms. Shouts are not magic per se, but the use of them clearly violates the spirit of the idea of a duel of arms.
To say they are within the rules is to use the same logic that undergirds all those kids sports movies where they include a dog as part of their team. "Oh, the rules don't explicitly prohibit dogs from being players" is a cop out. The spirit of the rules was clearly violated, and no amount of Stormcloak apologism can change that. It's one of the reasons I have slowly shifted to disliking the Stormcloaks. Their entire justification for rebellion is, functionally, premised on splitting hairs to justify rebelling.
As to your reasons for expelling Argonians and Khajit, while I understand the logic, I think that it actually shows just how weak a ruler he is. If he cannot maintain law and order between two minority groups living in his capital by means other than expelling one, it shows poor management and priorities for the ruler. It also shows a penchant for collective punishments, as what you're saying is, in effect, he's punishing the Argonians for the actions (or potential actions) of a few. Again, that's not justice, nor does it show a man who's concerned with justice or proper leadership. Rather it showcases a man who is basically using the easiest possible means to solve a problem regardless of the collateral damage or the justice of the matter.
Which also means he's going to end up making things worse for Skyrim in the long run. What sacrifices will he make? What deals will he be willing to broker? Ulfric is designed to appeal to the people who like "hard men making hard decisions", but those types rarely make actual good rulers, rather, they usually become nothing more than tyrants with oceans of blood on their hands.