Personally, what turned me against Ulric for good was his introduction scene in his castle. "Damn the Jarls! And damn the Moot!"
Winning the duel with shouts, I can accept that. I mean, as the Dragonborn, I use shouts to win duels as well, so it would be hypocritical of me to denounce its uses...