The answer should be obvious, because refusing to fight someone suggests to the mob that you're afraid to fight them, thus diminishing your status, particularly if it's based on being more masculine than your opponent.
That said, the only person who would ever accept a duel would be an idiot...