The biggest problem is that government doesn't exist to legislate morality, and it shouldn't. Adultery is shitty, sure, but is none of the goernment's business, much like most of the times people are shitty people. Or why stop at adultery? Why not legislate forgetting you spouses birthday?