What purpose would an actual Olympic boycott do anyway? Nothing, except ruin the dreams of athletes who dedicated their lives to go to the Olympics.
The Soviet Union was terrible, but Carter pulling us out of the Olympics was one of the biggest asshole moves of his presidency.