It's not that bad. I doubt anybody of significance is calling neocons alt-right, for example. But I agree the term has been muddied somewhat. Too muddied to be useful? Probably worth a debate in a different thread.
It sounds kinda like you're wanting to ask people "When you say you want immigration reform do you want actual immigration reform or just fewer immigrants?" to which I would say actual reform.