I think it comes down to a difference in how I view human nature, ethics, and politics.
First, libertarians mistakenly believe that there is a "right to do wrong"; that is to say, they believe you have a right to violate the moral law so long as it does not violate the rights of others. But if the moral law gave people the right to do what it itself said was intrinsically immoral, it would undermine itself, so such a right could not exist. But libertarian philosophy seems to be committed to the proposition that everyone has a "right to do wrong." So, libertarianism would undermine any notion of transcendent moral law as traditionalists would understand it.
Second, libertarians prioritize liberty as the highest political end, if not the highest end, period. But abstract liberty cannot exist outside order. People in a society in which everyone is violent and treacherous cannot be free in the libertarian sense. Order must first be established before we can start debating how much freedom to give individuals. The radical individualism assumed by libertarianism is destructive to order because it leads to the dissolution to civic bonds that promote trust between individuals, leading them to seek to reestablish order by any means necessary. It was no surprise that totalitarianism was the allergic reaction by people to the atomizing forces of individualism.
Third, libertarians seem to believe that order can arise spontaneously by the "invisible hand" mechanism. In other words, it is self-interest and market transactions that holds civil society together. But in order for the market to exist, certain rules must govern the market. Now, in order for you to see yourself as being bound by the rules of a society, you must first see yourself as part of that society, bound together by shared language, territory, culture, and history. Religious, ethnic, and cultural ties and loyalties run far deeper than considerations of abstract right and rational self-interest.
Fourth, libertarians have this idea that the state is a man-made invention arising from human convention or social contract. Therefore, it's plausible to suggest that the state is some "great oppressor" whose pretentions towards the common good are always and everywhere smokescreen for the self-interest of public officials. This is simply false. The state is a natural institution that is necessary for the fulfillment of human nature and the growth of human civilization. Human beings are political animals; by engaging in civil society, human beings can consider the common good rather than his own individual good and can become more self-actualized by being a good citizen. Now, this isn't to say that some government official could become corrupt, but it's because the individuals running the state have no sense of
noblesse oblige, no concept of government as a sacred trust vouchsafed to men by God for the public interest, that they become corrupt and subordinate their duty to the common good in favor of pursuing their individual passion and interests.
Fifth, libertarians believe that we do not have any positive obligations that we do not explicitly consent to. But, in fact, many such obligations are incumbent upon us: to our parents, children, and other kin; to our country; and to members of society who are in extreme need. Each of us plays an irreplaceable role in the overall social body, and failing to recognize that role will invariably lead to the society's slow decline.
These are the five main reasons that I am not a libertarian, though they aren't the only ones. I can point to
the inadequacy of the non-aggression principle, the
indeterminacy of the so-called "self-ownership axiom," the
intellectual flatulence that is "libertarian neutrality," inadequacy of methodological individualism to explain certain phenomena like tribalism,
the unwillingness, if not inability of libertarianism to deal with the problem of corporate power, and
the arrogance with which libertarians
make unjustified assertions about reality. I mean, I've been hearing for
years how libertarians have the only internally consistent moral system and this justifies dismissing all other political philosophies. So forgive me if I am a bit tired of hearing the same libertarian talking points with regards to social conservative legislation being trodden out time and again even as the Left succeeds in cramming down its agenda onto us. Sorry, been there, done that. Didn't work out the first time 'round.
I am assuming that the party in power that will enact my policies conforms to my moral and ethical beliefs, yes. I think you'll agree with me that no mainstream political parties today will even consider censorship of pornography, and I don't see a reactionary party coming to power in a democracy because I believe democracy is in many ways rigged against reactionaries. Liberal egalitarianism is only tolerant towards those that accept its premises, after all. Dissident reactionaries will be exiled into the political wilderness. Therefore, there is no way to practically implement the policies I am describing unless the status quo radically shifts and makes reactionaries a viable political force.
Now, assuming a right-wing party does (somehow) come to power, I see no reason why it couldn't force its views on a wider public that disagrees with them. I mean, the Left was able to force same-sex "marriage" and mass immigration upon an unwilling public without difficulty. I see no reason why a reactionary government in the future would forgo (ab)using the same institutional power to achieve the opposite ends.