The simplest answer to this is that when hatred for a particular group or class or race is the obvious motive for an attack, that attack becomes, in effect, two crimes. The first is the offense itself. The second is the implicit threat that offense makes to other members of that group, class or race.In essence, the hate-crime punishment is not for having bad thoughts, but for issuing a threat of future violence – something we generally find acceptable to punish, even on libertarian grounds.
That second crime has new victims.
Consider an incident in which someone uses spray paint to deface the garage of a house into which a gay family has just moved.
The crime is vandalism, no matter what. But to argue against the idea of hate crimes is to argue that it shouldn't matter at all to the law whether the graffiti is a smiley face or some hostile, anti-gay slur.
The smiley face is a petty annoyance. The hateful slogan is, in effect, a threat to other gay people in the area -- they might be next.
Compare David Friedman’s argument for punitive damages in the case of “strategic torts,” that is, torts intended to send a threatening message to other potential victims. If the threat is successful, no future torts will need to be performed. As a result, the tortfeasor gets the benefit of committing many torts – by altering other people’s behavior – while only getting punished for one tort. Similarly, in the case of a hate crime, the hater can get punished for just one or a few crimes, possibly minor crimes, while getting the benefit of affecting many people in the threatened group.
My main concern with this position is that I think most people will tend to perceive hate crime laws as simply protecting the feelings or sensibilities of the protected groups, just as I did before I thought about it Zorn’s way. Indeed, that is often how the laws are justified even by their proponents. When they speak of harms to the protected groups, they often describe the harm in terms of “dignity” rather than threatened rights of person and property. Furthering this perception is the fact that hate crimes laws can be applied even in cases where the motive is not obvious – and thus could not plausibly constitute a viable threat to others.
And the perception matters, since laws like these can set precedents that pave the way for more laws based on (what are perceived to be) similar justifications. There is a slippery slope risk here.
UPDATE: Constant has a reaction over at Distributed Republic, with some good back-and-forth in the comments section (where I've chimed in).