Consent to constitutional governance varies by degrees and from person to person. Different levels of consent give different levels of justification. The hard problem is justifying the use of political violence against someone who expressly objects to it.
We respect the right to defend against a tort even absent the tortfeasor's express agreement, true. Everyone impliedly consents to the fundamental principles of tort law; they offer background rules for human conduct on which all socialized people depend. Our implied consent to tort law counts for a lot. It trumps merely hypothetical consent to the contrary, for instance. Some people offer fancy theories about that to which we would consent (usually something that benefits them). The implied protections of tort law trump such arrogant presumptions.
How, then, can a statist justify initiating coercion? Note that tort law boasts only a moderately powerful justification--one founded on implied consent. Express consent has greater power to justify. Therefore, the implied consent justifying tort law gives way before express consent, as when sparring partners tap fists before attacking each other. (Funny thing I've noticed about the BJJ studio I frequent: Lots of fighting; smiles all around.)
Have we expressly consented to constitutional governance? Some individuals undoubtedly have, such as those who have sworn oaths to uphold the Constitution. Perhaps run-of-the-mill citizens and residents show their implied consent to the authority of the U.S. federal government simply by not emigrating. And eloquent arguments have been made that the Constitution, if interpreted well, merits our hypothetical consent. See, e.g., Randy Barnett on the presumption of liberty.
Against those measures of consent, we hear polls suggesting that "consent of the governed" has fallen to a new low. We hear strong arguments that mere residency implies nothing about political allegiance, and philosophical claims that nobody would agree to a system of institutionalized coercion.
Consent weighs on both side of the scale measuring the justification of constitutional governance, both for and against. I cannot answer for anyone else about justifying the Constitution; you must answer for yourself. But with graduated consent theory, I offer you a way to tackle the question.
(With thanks to Sasha Volokh for stimulating discussion.)