When is it totally justified to turn something from a verbal confrontation to a physical one?

Never.

If the other person/people start throwing hands first, you have every right to defend yourself/fight back/retain them, whichever side of the argument you’re on.

However, any intelligent conversation/debate/conflict ideally, should never result in violence.

Everyone is just going to have to sit down and talk it through like mature adults eventually, anyway.