Realistically speaking, when is it too late to shift careers?

It really depends on what career you want to switch to.

If are in your fifties and suddenly got a dream of flying large airliners on intercontinental routes, then unfortunately you are too late.

Want to open your own pub? Sure, go for it.