Are men carrying millions of unborn babies inside them all the time? If not, why?

I understand that you can’t have one without the other but most of the chromosomes does come from the sperm so my logic is not horribly invalid.

Technically you were in your father before you were in your mother. If your dad had sex with another woman, assuming you were the winner sperm, you’d still be born but have a different mother.