It doesn’t happen because people instinctually express contempt on their face when they feel it. It is one of the famous seven involuntary facial expressions in response to emotion. Sure there are rare people out there who do not show any facial expressions, but they’re so rare; they are also so weird, they probably wouldn’t have a real opportunity to pull it off, because they would creep people out long before they’d have the opportunity to do know someone well enough to try to pull off a “good” backhanded complement.
Now, there may be people who are bad at perceiving contempt on other people’s faces. But that is their own misjudgment.