What’s bad about being an empire or have colonies?

What’s bad for the enslaved, killed, and culturally conquered or what’s bad for the “hey we got slaves, people we can force our beliefs on, and free corpse covered land for our use?”