Just say “no” when it’s not something you’re willing to budge on something that is not life threatening. If you give someone an excuse, you give them an avenue to argue either that it’s not a good enough reason or that the issue can be resolved.
“No” is a complete sentence.