The unfortunate answer is you cannot control this. It is simply how SOLIDWORKS works. It cannot detect if that is a change you want to save or not. You can certainly submit an enhancement request if you'd like to see that behavior changed.
One workaround, if you have PDM, is to check Tools/Options/System Options/External References/"Don't prompt to save read-only referenced documents (discard changes)". Actually, you don't need PDM, you just need to keep your files as read-only except when you actually want to make a change.