So I have noticed when I use a Hole Callout annotation I can change the tolerancing in a drawing and it affects the model.
I had assumed the Hole Callout was a smart dimension or driven dimension.
Is it really a model item?
For example I added a basic tolerance to these depths...
...and was surprised to see that in my model!
(Ignore the blue cosmetic thread dim, they never come out right anyway.)