I added a mcmaster shoulder screw into my assembly and it has a whole mess of reference dimensions that I can't turn off.
Turning off sketches doesn't make a difference.
RMB on Annotations at the top of the design tree and un-check "Show Feature Dimensions".
In the feature manager of the part like Glenn said, RC on annotations
Also make sure in the assembly this is also turned off.
Retrieving data ...