By going into Document properties, I was able to make them bigger by changing the Dimension Font bigger.
This is the font for all dimensions in that document.
None of the items under "Dimensions" will change the datum target font. Only the general Dimensions Font.
Edit: I don't think that the general Dimensions font is the right place for this. There should be a better way.