Over in the feature manager on the left for the hole table select the check box to show ANSI size. SW defaults to show decimal unless that is checked. You can setup your document templates to keep this toggled on. See Document Properties for that toggle.
Thanks Anna that's awesome. All this time it was my hole table settings and not my hole wizard or callout settings.