Don't know if this will help...
That's exactly what my code looks like as well. My c# add-in code is below if you need the syntax for that language specifically. You can also adjust the font size in terms of font points rather than the physical height measurement by using TextFormat.CharHeightInPts.
tf = (TextFormat)swAnno.GetTextFormat(formatIndex); tf.CharHeightInPts = 6; tf.Bold = false; tf.Italic = false; tf.TypeFaceName = "Arial"; tf.CharSpacingFactor = 1.0; swAnno.SetTextFormat(formatIndex, false, tf);