I managed to get what I needed out of this part, but I still have no idea why solidworks is being such a pain, does anyone else?
I fixed the problem by changing my location refrence sketch from going to the outside surface, to the inside of the bore. From there I sketched colinear lines from the hole wizard sketch. The ends points of those construction lines are on the outer surface. Next all I had to do was place points at the end of those line segments.
Make sure you add an On Surface relation between the points and the outer surface. Without it, SW has no reference of which direction you want the hole to go.
The easiest way to do this is to place your hole wizard points on the surface first, then add relations to the control sketch.
Ahh, I see now. When I added that relation to my line segments I guess it also followed over to the points I placed coinsident to the ends of those segments.
I went back and tried this again, excpet with the original refrence sketch continuing all the way to the outer surface. I added that on surface relationship and all is well. Much easier and quicker to do it that way!
Another method would be to make one hole, then use a Circular Pattern to place the other two.