Use selection filters.
The behavior of this has definitely changed at some point in the recent past. It used to be that hovering over the end of the ballon would give you the 'A' indicator on the mouse pointer to let you know that clicking would select the end of the annotation leader. It doesn't do this consistently any more. Very frustrating.
When you try to move a notation, if you first click on it to highlight it and then hover near the end of the leader, a small green box will (or at least should) appear beside your cursor. When that box is visible if you will carefully click and drag then it should move the leader. If the box with the A is visible, then clicking and dragging will move the balloon or note without moving the leader. If the part symbol is there instead of the green box then you're about to move the drawing view.
If / when that doesn't work I usually just cut and paste the annotation, then position it where I want.
Besides Jum and Glenns comments, another method in SolidWorks 2013 and prior was to lock view position. This refocuses the drag activity to the balloon arrow tip. You can then unlock the drawing view.
Now this --->
In SolidWorks 2014, if you select the balloon first, and then select the arrow tip, the arrow tip moves, not the drawing view.