Where are the three main planes located?
Unless you mean rotated the current view using the middle mouse button? then where you click on the screen matters.
Sketch a reference line (or point) where you want the pivot to be. Then mate to that reference geometry. You can then move the reference geometry as needed.