I just figured it out. Had to have the Customize window open before I could drag the icon away.
you could also remove buttons from the View (Heads-up) by pressing the "Alt" key and dragging the button off into the Graphics area.
This will work even without the Customize dialog being up. This works for removing buttons from the Command Manager as well as toolbars.