I know it's bad form to answer your own question, but here I go anyway:
I still can't find a way to present our users an option to delete or cancel, but I can prevent them from being able to select the axes in the first place:
By setting the axes property 'HitTest' to 'off' selection and mouse events are effectively disabled. You can still right-click and get the properties options in a context menu, so that functionality is preserved.
Users can still interact with children of the axes and annotations.
All-in-all, it's a solution I'm happy with.
hs.ax.HitTest = 'off'