I don't use that Replace Components often, but I believe you have the function of the dual windows confused. The main screen is showing the assembly with the new component and asking you to pick a new surface, edge, etc. for the Mate. You select the appropriate entity from there and then click on the next mate in the "Mate Entities" section of the PropertyManager.
The other window is just for reference to help you see which mate you're working with. It's showing you the corresponding surface, edge, etc. on the Part being replaced.
If you're selecting a surface from each window I can see why you're getting an error message. I won't argue with you that it's not intuitive, but it works fine when you use the right steps.
Just to clarify a little bit, when you are clicking the new surface to reestablish the mate, you want to select the surface on the new component that matches the surface of the old component show in the little window. You most likely should not be clicking on any parts other than the new one that is replacing.
While, I'd like it if the component view tools in the replace mate command were better tuned for the workflow, I have found the improvements in the replace component process commendable. It used to be that when a mating reference went missing, you had to re-apply for each mate that used it. Now, SW will repair in one shot every mate that had that missing reference. I found that if I'm consistent in my approach, I can get all of the mates fully resolved much faster than before. When you have to re-reference the mounting face of a PCB, you appreciate things like that.
The only thing that second window does is highlight the face on the superseded component that formed a particular mate, so there isn't any need to click inside of it. I agree, it often gets in the way and if you already have multiple document windows open, it can be a mess trying to sort through them to find this one preview window.
As for the other component, the list of mates is arranged based on the component that's bearing the reference, so avoiding the 'you picked the same component twice' nag is just a matter of working down that list systematically. It helps if you expand each item to see the mate type and number of occurrences that are being repaired for a given reference.
In any event, you should create an enhancement request in the customer portal and let SW know you find the experience unsatisfactory.