That's unusual. It sounds like you're trying to run the setup in compatibility mode or you're trying to install it in the Virtual PC, Windows XP Mode.
I don't have XP mode installed, so I can't tell if it can assert itself as the default installation environment. But it's probably worth a cursory look at your machine to figure out why.
The alternative could be that this dialog box is showing up as part of another installation error. I would look at the installation logs (on my machine they're in c:\USERS\<Account>.<Domain>\appdata\Roaming\SolidWorks\Installation Logs\) and make sure that's what caused the installation to error out.
I'm not really sure I understand the issue you're having with licensing. Licensing is a seperate deal from installation. the installation wizard takes the serial number you entered (for this or a previous release) and checks your entitlement over the internet when you first launch the installer and then activates the license once you've completed the setup process. , So, compatibility doesn't really factor into it. If you're having a licensing issue, call your VAR (or have your IT manager call your VAR) and they should be able to straighten you out right away.
If your problems persist, post a screen shot of the error message you're getting and we'll see if we can help you narrow down the cause.