I'm not sure why performance degrades to that extent when the user is off-site. Perhaps, as it appears, it does have to do with the system determining which folders the user has rights to see.
Depending how closely you collaborate with the vendor, he could cache the files he needs on his machine and then work off-line until he needs to synchronize his changes in the vault.
The only other suggestion I can make is that you could create a folder for the contractor at the top level and share the files he needs access to in that folder. However, my advice is to only do this if all of your users understand that it is the same file that appears in different folders.