That isn't really a very easy question to answer. It comes down to hardware vs. number of files vs. size of files vs. what you consider "affecting performance".
A 64 bit server can handle huge databases, it is hard to believe there is a company out there that would have enough files to fill it.
Perhaps we can help you if we approach the problem from a different angle. You have a number in mind?
The largest file I have ever seen "attempted & failed" to store in EPDM is 2gb. It was a hard drive ISO image for production's use. They build using this image. They ended up storing it on a secured server elsewhere after I discovered it.
It would load fine but the user could never get a good "Get latest version" or check it out. Probably not a wise idea.
In 2007 the more folders inside EPDM the slower our system got too. But this has been fixed.