Do you actually need to model the pins to transfer the force to the plate or could you just accomplish this with a force or bearing load? I ask because obviously if you can get rid of the pins then you will not need the No Penetration contact and solve times would decrease dramatically. If the load from the pins is in plane with the hole in the plate using a bearing load can often work well to replicate the force.
Sounds like you have a stress concentration so the more refined you make your mesh the higher the stress is going to get. Unfortunately you do not have a lot of tools at your disposal to manage the mesh besides refining it but getting rid of the pins if you can could help. Also, have you tried using the curvature based mesher? This will concentrate the mesh around the holes automatically and may vary the mesh enough to help get rid of the concentrations. Be aware though that this mesher can gobble up more resources and increases solve times if you get it to refined.
as mentioned above, you should be able to apply a bearing load which should simulate a pin bearing against a hole and significantly reduce solve time and you will get better results. contact should be avoided unless there is not other way to model. you can also do this in a 2d model but not sure if SW sim provides this capability. this would solve the fastester and you can crank up the mnesh density w/o drastically increasing solve times.
a rule of thumb is the center of hole should be located no closer than of 2 diameters from the edge of the plate .(more for sheetmetal) w/o risk of bearing failure.