It is hard to say exactly why, but it could be that the contact is a bit unstable and parts might be undergoing rigid body motion.
You can activate the use soft spring or inertia relief option in the solver settings (Properties), and that might solve with a small force so you can see what happens to the parts. This might give you an idea which parts might need additional restraints (or perhaps it will show that the contact is not working).
Another way to find out is to use a bonded contact (assuming that you use no penetration) and see if that solves. If it does, then it means that with a frictional contact there is not enough restraint/stiffness to prevent it from being unstable. If this is the case apply some more restraints (e.g., from preventing the pin from spinning about its axis, or any other and as needed), or make sure that the contact works as intended.