Any time you add a 'connector', be it bolt, contact set, etc., it is going to increase your computation time. There are some threads here in the forum about how to mitigate that effect but not to eliminate it.
However, I agree that adding springs may be a very good choice to more realistically model the actual system. Another might be to model the hanger and place a non-penetration surface with some friction between the pipe and hanger.
Can you show a pic of your set up?