Many addins i.e. dispatch, you can load in both version and I think it is a matter of proper preparation of addin.
We use some addin and like you we need it in 32 and 64 bit.
So trick is to load 32bit first and 64bit as second and addin works ok in both version.
It works !!