It is best to save out individual frames then use a program like QuickTime Pro to compile your video. This is how it is done by those in the CGI field.
If you have an issue you do not need to re-render the whole animation again, only the affected frames. You can do post work in PhotoShop and After Effects. You will have way more choices on compression codecs, etc.
I've only done a few Render Animations and on a computer with way lower specs than yours!
It's "normal" to see the CPU running at 100% and your RAM is effected by what you are trying to render, reflective materials, size etc, but that should only effect the time it takes (per render) and not freeze
I've always used Microsoft AVI as the output without a problem.
Have you tried just rendering a few seconds at a time? To see what happens. Set Time Range 0 - 5, then 5 to 10 etc (These can then be put together even in Windows Live Movie Maker
I don't know if this simple post will help but it does show some settings!