Changed Solver to Direct Sparse and seems to be working.
This sounds like a case of SPR 1008322 entitled "Nonlinear memory leak on some systems (only FFEPlus & LPDS solvers) - Commit Charge grows, shows SW Resource Monitor error 'Available system memory is low' ".
Your current solution is to use either the Direct Sparse or the Intel Direct sparse solver.
Version 2017 SP4.0 should include the fix.
Please contact your SOLIDWORKS reseller to be on the notification list for this SPR.