Try purging the DXF file before importing. Also did you tried copy and paste option??
As Deepak suggested, anything you can do onthe AutoCAD side to clean up the drawing will help. Unfortunately, you are still quite likely to get large file sizes and unwieldy files in SolidWorks. I get injection mold drawings in .dxf format and all the lines and arcs just bog SolidWorks down. Many of them are just painfully slow to work with. I've tried working with them in Draftsight, which seems to be faster, but I dislike the AutoCAD interface and usually end up going back to SolidWorks.