Create a dummy Windows directory folder someplace on your computer. Maybe call it Junk...
Close your session of SolidWorks and re-open to clean up anything that may be loaded in RAM.
Open the assembly you want to clean up in SolidWorks.
Select and Move all the files in your assembly folder into the dummy folder.
You will get a dialog box from Windows that says it can't move all the files. Acknowledge and let Windows move the files it can (the ones not open in SolidWorks).
Your Junk folder will contain all the files not needed by your current assembly. Delete when you are comfortable that you do not need the files anymore.
And if you hate creating dummy Junk folder, create dummy work folder, Pack & Go the assembly with drawings to new folder & delete original folder completely & this way you can get read of unwanted drawings as well. ( In Anna's solution you need to do it manually for drawings)