How do I remove appearances defined in an assembly without affecting the sub assembly or part file appearances?

When looking at the assembly I see all appearances that were defined in part files or at the assembly level.

If I remove an appearance I could be changing the part or sub assembly.

I don't want to modify the parts or sub assemblies.

I have some parts without appearances at the part level. But they have color in the assembly.  I assume the color was defined in the assembly.

Other parts have color at the part level.  If I delete the color when in the assembly the part gets changed.

How do I know which is which when looking at the assembly?