I already access the iedmfile5 object, and try to cast it to IEdmEnumeratorCustomReference7 then meet this error.
That cast should indeed work according to this example:
2017 SOLIDWORKS API Help - Access Custom File References Example (C#)
Are you sure the itemBOM object is of type IEdmFile5? That's the only reason I can think of why it wouldn't work.
Have you found the solution?
The file5 object for the parent is pulled from IEDMvault7 not IEDMvault5. Look for the vault7.getfilefrompath in the example.
Retrieving data ...