How do you structure your files in EPDM?

Discussion created by Adam Lanners on Jan 28, 2011
Latest reply on Feb 2, 2011 by Rita Kanow

Like • Show 0 Likes0
Comment • 14

While attending the Birds-of-a-Feather session for EPDM at SWW11 and interesting topic was discussed about performance and how you structure your files within EPDM. The premise of the comment was to think strongly about how you index your files in EPDM.

For instance, we currently have a PDMWorkgroup vaults that are arrange by Project Number. While we migrate to EPDM we just assumed that we would keep this arrangement as we are planning to use EPDM to store all of our Project related documents. The problem is (as you may have experienced) that as you increase the number of files in a folder your navigating experience quickly suffers. Each time you select and open a folder you are querying the database server and collecting all the folder's children.

The suggestion was to index your files by their part number - in our case our filenames are an 8 digit partnumber (12345678.sldprt) and ensure good distribution by indexing on the last three digits. Thus:

My question is, how are you arranging your files in EPDM? What does SolidWorks recommend? What have you found to be the best way to optimize end user's navigating experience vs. performance.

I paraphased so if anyone from SWW needs to correct my interpretation, feel free.

Thanks,

Adam

Outcomes

14 Replies

Jeff Sweeney Jan 31, 2011 1:00 PM
I don't think SolidWorks will ever recommend a structure.

Most people divide their design data by products, projects or assemblies.

Honestly I have never seen anyone divide the structure by the last digits of the file name, usually if they divide them by file name, they will go by the first X digits of the file name. A nice advantage to this is that most of an assembly's files are all in the same location, and users don't have to change SaveAs directories as often.
Like • Show 0 Likes0
Actions
- Mike Ramsey @ Jeff Sweeney on Jan 31, 2011 3:16 PM
  Jeff Sweeney wrote:
  
  ......
  
  Honestly I have never seen anyone divide the structure by the last digits of the file name, usually if they divide them by file name, they will go by the first X digits of the file name. A nice advantage to this is that most of an assembly's files are all in the same location, and users don't have to change SaveAs directories as often.
  Actually there is a company that makes large green farming equipment that does just that. You read their part numbers from right to left.
  Like • Show 0 Likes0
  Actions
Lee CS Young Jan 31, 2011 2:09 PM
Adam Lanners wrote:

The problem is (as you may have experienced) that as you increase the number of files in a folder your navigating experience quickly suffers. Each time you select and open a folder you are querying the database server and collecting all the folder's children.

That may be true for PDMWorkgroup where the "database" is flat file based, but it's not for EPDM. Whether you have 50 files or 5000 files in a folder your access time will roughly be the same. SQL is much faster than flat files.

I wouldn't change your folder structure going from Workgroup to Enterprise. If it's not broke, don't fix it, right?
Like • Show 0 Likes0
Actions
- Jason Capriotti @ Lee CS Young on Jan 31, 2011 2:38 PM
  That's not 100% true Lee.......
  
  I have one folder with 1500 files in it....takes about 2 seconds to open and display. Another folder with 6100 files takes around 7 seconds.
  Then we have a folder containing 24,000 sub folders.....it take about 25 seconds to display this folder. This occurs every time you open it.
  
  The relationship seems to be about a second for every 1000 files/folders in the folder.
  
  Luckily, the folder we have that has the 25,000 sub-folders we don't have to access directly. It contains PDF files that a program adds automatically to the system from a watch folder on our network. The users only access the files via the search tool, never directly in the vault.
  Like • Show 0 Likes0
  Actions
  - Adam Lanners @ Jason Capriotti on Jan 31, 2011 2:57 PM
    I am experiencing the same as Jason only worse.
    
    I have a smaller project which contains 7180 files (two sub folders). It takes 30 seconds to open the folder the first time equating to roughly 240 files per second.
    
    I've uploaded a short video (60 secs) that demonstrates this procedure (if you have time take a look).
    
    I do realize that when I select the folder it is querying the SQL database which in theory should be a lot faster than the flat file data structure. I just haven't seen conclusive evidence proving that.
    
    Adam
    Attachments
    epdm explorer.avi27.4 MB
    Like • Show 0 Likes0
    Actions
  - Lee CS Young @ Jason Capriotti on Jan 31, 2011 3:00 PM
    Interesting. I haven't seen that kind of delay, but then again I don't have a folder with 24,000 sub folders in it.
    
    I wonder if you added some indexes and rebuilt others if it would speed up the access time.
    
    Maybe I'll do some testing.
    Like • Show 0 Likes0
    Actions
    - Ben Kalinowski @ Lee CS Young on Jan 31, 2011 5:04 PM
      I second the point that many files in a folder slows down PDM. We work with one folder (globally) which has almost 11,000 files in it and it definitely takes longer to access.
      
      I would say (and agree to Lee's point) SQL is not the bottle neck. IMHO, the problem is queuing up/matching up all that meta data across the archive servers is was takes time - especially if you're in a global enviroment. We have archive servers in Asia, Europe, US and soon South America. Each location has its own PDM Admin and we are working towards limiting our folder size to 2,000 files and we'll see how that works.
      
      The problem is really evident when you run SW and perform a "Save As". When you try this, the window opens and want to get all the metadata ready for all the files in the folder. When you have 11K files, you get slowed down.
      
      I would try and come up with a scheme which limits you to around 2,000 files. That's what we're gonna try.
      Like • Show 0 Likes0
      Actions
Lee CS Young Feb 1, 2011 9:11 AM
I stand corrected. Accessing a directory with 10,000 files in it took approximately 5-6 seconds to load up, and a few seconds longer to do a Save-As in SolidWorks. Once I broke the directory down to 1000 files, the access time was instantaneous.

That will teach me for opening my mouth before testing!
Like • Show 0 Likes0
Actions
- Ben Kalinowski @ Lee CS Young on Feb 1, 2011 9:15 AM
  I've got a follow on question. Does anyone think We'll have a problem with relations in assemblies after we shuffle all the files around? I expect SW will keep track of this, but moving around such a large chunk of data makes us nervous. We are testing this, but I'd like to know what others think....
  Like • Show 0 Likes0
  Actions
  - Jim Sculley @ Ben Kalinowski on Feb 1, 2011 1:19 PM
    Ben Kalinowski wrote:
    
    I've got a follow on question. Does anyone think We'll have a problem with relations in assemblies after we shuffle all the files around? I expect SW will keep track of this, but moving around such a large chunk of data makes us nervous. We are testing this, but I'd like to know what others think....
    EPDM will handle all of that as long as you do your original data import properly. The files should be added to the vault into folders that have the same structure as your current system, relative to their root directory. For example, before EPDM my company's data resided on four different network drives, which were mapped to drives P:, M:, N:, and T: on each user machine. The EPDM vault was named _EPDM. All the files were brought into the directory _EPDM\Admin\Incoming Files, maintaining the directory structure from the old file system, with the drive letter stripped off. So if a file was originally in P:\Oracle\401xxx, it was added to the vault in _EPDM\Admin\Incoming Files\Oracle\401xxx. Next the files were checked in to make sure no references still pointed to locations outside the vault (EPDM will generate an error if you try to check in something the doesn't reside in the vault). The files remained under the Incoming Files directory while I did extensive clean-up on them to adjust custom properties, sheet formats, revision tables and such. Once all that was done, the files were moved to their final destination elsewhere in the vault.
    
    Once they have all been checked in and EPDM has stored all the reference information in the database, you are free to move them around. Moving files and/or directories will not break references. Nor will renaming files. However, be sure that all the files have been checked in at least once. Just adding them to the vault is not sufficient.
    
    Jim S.
    Like • Show 0 Likes0
    Actions
    - Ben Kalinowski @ Jim Sculley on Feb 1, 2011 1:28 PM
      Thanks for the response. I think we'll be fine. I'll try and remember to post a response, could be a while though.
      Like • Show 0 Likes0
      Actions
    - Rita Kanow @ Jim Sculley on Feb 1, 2011 3:56 PM
      The company I work for has not started using PDM or Epdm yet, still in the talkign stage. Looking more at the EPDM. We will be dealing with the company expanding across the ocean. Right now our files are save in folders by file numbers, ex: part number folders, assembly number folders & type of parts like bearings, rollers, gearboxes, etc. So if I'm understanding you, we would check in all of these into the vault into folders by the same names, then we can move them to the new structure? For instance: project based, which is what we are thinking about doing. Then they will not loose their connection to the assemblies?
      
      Reading all these comments are a f=great help.
      
      Thank you everyone,
      Rita
      Like • Show 0 Likes0
      Actions
      - Jim Sculley @ Rita Kanow on Feb 2, 2011 11:23 AM
        Rita Kanow wrote:
        
        ...So if I'm understanding you, we would check in all of these into the vault into folders by the same names, then we can move them to the new structure? For instance: project based, which is what we are thinking about doing. Then they will not loose their connection to the assemblies?
        
        Reading all these comments are a f=great help.
        
        Thank you everyone,
        Rita
        Correct. If an assembly is inside the vault, and it has been checked in (along with all its referenced files), the EPDM system will ensure that the references stay intact as you move and/or rename the files inside the vault. The difficult part is when you first add the files to the vault. If your files are spread out all over the place in various directories on various network drives, you will want to set up the same structure relative to the vault root folder and then copy your files into the vault. If you do this correctly, all the references will still be intact. You can then check the files in to the vault. If you have made any mistakes, EPDM will let you know by pointing out that certain files wer not found or are still in directories outside the vault. You have to fix them all before continuing. Once any problems are resolved, you can finish the initial check-in. EPDM will look at all the reference information and add that data to the database. Now you are free to move and rename files without having to worry about breaking references.
        
        Jim S.
        Like • Show 0 Likes0
        Actions
        Rita Kanow @ Jim Sculley on Feb 2, 2011 2:19 PM
        Thank you Jim for your help.
        
        Much appreciated,
        Rita
        Like • Show 0 Likes0
        Actions

How do you structure your files in EPDM?

Outcomes

Attachments

Related Content

Recommended Content