31 Replies Latest reply on Oct 7, 2016 4:57 PM by Liona Frank

    Duplicate files

    Alex Paulukiewicz

      Hi all,

      I am having a bit of a nightmare. I have recently joined a new company, due to their previous design engineer leaving. He worked in Solidworks in a quite peculiar way (in my opinion). Instead of having a library of parts all called by the part number in one location, read only and then inserting them into the new assembly, from the original location (thus keeping one file in one location), he copy and pasted each time the mass of components into the new folder. I want to do away with this and go back to having a library with only the single part in, but is there any easy way to sort all the duplicates out? If i just delete them all, and then keep one in the library i know this will mess the assemblies they are used in. Below is an image.

       

      Best regards

      Alex

        • Re: Duplicate files
          Michael Dekoning

          Alex,

          If they are all named the same, I think you could use the Update References function in PDM to resolve this.

            • Re: Duplicate files
              Ryan McVay

              Ouch..data management nightmare. But I can also understand why the person did what they did. It keeps all the project files in one location and you don't have to worry about other people changing "your" files. Especially if you lock your folder permissions down.

              • Re: Duplicate files
                Alex Paulukiewicz

                Thank you all for the replies so far, and yes it is a data management nightmare! I am not currently using a PDM server, (I do have SW coming out to do a demo of PDM to my superiors to hopefully show them the value of the software) but currently all filing is done on a server using windows explorer . (Admins, i couldn't see  anywhere to post none PDM, data management queries, apologies in advanced).

                I can see why he has done what he has, but it is not my preferred way of  working, as it leaves lots of files all over the place, at differing revision numbers and states, and when in an open system (such as windows explorer) where anybody can view and act on files it can (and is causing problems).

                I did think to delete all the duplicates, but then there will probably be thousands of differing parts duplicated (I.E. 100010 is an m10 screw, 100012 is m12 etc) and then just sorting the references out inside the assemblies. But this would take years as there is thousands of everything. Surely there has to be an easier way?

                Alex

                  • Re: Duplicate files
                    Wojciech Paterski

                    there is.

                    move all old project to archive folder, and then lock it, so it cant be changed anymore and start in fresh - unless with PDM that should make life easier

                    • Re: Duplicate files
                      Jeff Hamilton

                      Sounds very familiar. We have had Workgroup for several years, but it didn't mean everyone used it. We recently found over 72,000 Solidworks files in our primary network locations and about half of that were duplicates of exactly what you are describing. Some were duplicated 200 times!

                      We are in process of moving on to PDM Professional and I believe it will be well worth the investment. We will also take out the choice in using it as much as possible to avoid this mess in the future.

                      I'm not sure about PDM Standard, but you can ask tour reseller about the possibility of using their Data Doctor to assist will the file migration. It will identify where your files are and how old they are. You can choose how you want them migrated to the new vault, so you can have all locations be specified to use only the latest released version if you choose.

                      The archive folder is still a good option to lock down things as they are, even with it inside a PDM vault. Then you can take out things as they are needed to get a good fresh start without losing anything.

                      • Re: Duplicate files
                        Ryan McVay

                        Don't purchase EPDM just for data management! You will never get your ROI out of it. If you are looking at EPDM you need to look at automating your business processes and then data management is a secondary benefit. And please, please, don't think that EPDM is a recycling center. It's not. If your data is garbage it is going to be very difficult to structure properly inside of EPDM (or and PDM system for that matter).

                         

                        If you are looking for just simple data management then use network folders and lock them down with permissions.

                         

                        If you multiple files with the same name (whether they are copies or not) you will still have to validate the data to ensure that they are true duplicates- based more than on the "date modified" file property. This in itself can be a nightmare and goes back to my recycling center remark.

                         

                        Oh, and don't forget...you can't upgrade your CAD without upgrading your EPDM! Even SP require upgrades of both! So this means a lot of testing. And one last "gotcha". You CANNOT miss any maintenance payment on PDM. There is no back maintenance charges...it is buy it all over again. (Some VARs may argue this but I have been told this by several VARs.)

                          • Re: Duplicate files
                            Jeff Hamilton

                            Good point! In our case, we are looking to automate our processes. In which case EPDM is a good choice. I'm not sure I understand the recycling part though...

                            I am hesitant to recommend the network folder option, primarily because I've tried it and it's not a good long term solution. However it may be the only one open to him. I was thinking about one seat of Solidworks Professional that contained PDM Standard (formerly Workgroup), however after looking at the product matrix again, it does not contain the ability to manage file types other than Solidworks . So unfortunately there is no inexpensive option there. My error. It's a shame. Nice product.

                             

                            Thanks for the info on the EPDM maintenance. That was the first I had heard of it.

                              • Re: Duplicate files
                                Ryan McVay

                                Jeff Hamilton If you data is in poor shape you will have a difficult time importing the data into epdm (or any PDM). Consistent named variables, file properties, attributes, title block and part naming, etc. If these things aren't consistent then you will have a tough time. Don't expect a new system to fix these issues unless you are willing to invest heavily in preprocessing your data before it moves into EPDM- garbage in...garbage out..EPDM won't fix that on its own.

                                 

                                If you are dealing with a multi-CAD environment or multi-site situations there are more robust PLM systems to look at. EPDM is a decent tool but if you require multi-CAD I would shy away and look at the systems that have been designed to work in those types of environments.

                                  • Re: Duplicate files
                                    Jeff Hamilton

                                    We are predominately Solidworks with some legacy AutoCAD and Anvil (gag!). The Workgroup was utilized much like an old cinderblock room used for with paper drawings. Nothing but "production released" drawings went there. (Think 21st century technology used like it was the 1970s...)  Except for what I put in for project work, little else made it to the vault. While I setup everything for the use of custom properties, it was almost never used. Only the drawing itself was viewed as important. No descriptions in the parts, use of materials or anything, no naming consistency. Except for Manufacturing Engineering, the Solidworks BOM wasn't even used. So I'm expecting a lot of things that the software can't fix. I'm hoping that this will highlight the need for such things with management as well as doing it right the first time.

                                     

                                    So except for the "released" section of the existing vault, I am putting everything into legacy and bringing it over on a as needed basis. I also have a portable hard drive to copy everything "as is".

                                    But, Hey, it's a start in the right direction.

                                      • Re: Duplicate files
                                        Ryan McVay

                                        If you were to implement EPDM you will have to have a serious cultural change (attitude) as well. Don't let this aspect set you back- and it will if not implemented correctly. You need to get user buy-in if not you are buying and implementing an expensive tool that will not provide the ROI you are expecting. It would be like buying a high end entertainment system with 4 different video and audio inputs. If not shown the benefits people are going to use the on/off and volume control rocker button only.

                                         

                                        Here's something to consider. Data cards are the bane of the engineer/designers life. Why do I need to fill out a data card to create a new part! This is one of the main complaints. And if this card isn't filled out properly all your down-stream customers and processes are not going to work...but it has to be done initially by the part creator but doesn't benefit them in the least.

                                         

                                        If you are thinking EPDM or any PDM you need to look at different  workflows and possibly different vaults. Do you want a design vault and move data to a released vault? Do you want in-design and released processes in the same vault? How do you want to look for data? What is critical to the success of PDM? There are lots and lots of things to think about....and it should be done upfront...making changes afterwards is pretty hard to do and may require automation or custom-programming to clean up the older data already in the system. Planning, processes and approvals is critical! Don't get stuck with fingers point to you saying with staff saying that they can't find or work in the environment!

                                          • Re: Duplicate files
                                            Alex Paulukiewicz

                                            Data cards are great where i used to work (they had PDM and used it very effectively). They had everything set up in the data card so i fill it in when i make a new part, that in turn fills all the description boxes in the sheet format when the drawing is made. All the info in the data car could then be exported to sage for stock allocations etc. The problem is if you have many user is getting them all to fill the data in the same manner. One might type 60 x 60 x 3mm SHS x 1800mm Lg for example another  1800mm Lg SHS 60 x 60 x 3mm. I guess it comes down to having a procedure everybody understands and it happy with.

                                            • Re: Duplicate files
                                              Jeff Hamilton

                                              Unless someone kills it soon, EPDM is going to be implemented here. It's already purchased  and the workflows established per everyone's guidelines and out existing engineering specifications. Doubtless there will be some refinements, so we will be working it out in a "sandbox" vault.

                                              I did expect some resistance to the data cards, so as much as possible I added some little things to sweeten the deal, like filling out title block and release form information from them and automatically generate things like serial numbers for ENs and development part numbers. Take as much of the mundane work as possible out of it.

                                              With regard to a design vault and a release vault, this was something I had not considered. Our Workgroup vault was used only for released files. All in-process work was done over the network. The new vault had that release section of the old vault migrated to it, but also has a project section for new designs. I really didn't think I wanted them to be moving files around from place to place like they do over the network, so in some instances I removed the move permissions to the folder, particularly where it pertains to a specific workflow.

                                              What would be the advantage to having both a design or in-process vault and a release vault? Is there an automatic task that could perform the moving of the files after release?

                                                • Re: Duplicate files
                                                  Ryan McVay

                                                  I work at one location where all the pre-production files used alpha suffixes. There was a separate workflow for pre-production parts that went through a complete design review with higher engineers, product managers, mfg engineers, etc that needed to approve the drawings and models (each had separate workflows). Once the data was all approved it moved into a bin and awaited the SAP utilities to approve the part numbers. Once that was done the data is moved to Released workflow that uses numeric revisions. The files were moved to released vault and then utilized the global ECO process which was quite extensive. Released data was used for marketing materials and for service lifecycle management. Granted none of this was done with EPDM.

                                                  My current situation has a company with an engineering vault with easily over 750,000 CAD files that are not truly managed and the data cards are generic- literally generic using OTB EMPD vault. I'm in the process of putting together a more robust vault but it will require changes to our automation tools (DriveWorks and DriveWorks Live). We also have a second EPDM vault that is our document management system that is tied together with our CRM and Sales Configuration tools. (This is complicated system utilizing a not very well know components of EPDM, the item and item explorer).

                                                  We currently have automation tools that bring data into EPDM so I don't see any reason why moving data and associated meta-data over to a different vault would be difficult to do- It's more of a standard import process.

                                  • Re: Duplicate files
                                    Matthijs Ebbe

                                    Hi, would it help when you add your Library parts to a predifined folder?

                                     

                                    when you add that location in your settings to the "search paths" solidworks looks into that folder and changes the file location.

                                     

                                    there is one but.. the searchpaths cannot search precursively (hope ive spelled it right).

                                    that means that a structure like this wo'nt work:

                                     

                                    Mechanical/E-motors/ABB

                                    Mechanical/E-motors/siemens

                                    Mechanical/E-motors/rotor

                                     

                                    all parts must be in one folder and have a separate searchpath.

                                     

                                    hope this helps a bit

                                     

                                    regards Matthijs Ebbe

                                    • Re: Duplicate files
                                      Ryan Milotte

                                      You should be able to move everything to one folder then remove the duplicates, when you open the assembly you might have to redirect where to find the file, but if they are all in the same folder, then it should find it automatically. Good Luck!

                                        • Re: Duplicate files
                                          Ryan McVay

                                          Ryan Milotte That is an option but you may experience issues with performance. The more files to search through the slower the performance. EPDM may help but that all depends on how many files you are storing (versions, revisions) etc. The system still has to hit a database for archive folder location, name, and then still search a specific directory. The last time I looked there were only 15 main folders inside EPDM Archive structure then another ump-teen folders inside each of these.

                                            • Re: Duplicate files
                                              Jeff Hamilton

                                              Assuming you go with the network folder option with permissions, any thoughts on how you might keep all the duplicates from happening again from another location. My people had stuff squirreled away everywhere. If I locked down one, they'd find another. Anywhere but where they are supposed to....

                                                • Re: Duplicate files
                                                  Ryan McVay

                                                  Yes.

                                                   

                                                  1. Have design standards, procedures and best practices guide.

                                                  2. Have management approve and make it part of the review process.

                                                   

                                                  No, seriously.

                                                   

                                                  What needs to happen first and foremost is to have everyone understand that the data they create is NOT their data but the companies data. This takes away individual "ownership" of files and design. Having the staff understand that is imperative. Use something like a branding message as an example..you can only have one of those or your customers won't understand or be confused about the company. Having design standards, procedures and best practices allows you to build that consistent message/data.

                                                  Remember you are being paid to create solutions to problems and by storing, duplicating and using bad modeling and drafting practices is not solving problems but making more of them for the business. Plain and simple it is bad business.

                                                    • Re: Duplicate files
                                                      Jeff Hamilton

                                                      Absolutely!

                                                      I especially like the part about their individual "ownership" which is exactly how they view it.

                                                      Unfortunately, in my case, I am told to create the solutions, but in the past management is not around to back it up or give me the authority to. So in some cases it got ignored. Hopefully, now that they paid a big chunk of change to get EPDM, it will make a difference.

                                                      • Re: Duplicate files
                                                        Stephen Lapic

                                                        Agreed.  Drafting/Design Standards and SOP are just that - Standards.  This is what you are to do.  And when you write them up make sure you use the proper language.  Do NOT say words like should or even preferred or best way as these all imply that you can weasel around something and do it your own way.  You need to state things like "All files are to be ..." then there is no room for someone to interpret it some other way.  If there is an exception to the rule then make sure you put it into the standards.  But everyone needs to understand that those are the rules.

                                                          • Re: Duplicate files
                                                            Ryan McVay

                                                            Stephen Lapic You are close on your wording. But having participated on several global manufactures' CAD/PLM committees I can tell you that you need to be much more formal. We had to deal with vendors recreating legacy data or with other design partners and pulling in their data. We always supplied templates and in one case it was a complete company add-in utility. You need to use legal terms, even for your staff. If you are letting people go because they are not conforming to the data requirements of the job you still can't use the word "are." You need to use the word "shall" for those items or tasks that are required to be done to meet contractual needs.

                                                            Keeping your standards simple is a key. If you are controlling the color and layer of entities you are way overboard. You are much better served to control only CAD entities if you have automation tools that will utilize those entities. Title blocks that get auto-populated by the CAD tool or revision symbols that reside on a specific layer because you have a program that wipes all existing symbols and then allows you add new symbols for the current revision.

                                                            Examples:
                                                            All formed sheet metal part files shall consists of a single solid body.

                                                            All manufactured and purchased part files shall have all meta-data fields populated. See subsection x.xx.xx for meta-data identification requirements.

                                                            All title block formats shall reside on Layer 100.

                                                            All revision balloons or revision identification symbols shall reside on Layer 200.

                                                            • Re: Duplicate files
                                                              Jeff Hamilton

                                                              Good information. I have already been approached by a well meaning manager telling me that I should approach it as the "preferred" method, and that there may be times when you may have to go back to the old way in a pinch. It's hard enough to deal with stubborn users who are set in their ways, but things like that from a manager just undermines everything.

                                                              As discouraging as that is, you guys have actually reinforced the idea that my original approaches were the best ones, and that without manager support, I can do very little to improve anything.

                                                    • Re: Duplicate files
                                                      Jim Steinmeyer

                                                      Alex,

                                                           I feel your pain. A couple of jobs ago I came into a similar situation only compounded by the fact that after the files were copied they were changed, with no name change. Add to that that the company changed Engineering Managers on average of every 9 months. It seemed like each manager would attempt to clean things up by taking all the old files, creating an archive and saving it as "old files" or "really old files" or "really really old"... you get the picture. And after that they would just continue the same pattern. And the files from the Really really really old folders were still being used.

                                                           As Ryan pointed out a PDM will not solve any problems if there is not buy in by the staff. There will also be the problem in making sure that the duplicate files were not changed. Then there will be the problem of how to handle files that are not in the system but need to be used in the transition between implementing the PDM system and when everything is vetted and entered. I would expect this lag to take several days with the mess you have been handed. My boss expected everything to be perfect on Tuesday after implementing the vault on Monday. (wonder why they changed managers so often?)

                                                       

                                                      Good Luck

                                                        • Re: Duplicate files
                                                          Alex Paulukiewicz

                                                          Yes this sounds very familiar! I was digging more yesterday and really didn't like what i found (this happens alot) but i found two identical part numbered parts, but completely different models. i have also been finding so many models and components that are not fully defined i  believe it to be a mammoth task. Part of me knows it would be easiest just to completely start again afresh (having seen the poor standards before hand how can i ensure what is draw, is  drawn actually correct), and believe me I would be ecstatic if i was just left to reverse engineer, measure  and draw existing parts all day, but we don't have the time to do that.

                                                          One thing i think i do have on my side at the moment is that each department works on a separate server which no other department can view/work on. (Ex Design can't see what sales have quoted, manufacturing cant see when purchasing are expecting the parts, logistics cant see when manufacturing has made etc.) If everybody that is not on solidworks had reader/contributor seats then all documents can go onto the one server in the correct places. We are 9001 so in my mind surely this has to be beneficial of all documents being stored in the right place?    

                                                            • Re: Duplicate files
                                                              Ryan McVay

                                                              Alex Paulukiewicz Yes, technically you could control all those departments with EPDM. But I would advice you to first look at document management systems first. The simplest one would be using SharePoint. Having EPDM control that many different types of documents and those documents having all sorts of different workflows and datacards might be more than you want to do inside of EPDM.

                                                              You would, more than likely, need to have custom coding done inside of EPDM to automate your document processes. It is much more fiscally sound to control those types of documents using a more common/universal tool like SharePoint- you consulting fees and customization fees with be much less and you will have more functionality within SharePoint than the APIs inside of EPDM.

                                                              You have to remember that if you are installing EPDM that you are doing a business logic and business workflow analysis to determine if your current process is lean and to determine what needs to happen. This ALL needs to be defined before hand. To go and install the default setup and expect to dump all your data into it. It won't provide you any value and may actually set you back!

                                                          • Re: Duplicate files
                                                            Lukasz Rybacki

                                                            Hello,

                                                             

                                                            Like may other users pointed out I agree that EPDM like any other system works on the principle 'crap in, crap out'. Therefore everybody has to follow procedures and rules that are applied. In our case we have quite a small design team (5 CAD stations) and I volunteered to become admin, and be in charge of the whole import of legacy data (about 48k files prt, asm, drw). We set up rules before we even got the data into it, permissions, workflows, templates etc. Together with my collogue we spent last month on cleanning the data, and filling in the blanks in it, so that we have as best starting point as possible.

                                                            One of the biggest challenges was identifying all cases that look similar to yours. We Purchased a tool from SolidSolitions UK that helps you out with populating properties that you want to have in all files. We used it also to add extra flags that will help sort the legacy data on import, and push it through the legacy data workflow. As we had some duplicates and our Vault doesn't allow duplicates there are two ways of handling it.

                                                            1) You rename the files before import - you end-up with lost references.

                                                            2) You check duplicate names of files if they are the same parts, mark all duplicates before import, allow the Vault to have duplicates ONLY FOR THE TIME OF IMPORT. Import the data, search for marked duplicates, change references to point to one file, remove/rename copies where applicable, disallow duplicates.

                                                             

                                                            I know that this will only work if you have some files like that... if you are trying to face thousands of files like that... I think option 3) will be better... freeze the old data, and start with the minimum import, so that you can handle it from now on in a controlled environment.

                                                             

                                                            We decided that we are taking with us to PDM only files from projects from the last 4 years. the rest will become an archive.

                                                             

                                                            I hope this helps.

                                                             

                                                            Best Regards

                                                            Lukasz

                                                            • Re: Duplicate files
                                                              Jason Edelman

                                                              This is a good tool, does not fix issues, but will give you more insight. GoDataAnalysis - GoEngineer

                                                              • Re: Duplicate files
                                                                Chris Saller

                                                                Use some type of PDM, then train all users computer file management. This is difficult because most users want to handle files their own way. Not everyone is computer literate.

                                                                Training them how your new system works, and enforcing it, is the best way.

                                                                • Re: Duplicate files
                                                                  Liona Frank

                                                                  Thanks for the post. Please use Dulpicate files deleter,it is very simple to use. But make it sure to keep the important files in back up. .