5 Replies Latest reply on Mar 31, 2016 4:48 PM by Rodney Michels

    Mass Restore of EPDM Cold Stored Files

    Josiah Sansone

      If you've ever had an EPDM vault which either intentionally or accidentally cold stored a lot of files, you know it can be a pain to get them back.  EPDM includes no functionality for restoring multiple Cold-Stored files at once.  This can be an issue especially in a replicated environment if files are cold-stored on one archive before being replicated, then you end up with files that cannot replicate.  So, I came up with a solution.  In the spirit of sharing, here is a Powershell script I wrote which will perform a mass restore of any cold storage archive or archives.  Note that you need to customize lines 3 and 5 with your cold storage locations and your archive path, and you will also need to clear all pertinent rows from the ColdStorage table in SQL once you've run this script. I will discuss that more below.  I have used this twice and it has worked flawlessly both times.

      NOTE: USE AT YOUR OWN RISK AND BACKUP BACKUP BACKUP and TEST TEST TEST before running this on a production vault.  This is only intended for users with a very strong knowledge of EPDM, SQL, and Powershell.  This is informational only and I take no responsibility for anyone using this on their own vault without proper testing and validation.

       

      #BEGINSCRIPT

       

      $ColdStores = @()

      #Add all cold storage locations to $Coldstores as array below, like "C:\VaultNameColdStored\ColdStoreSchemaName\Date"

      $ColdStores = @("Cold Storage Location 1","Cold Storage Location 2","Cold Storage Location 3")

      #Set $ArchivePath to the archive root of the vault to which files are being restored - include trailing backslash!

      $ArchivePath = "Vault Archive Path Here"

      Foreach ($Store in $ColdStores)

      {

          $dir = dir $store | ? {$_.PSISContainer}

          foreach ($d in $dir)

              {

              $name = $d.Fullname

              $folder = $d.name

              $destPath = $ArchivePath + $folder[-1] + "\" + $folder + "\"

              $indexpath = $destpath + "index.xml"

              IF (Test-Path $indexpath)

                  {   

                  [xml]$xmlfile = Get-Content $indexpath

                  $Extension = [System.IO.Path]::GetExtension($xmlfile.index.filename)

                  (Get-Content $indexpath) -replace " status=`"archived`"", "" | Set-Content $indexpath

                  $files = Get-ChildItem $name | where {! $_.PSISContainer}

                  foreach ($file in $files)

                      {

                      $dest = $destpath + $file.name + $Extension

                      Write-Host "Copying $($file.name)$($Extension)"

                      $file | Copy-Item -Destination $dest -Force

                      }

                  }

              }

      }

       

      #ENDSCRIPT

       

      Paste the above script into a PS1 file and customize the two lines for the cold storage path and the vault archive path.  Again, backup and test thoroughly before running this. Once it's run, you can delete all rows from the dbo.ColdStorage table in SQL for the Cold Storage media which you restored.  If you did not restore all Cold Storage locations, only delete the rows for the Cold Storage Media that you restored!

      Please feel free to ask any questions.

      Cheers,

      Josiah

        • Re: Mass Restore of EPDM Cold Stored Files
          Rodney Michels

          Josiah, I have been looking for a way to complete something like this.  I don't understand Powershell.  Do you know of a conversion tool I could use, to convert this to vb.net?

            • Re: Mass Restore of EPDM Cold Stored Files
              Josiah Sansone

              I don't, sorry.  Maybe someone else can translate into VB.  However Powershell is really easy to use, and you shouldn't have to modify much in this to make it work.  You can edit the script in Notepad++ and save it as a .ps1 file.  Then you can just run it from either powershell or command prompt by typing "powershell -f "path to ps1 file"

              Just make sure the Powershell window or CMD window is run with administrator privilege, and don't forget to test.

              Josiah

                • Re: Mass Restore of EPDM Cold Stored Files
                  Rodney Michels

                  I am building it myself.  I'm about 75% completed.  Just wanted to say thanks for posting the idea, because I have been wanting to do this for more than a year.

                   

                  Thanks,

                  Rodney

                    • Re: Mass Restore of EPDM Cold Stored Files
                      Josiah Sansone

                      Cool,

                      I was going to post a breakdown of what the script does:

                      $ColdStores = @("Cold Storage Location 1","Cold Storage Location 2","Cold Storage Location 3")

                      ^^Here I am making an array of each cold storage location to loop through them

                      #Set $ArchivePath to the archive root of the vault to which files are being restored - include trailing backslash!

                      $ArchivePath = "Vault Archive Path Here"

                      ^^Setting the root destination path for the files being moved back into the archive

                      Foreach ($Store in $ColdStores)

                      ^^start with a loop through each cold storage location

                      {

                          $dir = dir $store | ? {$_.PSISContainer}

                           ^^getting a list of all directories in the current cold storage location I am executing on, and assigning it to the $dir variable

                          foreach ($d in $dir)

                           ^^looping through each directory in the current cold storage location

                              {

                              $name = $d.Fullname

                                ^^setting the $name variable to the full path of the current directory being executed on

                              $folder = $d.name

                                ^^setting the $folder variable to just the basename of the directory

                              $destPath = $ArchivePath + $folder[-1] + "\" + $folder + "\"

                                ^^creating a $destpath variable which is a combination of the archive root, plus the last character in the folder name of the current folder being executed, then the full folder name again, with the necessary backslashes (Very important you keep the trailing backslash here)

                              $indexpath = $destpath + "index.xml"

                                ^^setting the $indexpath variable to the path where the index.xml file is located for this particular file/folder being executed

                              IF (Test-Path $indexpath)

                                ^^checking if the index.xml file actually exists! this is important because sometimes files that have been coldstored have been deleted from the archive afterwards, and you don't want to go dumping them back in there.  IF the index file doesn't exist, then the loop exits back to the next directory in the current cold storage folder

                                  {   

                                  [xml]$xmlfile = Get-Content $indexpath

                                    ^^if the index.xml exists, then we get the content of the xml file and pull it into the $xmlfile variable

                                  $Extension = [System.IO.Path]::GetExtension($xmlfile.index.filename)

                                     ^^here we have to use the built in xml support in Powershell to grab the extension of the file we're restoring from the index.xml content, because files in coldstorage have no extensions, but files in the archive do, so we need to rename the files as they're being copied back to the real archive with the correct extension.

                                  (Get-Content $indexpath) -replace " status=`"archived`"", "" | Set-Content $indexpath

                                ^^Now we have to actually edit the index.xml file to remove the "status="archived" comment with a comma for each file version that is being restored, and commit the xml content back to the original xml file.

                                  $files = Get-ChildItem $name | where {! $_.PSISContainer}

                                ^^once we've gotten all that info, we get a list of the files in the cold storage archive directory we are currently looping through (just the subdirectory)

                                  foreach ($file in $files)

                                     ^^now we do another loop for all the files in that directory

                                      {

                                      $dest = $destpath + $file.name + $Extension

                                     ^^setting the destination for each file being copied as it is looped

                                      Write-Host "Copying $($file.name)$($Extension)"

                                     ^^just writing to the powershell window so I can see it running

                                      $file | Copy-Item -Destination $dest -Force

                                      ^^finally I'm just copying all the files over with the new extension to their destination path, then looping to the next file.  Once all files in each subfolder are done, it goes to the next, does the same, then once all subdirectories are done, it goes and starts again with the next coldstorage folder.

                                      }

                                  }

                              }

                      }

                       

                      Hope that helps anyone.

                      Josiah

                        • Re: Mass Restore of EPDM Cold Stored Files
                          Rodney Michels

                          I finished the job!  Took me about a 40 hours, but I feel its solid.  You can select multiple rows to retrieve versions, or retrieve the complete list.  I also added a button to copy the archive folders from my production server to my test archive server.  I was getting tired of coping the files manually, which could take hours, depending on how many you needed to copy, and the actual test only taking 30 seconds.  We have 473 thousand version in cold storage, and putting them back, one by one, so they can be compressed, would take years!!!

                          Again, Josiah, thanks for putting this idea in my head. 

                          ColdStorage.JPG