17 Replies Latest reply on Jun 1, 2018 8:16 AM by Damir Galic

Importing (very) Large Assemblies

Stephen Mounioloux May 16, 2018 5:04 PM

Hello,

I am a pretty new Visualize user and I am looking for some good tips to help handle the largest scene I will probably ever need to render.

I have a fairly complex chassis enclosure which has been modeled with great details (several electronics board, cooling fans, hardware, etc.) My workstation had no problem with it - exploded view animation - hence the high number of internal parts. I was able to even import a full 48U rack filled with this chassis, without lightening the assembly even though it was only for a few external renders.

The final step however is to model a lot of these racks on a data-center floor. I quickly maxed out nearly 400GB of ram in the process and killed it after it started swapping. I knew it wouldn't have been optimized considering I didn't need any of the internal parts of the chassis. So I made a simplified assembly removed as much as I possibly could, however I am still hitting a wall. I didn't stare at the screen for the 5-6 hours I let it try to import the assembly, I was not swapping (75% memory used).

At this point there is no a whole lot of parts I can strip off the assembly. So being pretty new to Visualize I wanted to ask if there were any trick to import large assemblies?

My assembly structure goes as Enclosure -> Linear Pattern (to fill a rack) -> Linear Pattern (for the first row of racks) -> Mirror for the second Row. I won't be modeling any additional row of racks, just these 2. I am not sure if that makes any difference.

As for the import in visualize itself, I have tried both appearance and automatic.

I haven't re-imported the emptied/simplified enclosure by itself to check the polygon counts, but I estimate a total of a couple hundred millions polygons in the current state of things.

Thanks

Re: Importing (very) Large Assemblies

Chris Cunningham May 16, 2018 6:33 PM (in response to Stephen Mounioloux)

For large assemblies I build out a scene containing final camera angle, 1-3 duplicates of the product and the final settings on my HDR.

Then I save and copy and paste the actual file on my hard drive. Then go into one of the copies and move the duplicates to a secondary position. And continue until I have all my save files representing all the different locations of the same product in the scene.

I render out each scene from the saved files and composite them inside of Photoshop. It works very well for me.

I believe one day Visualize will have a feature to duplicate out a model without adding poligons and taxing the system.
Like Show 1 Likes(1)

Actions
- Re: Importing (very) Large Assemblies
  
  Stephen Mounioloux May 16, 2018 7:16 PM (in response to Chris Cunningham)
  
  Thank you, Chris.
  
  This is an approach I actually considered and quickly ruled out since the main goal for that challenging render is to actually output a walk through animation.
  
  I am thinking that if everything else fails I will have to simplify the geometry of the enclosure, trading off polygons for a decal. It kind of defeats the purpose of trying to make a realistic animation though.
  
  Like Show 1 Likes(1)
  
  Actions
  - Re: Importing (very) Large Assemblies
    
    Rob Rodríguez May 16, 2018 7:47 PM (in response to Stephen Mounioloux)
    
    Not to deter you but I recently wanted to do an animation of an assembly. The assembly had about 25 unique components that were patterned to give a total number of components in the 10K range. My first attempts and Visualize wouldn't even open the assembly. I did manage to get it to import (using appearance import makes the model tree the smallest) but it just wasn't workable once open. After opening I fired off a quick render and it would crash Visualize. Tried using the queue and closing Visualize after sending the render and it still would crash. Eventually I gave up and used different software.
    
    I'm not saying you shouldn't give it a try in Visualize but you will definitely be in for a challenge dealing with a large assembly.
    
    Some advice, the image quality setting in SolidWorks will send polygon counts through the roof if you set it too high. Keep it set as low as you can get away with.
    
    Good luck
    
    Like Show 1 Likes(1)
    
    Actions
    - Re: Importing (very) Large Assemblies
      
      Stephen Mounioloux May 16, 2018 7:55 PM (in response to Rob Rodríguez)
      
      Thank you, Rob.
      
      I read that tip on the image quality setting before, but forgot to check if the main assembly is not set too high. Thanks for the reminder, I'll double check that.
      
      Out of curiosity would you recall how many polygons was your scene before you gave up on Visualize? Also would you mind sharing which software you decided to go with? I have exported Solidworks assembly into STEP files and then imported into Blender, however it was a fairly small assembly. I worked it out but it was a whole lot of work compared to what my workflow is with Visualize.
      
      Thanks again, Cheers
      
      Like Show 1 Likes(1)
      
      Actions
      - Re: Importing (very) Large Assemblies
        
        Rob Rodríguez May 16, 2018 9:20 PM (in response to Stephen Mounioloux)
        
        Not sure what the poly count was? I could check but I have 2 systems running animation frames currently and it will be a couple days before they are finished.
        
        I used Modo. PV360 is like a light version of Modo so all SolidWorks "stuff" (materials, scene setup, animations, decals, geometry, etc) transfer into modo pretty seamlessly (whatever your model looks like in PV360 it will look like in Modo on initial import). Much the same as Visualize. Modo has instancing so it handles large assemblies better. That's not to say that Modo doesn't have issues of its own. It does (plenty of them) and ease of use is not its strong point but, if you can find your way around Blender you should be able to find your way around Modo.
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Stephen Mounioloux May 16, 2018 10:12 PM (in response to Rob Rodríguez)
        
        Thanks Rob, this helps!
        I do as much prep work in SW prior to importing in visualize. I found Photoview to be fast enough to get an idea if my texture/materials look right. So for me if it wasn't for the the camera paths/keyframes and the faster renders from using my GPU's (and now denoiser on SP3 which I love) I would probably just use photoview.
        
        Still a few things to try before giving up, like setting the software to either CPU or Hybrid prior to importing (thinking maybe I am maxing out the GPU memory?)
        
        thanks again
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Ron Bates May 17, 2018 9:04 AM (in response to Stephen Mounioloux)
        
        Large assemblies... particularly with a bunch of patterned stuff, is not handled as well as it could be in Visualize at the moment. It is very much on our radar to improve in the near future as there are things we should be doing to deal with patterned/instance geometry better.
        
        Sounds like you're doing most of the things you can/should... Summary from my point of view:
        
        Suppress (don't just HIDE) any geometry that you don't absolutely need in the rendering.
        Simplify geometry where you can (cylindrical holes/bosses for example, particularly if feature patterned a lot) will explode triangle count. If they are not easily seen in the rendering consider simplifying
        Keep tessellation slider in SW as low as you can get away with
        Experiment with import modes a bit up front with smaller sub-assemblies
        Appearance import mode may take the longest to import b/c it will be grouping all like-appearance geometry into parts (perf should also be improved in the future) but once it is imported it will be fastest to work with as really large model trees have a performance impact on Visualize.
        Automatic would import faster, but the bigger model tree after import could be problematic when working with the scene.
        Using textures or decals as you mentioned, to replicate some model detail is also a super smart idea.
        
        Good luck. Hopefully we can roll out some updates in not so distant release that will improve a lot of these large assembly performance issues.
        
        Like Show 3 Likes(3)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Stephen Mounioloux May 17, 2018 11:45 AM (in response to Ron Bates)
        
        Thanks Ron, this is good info.
        I'll mark the thread as solved now since I believe everything has been covered.
        I will be doing another pass through my sub-assemblies and try to remove as much as I can.
        
        Good to know you guys are working on improving handing of large assemblies with lots of repeat patterns.
        
        Thanks!
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Stephen Mounioloux May 17, 2018 9:48 PM (in response to Ron Bates)
        
        Actually Ron, let me ask you one more question.
        
        I am still in the process of slowly removing parts and features without compromising the final output and I am trying to understand what the bottleneck.
        
        At this point I have removed enough not to be close from swapping so I am almost certain it's not a main RAM issue.
        However from all the tests I've done, I never actually crashed Visualize and it's me who eventually stop the import (not even killing the task, etc). Outside of my very first try (with all internal components to my enclosure) I always pass the initial "loading" part of the import process and get to the part called "Loading and analyzing model data" to about 25%.
        
        I get to that exact same point all the time in my attempts of removing components and features.
        
        So my question is, can it be a GPU memory bottleneck? if so, should I see an error message?
        if it is a GPU memory, should I try to set the settings to CPU or Hybrid instead? would that make a difference during the import? or does that only affect renders and during actual use of the software?
        
        as of my last attempt, I have a total of:
        ~ 100,000 parts
        ~ 670 M polygons
        (Keep in mind I have ~2,000 enclosures or an average of about 50 visible parts I need to import per enclosure.
        
        Assuming the 5 Million polygon per 1GB of GPU RAM, my maximum polygon count would be 5Mx48 which is only 240 M polygons. So yeah I am probably pushing it a little, however I don't know if it's a hard or soft limit? Would going over the max make the software substantially slower? Crash? Not load without actually crashing?
        
        bonus question: Should I use a SLI bridge for my 2 GPU's? I get nice render times with both GPU working 100% when rendering - just not sure if for visualize to see and benefit from the 2x 24GB cards I need to have them bridged physically or not.
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Dave Goetsch May 18, 2018 11:08 AM (in response to Stephen Mounioloux)
        
        I think I can answer the last part. SLI will not improve anything in Visualize. It may even cause issues.
        
        I don't think you have 48 GB of GPU ram available for your scenes. Visualize does not combine the ram of two or more cards. Your entire scene has to be able to fit into the v-ram of all GPUs individually. So if you have one GPU with 24 GB and one GPU with 12 GB you will be limited to 12 GB of ram for your scene.
        
        Vis guys, correct me if I'm wrong...
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Stephen Mounioloux May 19, 2018 9:16 PM (in response to Dave Goetsch)
        
        Thanks Dave,
        Such a bummer though. I have 2x 24GB cards which should give a maximum of 5M * 24 or about 100M.
        I'll extrapolate the number of enclosures which should net about 100M polygons and see if it imports.
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Dave Goetsch May 21, 2018 12:21 AM (in response to Stephen Mounioloux)
        
        Yes it's kind of a bummer but your poly count is beyond insane.
        
        I know you're trying to whittle down the count, here's how I would approach the issue. I assume you're modeling in Solidworks. Create a Solidworks assembly with only one 48U rack and fill the front of the rack with only the faces of your chassis. Do not include anything that you can not see inside the rack. No chassis sides, backs, internals, tops or bottoms. Well maybe backs and sides if you need to show them. Just a nice clean rack with front and maybe side and rear panels. Import to Visualize and check poly count. If necessary reduce the "Image Quality" option in Solidworks to reduce the count even further. The Image Quality setting can increase your poly count dramatically and only really helps to reduce tesselation of curved items. Your racks are mostly linear surfaces and you shouldn't need too high of setting to achieve acceptable results. Especially for something like a fly-through.
        
        If you need the side panels showing on the Racks at the end of the rows make one Solidworks assembly with the side panels and one assembly without. The racks in-between the end racks do not need to have the side panels as you can't see them anyway.
        
        So, I'm just telling you how I would approach this. EVERYTHING you can eliminate will help.
        
        Also, once you have one nice clean low-poly Rack you can import only one rack into Visualise and copy and paste the model to create as many as you like and translate them into position right within Visualize. This may help a lot also as only one model needs to be imported (and translated) into Visualize. Copying and pasting a rack in Visualize that has already been imported and translated will be much faster than importing all the racks that were patterned and mirrored in Solidworks. Scratch that 6+ hour import.
        
        Anyway... Just some suggestions... Please keep us posted on your success or lack thereof.
        
        Dave
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Stephen Mounioloux May 21, 2018 12:06 AM (in response to Dave Goetsch)
        
        Dave,
        
        This is very interesting, thank you. I was able to import an assembly with only 2 racks (filled with enclosures) but I didn't know that I could duplicate these instances. That was also prior to substantially optimizing my model for this animation, so this may be the best way for me to be successful with this project.
        
        I'll create a simple project to learn how to duplicate groups of parts and position them. Hopefully this can be done with inputing numbers rather drag and drop.
        
        Thank you
        
        Like Show 1 Likes(1)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Ron Bates May 18, 2018 12:42 PM (in response to Stephen Mounioloux)
        
        I honestly don't know if there's a good metric that can be reliably used to extrapolate how large your scene can be, given your hardware configuration. Too many variables.
        
        But I can say over half a billion polygons is crazy. I don't think I've ever seen a project in Visualize with more than 20-50 Million. So again, without trying to even extrapolate the bounds...even on the most insane setup...you'll be pushing ridiculous boundaries.
        
        You don't need SLI to use both cards. BUT, again I'm not certain about how memory is used between them. Dave may be correct,... I think this is what the more recent NVLink from nVidia is all about. sharing resources efficiently between GPU's and CPU.
        
        Like Show 2 Likes(2)
        
        Actions
        
        Re: Importing (very) Large Assemblies
        
        Stephen Mounioloux May 19, 2018 9:22 PM (in response to Ron Bates)
        
        Thanks Ron for confirming the bridge is not required.
        
        Any idea when/if there will be a new release with enhanced import? and maybe instancing support for scene with large patterns?
        
        Like Show 1 Likes(1)
        
        Actions
Re: Importing (very) Large Assemblies

Helge Feddersen May 21, 2018 4:13 AM (in response to Stephen Mounioloux)

Hi Stephen,

i have tried a couple of really big assemblies in Visualize - none as big as yours though

for me, the best solution was to go back to PV360. Main reason: what good is a loaded model if you cant work with it (change colors, views etc etc.) Chances are also high, due to the high Poly count, the render engine will default to CPU rather than GPU for rendering. i would, prior to any editing, make sure that rendering works on GPU

dont get me wrong, i love Visualize and am using V Prof as much as i can. and prefer Vizualize over PV360
For what you are doing i just dont think you will get very far

br
Helge
Like Show 1 Likes(1)

Actions
- Re: Importing (very) Large Assemblies
  
  Damir Galic Jun 1, 2018 8:16 AM (in response to Helge Feddersen)
  
  I actually like that visualize keeps the links so you can basically edit visualize objects seperately, then just click update from file in large assembly.
  
  Like Show 0 Likes(0)
  
  Actions

Go to original post