52 Replies Latest reply on Jan 18, 2019 11:35 AM by Justin Dailey

    What's your render rig?

    Jeffrey Model

      Lots of discussion about builds and GPUs lately. Figured I'd start a dedicated thread for folks to talk about and show what they're running Visualize on.

        • Re: What's your render rig?
          Jeffrey Model

          My current rig:

          Case: Corsair 500D

          CPU: i7-7820X

          CPU cooler: Corsair H110i

          Motherboard: ASUS PRIME x299 Deluxe

          RAM: 128GB (8x16) Corsair Vengeance LPX

          GPU1: Nvidia Titan Xp

          GPU2: Nvidia Titan Xp

          GPU cooling: EK Fluid Gaming with a 360mm radiator

          Storage: 2TB Samsung 970 PRO

          PSU: Corsair HX1200i

           

          Excuse the sloppy wiring - it was temporary and has been cleaned up.

          isFnltR.jpg

          • Re: What's your render rig?
            Dave Goetsch

            Looks like a fun thread Jeffrey!  It'll be interesting to see what hardware others are using. I just finished this new build about a week ago, so perfect timing !

             

            I'd also really like to see a new standard Visualize Benchmark that we could compare with our various builds. There are so many strategies on what is the best hardware (CPU's & GPU's) it'd be great to see benchmarks versus hardware for some real world comparisons. There are a lot of super talented users on this forum and a little sharing of knowledge can go a long way to getting the most bang for the buck when building a new workstation.

             

            My new Workstation:

            Case - Cooler Master COSMOS II 25th Anniversary Edition

            CPU's - Dual Intel® Xeon® Processor E5-2680 v3 - 24 cores 48 threads

            CPU Coolers - Dual Noctua NH-U12DXi4 120mm SSO2 (Push/Pull Config)

            Motherboard - ASUS Z10PE-D8 WS

            RAM - Samsung 128GB (8x16GB) DDR4 2133 EEC Server RAM

            GPU's - (4) EKWB Water cooled EVGA GeForce GTX 1080 Ti FTW3 + EK-FC Terminal X4

            Storage - (Boot) Samsung 970 EVO SSD 2TB & (Data) Samsung 860 EVO SSD 4TB

            PSU - EVGA SuperNOVA 1600W T2 80+ Titanium

            Radiator - EK-CoolStream PE 360

            Water Pump - EK-XRES 100 Revo D5 RGB PWM

            GPU Water Blocks - (4) EK-FC1080 GTX Ti FTW3

            Fans - Noctua everywhere (11)

             

             

            my-rig.JPG

            • Re: What's your render rig?
              Michael Dirisio

              Dell Precision Tower 7910 XCTO Base
              Dual Intel Xeon Processor E5-2630 v4 (10C, 2.2GHz, 3.1GHz Turbo, 2133MHz, 25MB, 85W)
              32GB (4x8GB) 2400MHz DDR4 RDIMM ECC
              2.5" 512GB SATA Class 30 Solid State Drive
              2.5" 512GB SATA Class 20 Solid State Drive (Additional)
              Dual NVIDIA Quadro P5000, (4 DP, DL-DVI-D), T7910 SLI
              1Gbit NIC add-in card (PCIe- Intel)
              Dell Precision T7910 1300W Chassis v2, BW

               

              Work didn't want me to build my own so this is the best I could do with ordering through a Dell Supplier

               

              20181026_122634.jpg

              • Re: What's your render rig?
                Mark Jackson

                My Current Rig:

                AMD Ryzen Threadripper 1920X CPU

                2 NVIDIA Titan Xp CPUs

                AsRock Taichi X399 mother board

                Crucial MX500 500 GB M.2 boot drive

                Samsun EVO 860 2TB M.2 data drive

                G.SKILL TridentZ 64GB RAM

                NZXT Kraken X62 CPU cooling

                EVGA SuperNOVA 1000 G1+ 1000W power supply

                All stuffed in a Corsair Carbide Series Air 540

                 

                Ryzen Render Rig 2018.jpg

                  • Re: What's your render rig?
                    Dave Goetsch

                    Very nice looking machine Mark !

                    • Re: What's your render rig?
                      Brent Gaspard

                      This is pretty sweet ... perhaps within my reach. How do you like the ThreadRipper 1920? And, this is a system I could build into, right ... I mean start with one Titan Xp and add the next over time - that is as long as they are to be found?

                        • Re: What's your render rig?
                          Mark Jackson

                          I think my choice of going with the Ryzen Threadripper 1920x was a very good one. With twelve cores, the Adobe and other software I run that can use the cores really rips! (pardon the pun)

                           

                          This build blows away my prior Intel based rig by a long shot. It should, it has way better specs. I don't have any problems what so ever with any of my software running an AMD CPU. In the past it may have mattered with some software, but I don't think AMD or Intel CPU software compatibility has any issues anymore. AMD or NVIDIA for GPU processing does still matter with some software like Visualize.

                            • Re: What's your render rig?
                              Dave Goetsch

                              You did right Mark. You pick your hardware based on your usage. That's the beauty of building your own rig. I did the same but it seems like I went a little overboard (as I usually do). But I won't be upgrading for a long time !  

                               

                              My decision to go with dual Xeons was really dictated by the fact I wanted to use 4 dual wide GPUs. So my motherboard has 7 PCIe slots. At that point you have very slim pickings and really need a server/workstation class motherboard.

                              I originally built mine with 2 1080 TI's but those empty PCIe slots looked so lonely I plugged in 2 more... 

                                • Re: What's your render rig?
                                  Mark Jackson

                                  Thanks Dave. The AsRock Taichi X399 mother board can accept 4 dual width GPUs.

                                  Dave Goetsch wrote:

                                  I originally built mine with 2 1080 TI's but those empty PCIe slots looked so lonely I plugged in 2 more...

                                  I know what you mean. The two empty PCIe slots look like barren unwatered soil. I need to plant them with tasty GPUs and make them bear fruit!   If I can get enough freelance rendering work I do plan on upgrading the power supply and installing two more Titan Xp cards.

                                   

                                  I'll have to find out how to convert the NVIDIA version cards I have to liquid cooled though. I've been thinking about converting the two I have now to liquid cooled if I can find the info on how to do it for my specific cards.

                                    • Re: What's your render rig?
                                      Jeffrey Model

                                      Watercooling is super easy. Happy to help walk you through it when you're ready to make the leap. My twin Titan Xp machine uses watercooling on the GPUs and the CPU, as does the new quad 2080 Ti rig.

                                       

                                      Once you've watercooled one of these GPUs, you'll never go back to air.  Even if there were no performance gains, it would be worth it just for the silence.

                                        • Re: What's your render rig?
                                          Mark Jackson

                                          My GPUs are actual NVIDIA cards, not EVGA or another licensed brand. I haven't been able to find a kit for my cards, but, I've never liquid cooled GPUs before, and don't know if I actually know what to look for. I thought there would be card specific kits because the chassis is different for the different brand GPU cards.

                                            • Re: What's your render rig?
                                              Jeffrey Model

                                              My Titans are Nvidia reference GPUs as well. In terms of physical layout, the Titan Xp is identical to the 1080 Ti but with one more RAM module populated where the 1080 Ti has a blank space. This means that anything which fits a 1080 Ti will also fit a Titan Xp.

                                               

                                              On my Titan rig, I found that the easiest way to get going for water cooling them was to buy a FluidGaming kit from EK. It included everything but the coolant. The caveat here is that these kits use aluminum radiators and therefore aren’t compatible with the much easier to find copper components (connectors, valves, radiators, etc). If you’re not intending to upgrade, these kits are great. If you want something modular which you’ll be expanding in the future, then something copper would work better. I did copper on my big rig.

                                               

                                              If it would be useful, I can send you a PM with a BOM for a couple of setups.

                                          • Re: What's your render rig?
                                            Dave Goetsch

                                            Mark Jackson wrote:

                                            Thanks Dave. The AsRock Taichi X399 mother board can accept 4 dual width GPUs.

                                             

                                            Ahhh...   I see.  Just had a look at your MB on Newegg. Very nice, 4 PCI slots but double spaced. That would have worked for me also. I've been tempted to go with Threadripper also. I'm sure the performance of one would be higher than the dual Xeons I ended up with. Less cores but much higher clocks. Much less expensive also....

                                              • Re: What's your render rig?
                                                Jeffrey Model

                                                Also be sure to check that the case you're using can fit 4 double spaced cards. On my Titan build, the Corsair 500D only has 7 PCI knockouts. Thus, the case can fit only 3 double spaced cards plus one single.

                                                • Re: What's your render rig?
                                                  Mark Jackson

                                                  Yeah, but can't you stick seven GPU cards in your rig?

                                                    • Re: What's your render rig?
                                                      Jeffrey Model

                                                      With a PCI extension, I could fit one more in the big rig. The 2080 Ti has a 2-slot faceplate because of the extra connector they added. The Titan Xp has a 2-slot faceplate but there’s nothing in the second slot, so you could theoretically do them on one slot spacing if you watercool them since that cuts them down to only one slot wide.

                                                       

                                                      At some point, you start bumping into limits on the number of available PCI-E lanes. With an i9, you get 44 lanes. Set 4 aside from an M.2 and that leaves you 40. The top end Nvidia cards (ie Titan Xp) don’t like having less than 8 even though the bus sits idle almost always. So that means you could do up to 5 GPUs on an i9 box if you had enough physical slots available and a big enough case.

                                                       

                                                      With a Threadripper 1950X, you get 64 PCI-E lanes (but only 4 memory channels, so you’ll be limited to 64GB). 4 go to the chipset, assume 4 more for the M.2 drive, and that leaves 56. So, with Threadripper you could do a 7 GPU rig.

                                                      • Re: What's your render rig?
                                                        Dave Goetsch

                                                        I could do 7 GPUs if they are single slot GPUs or water cooled double slot cards. But as Jeffery mentioned the 1080 TIs have a 2-slot wide faceplate to accommodate the DVI connector, which I don't actually need. I've actually thought about removing the DVI port from the cards but that could be a can of worms. My case is HUGE and has 10 (single) PCIe openings on the rear. My CPU's have 40 PCIe lanes each so I have 80 total. No restriction here. But each EVGA 1080 Ti I have can draw up to 330w. I don't think my 1600w PSU could deliver enough to feed them along with everything else. The AC power cord to my PSU is MASSIVE and could easily be used to jump start a locomotive. For cable management it's like trying to stuff a very pissed off Anaconda behind your desk. Plus, as it is, every time I turn on my computer it dims all the street lights down my street.  

                                                         

                                                        On another note. I use a EKWB Terminal to tie my 4 GPUs together which I found is really nice as it simplifies your water connections and greatly reduces the number of fittings you need (fittings can be a little expensive). Also, it turns your GPUs into one integrated structural unit that is super stable. Water blocks usually have a large chunk of copper plate that actually makes them heavier than their air-cooled counterpart. Your GPUs can sag from weight and put stress on your PCIe slots. Not so when you use a Terminal to connect all your cards together. No sag at all. It ties your cards together with 12 allen bolts making it one structurally integrated unit.

                                                         

                                                        I use this one which is a X4 (for 4 GPUs) ...

                                                         

                                                        Screen Shot 2018-11-23 at 4.47.20 PM.png

                                                         

                                                        But, they also have a X7 version for tieing 7 GPUs together...

                                                         

                                                        Screen Shot 2018-11-23 at 4.47.44 PM.png

                                                         

                                                        my-rig2.jpg

                                                          • Re: What's your render rig?
                                                            Jeffrey Model

                                                            Those terminal blocks are nice. They go a long way towards cleaning up routing and, like you said, the number of fittings.

                                                             

                                                            I have my big rig set up with two fully modular loops that each have 2 GPUs. My thinking was that this will allow for the easiest upgrade path as I'll be able to just swap cards in and out on a whim. An issue I ran into with this concept was that the fittings are large enough to make tube routing a bit of a pain without a nice terminal block. I went with Alphacool GPU blocks, and it turns out that the Alphacool terminal blocks (SLI blocks) are too large for this usage both because two blocks can't fit side by side, but also because the fittings stick off of the sides rather than the top.

                                                             

                                                            So, I designed my own bridge blocks. And rendered them in Visualize, naturally.

                                                            Waterblock ASY Rev 01 16.jpg

                                                            First set came in with a couple of hiccups in the machining. I'll have the second set installed in the machine in another couple of weeks.

                                                            Image from iOS (1).jpg

                                                            • Re: What's your render rig?
                                                              Jeffrey Model

                                                              Oh, and you're not kidding about lights dimming when the GPUs are cranking. I'm having a dedicated circuit run just for the big machine. At full bore, it's pulling about 1600W. That's 14.5 amps assuming no imaginary load. I only run it at about 60% power at this point because it really does make the lights in the room flicker.

                                                                • Re: What's your render rig?
                                                                  Dave Goetsch

                                                                  Very nice work on the blocks !

                                                                  So you're having them 3D printed then machined threads ?

                                                                   

                                                                  Jeffrey Model wrote:

                                                                   

                                                                  Oh, and you're not kidding about lights dimming when the GPUs are cranking. I'm having a dedicated circuit run just for the big machine. At full bore, it's pulling about 1600W. That's 14.5 amps assuming no imaginary load. I only run it at about 60% power at this point because it really does make the lights in the room flicker.

                                                                  rolling-on-the-floor-laughing.png

                                                                    • Re: What's your render rig?
                                                                      Jeffrey Model

                                                                      They're machined polycarb. For some reason, the shop sprayed them with clearcoat. I'm undecided on whether that's a good thing. I appreciate that I don't have to polish them myself, but I'm not sure about how compatible the clearcoat is with the coolant. I may request no clear coat on the next set.

                                                                        • Re: What's your render rig?
                                                                          Dave Goetsch

                                                                          In a past life I was a Senior CNC Programmer in the Aerospace Industry. McDonnell Douglas, Boeing and others. I worked on many military and commercial aircraft including the Space Shuttle for North American Rockwell in Downey, CA. That is not is simple part to produce. Not super hard but not easy either. From the picture it looks like they did quite well. The clear coat was used to eliminate the machine marks (which is why I thought it may have been 3D-Printed) and bring back the transparency. You may be right though about compatibility with your coolant, especially heated coolant.

                                                                           

                                                                          Best of luck, and it looks like you're on the right path.

                                                                           

                                                                          Dave

                                                                            • Re: What's your render rig?
                                                                              Jeffrey Model

                                                                              Yeah, the problem with having to replicate features from molded parts is that they routinely don't translate very well to machining. This ended up at 4 setups on a 3-axis plus a bunch of contouring on the final setup. Could have knocked it down to 3 setups, but wanted symmetry in one feature "just in case."

                                                                               

                                                                              The shop did quite well, but there was a file error which put a half-inch notch straight through a critical area. They're making a new set now without that notch, plus a few dimensional tweaks from my test fits.

                                                      • Re: What's your render rig?
                                                        Jeffrey Model

                                                        Just put together a new rig. The cooling and wiring need to be cleaned up and I need a new circuit run to handle the full power, but I just did a quick test to see how it runs. Using the reference model (Bunkspeed sign) at 3840x2160 and 500 passes in GPU-only mode, I get:

                                                        No denoiser: 1:06

                                                        W/ denoiser: 1:06 (same)

                                                         

                                                        This is almost exactly double the speed of my other rig with twin Titan Xps in it (also water-cooled). That's pretty much what I'd expect given that the new Turing functions aren't being utilized in Visualize (yet).

                                                         

                                                        Details on the new rig incl pics and proper performance specs to follow once the new circuit has been run. In the mean time, it's an i9 with quad 2080Tis and water cooling.

                                                        • Re: What's your render rig?
                                                          Dave Goetsch

                                                          Found a new use for my rig....

                                                           

                                                          1. Turn radiator fans on low.

                                                          2. Start a heavy render and set time for 15-20 minutes.

                                                          3. Place lunch on rig above radiator vents.

                                                          4. Wait for render to complete.

                                                           

                                                          Take lunch break and Enjoy !  It really works great !

                                                           

                                                          Burrito-Lunch.jpeg

                                                          • Re: What's your render rig?
                                                            Pete Wetzel

                                                            I am looking to create a "budget" Boost farm with 5x Dell 5810 Precision Towers and a single GeForce RTX 2070 card in each one. Would I be better off with one computer with a single Titan RTX ($2500) or 5x Towers each with a $500 RTX 2070?

                                                             

                                                            Running 2018 Sp3.

                                                             

                                                            5x RTX 2070 @2304 Cuda cores = 11,520

                                                            1x Titan RTX = 4608 Cuda cores

                                                              • Re: What's your render rig?
                                                                Stefan Ilic

                                                                More CUDA cores means faster render. Look at the 2080Ti, it might be the best price/performance option.

                                                                  • Re: What's your render rig?
                                                                    Pete Wetzel

                                                                    I think I could justify the jump from 2070 to 2080 but the jump to 2080 Ti is a little more than I want to propose.

                                                                     

                                                                    GPUCoresCostx5 Coresx5 CostCore DiffCost DiffWattsCableRAM
                                                                    Titan RTX4608 2,500 23040 12,500 100.0%371.7%2808+824
                                                                    2080 Ti4352 1,300 21760 6,500 88.9%145.3%275/6508+811
                                                                    1080 Ti3584 1,350 17920 6,750 55.6%154.7%250/6006+811
                                                                    10802560 900 12800 4,500 11.1%69.8%180/5006+88
                                                                    20802944 700 14720 3,500 27.8%32.1%6506+88
                                                                    20702304 530 11520 2,650 006506+88
                                                                      • Re: What's your render rig?
                                                                        Stefan Ilic

                                                                        Let's consider following math in one single PC:

                                                                         

                                                                        5 x 2070 = 2500$ = 11520 CUDA cores (you can actually fit 4 GPUs in one PC)

                                                                        2 x 2080Ti = 2400$ = 8704 CUDA cores

                                                                        4 x 2080 = 2800$ = 11776 CUDA cores

                                                                         

                                                                        Not factoring electricity costs which should be higher for 4x and 5x GPU configurations.

                                                                         

                                                                        Your best price/performance option should be - 1 single PC with 1 CPU and 4 GPUs:

                                                                         

                                                                        • Intel Extreme edition CPUs or AMD Threadripper with 64 PCIe Lanes (2950x)

                                                                        • 4 x 2080 or 4 x 2070 since 4 GPUs can be stacked in one PC and supported by one CPU on x16 speed if motherboard supports it (4 x PCIe 3.0 x16 GPU = 64 PCIe Lanes). If GPUs are on x8 speed, they perform ~3% slower according to most tests.

                                                                         

                                                                        Also You can go with two PCs with two or three GPUs per PC. In all these calculations factor the cost of CPUs, RAM, Motherboard which is at least additional 500$.

                                                                         

                                                                        Considering licenses, with every purchase of SolidWorks Visualize Professional You get Visualize Boost license, but You will loose it if You are not on subscription contract. Cost of one Visualize Boost license is around probably around 1k euro/usd, maybe even cheaper.

                                                                        • Re: What's your render rig?
                                                                          Paul Salvador

                                                                          Pete,.. for us reading... I think we all either feel equal hate or envy for you.   I mean,.. why stop there,.. just throw in your own personal  generator!?

                                                                          CARB Complaint Generators | Air Cooled Generators | MA Emissions Regulations Approved

                                                                            • Re: What's your render rig?
                                                                              Pete Wetzel

                                                                              IMG_2530b.jpg

                                                                              Paul will this do? And these desktops are going in a clean data center that is air conditioned so the noise of the fans is not a concern.

                                                                               

                                                                              I am currently running a turntable animation that is 10 seconds long at 60fps = 600 images to render. It will take 63 hours to complete on my current 5 laptop Boost farm. My goal is to cut this by at least 60% with the desktops and decent GPU's.

                                                                        • Re: What's your render rig?
                                                                          Mark Jackson

                                                                          I think you have to purchase 5 boost licenses for the 5 PC don't you. Wouldn't that factor into the costs?

                                                                          • Re: What's your render rig?
                                                                            Jeffrey Model

                                                                            I think the first consideration should be whether your uses need more than 11GB of VRAM. If the answer is yes, then the Titan RTX is essentially your only option (until your apps support VRAM pooling via NVLink).

                                                                             

                                                                            If the answer is no, then I would ask whether five standalone machines are required or if you simply need more GPUs. Figure that each standalone machine comes with about $1500 in cost overhead due to component duplication (PSU, motherboard, RAM, screen, etc). For five machines with 2070s inside, you're looking at around a $10k spend.

                                                                             

                                                                            If you do not need 5 individual machines, then you'll be best served by putting four GPUs into one machine. For the same $10k budget, you could put four 2080 Tis into one machine and water cool the whole thing. You'll end up at around 3x the performance of five machines with 2070s, not including the even larger improvement on shorter renders since you won't have to wait for data to be passed around the Boost farm.

                                                                             

                                                                            If you're simply looking to beat the performance of 5 laptops, just put two 2080s or 2080 Tis into a desktop. Laptop GPUs are significantly slower at sustained renders even when they have GPUs with the same branding as desktop ones.

                                                                          • Re: What's your render rig?
                                                                            Justin Dailey

                                                                            Fun thread!

                                                                             

                                                                            Here's mine..  Visualize Pro runs very slow & glitchy on it.  My last two overnight animation renders failed and effectively set me back two days on my project.  Should have bought Keyshot.. 

                                                                             

                                                                            AMD Threadripper 1950x @4.0GHz - 360mm AIO Water Cooler

                                                                            Asus Zenith Extreme Motherboard

                                                                            128GB GSkill Trident Z @2666MHz

                                                                            Nvidia RTX2080ti + 2X GTX1080FE

                                                                            Render Rig.jpg

                                                                              • Re: What's your render rig?
                                                                                Brent Gaspard

                                                                                See, this is disappointing as I am still new and observing all these inputs. I was favoring a ThreadRipper build, but I sure as hell wasn't planning on three GPUs out of the starting block - so, I would see even worse performance?! Jeez!

                                                                                  • Re: What's your render rig?
                                                                                    Justin Dailey

                                                                                    TR build will still give you the best possible performance per dollar vs anything intel.  I'm not sure how it compares to other AMD offerings atm.  My render rig is also my gaming pc and I have a separate SW workstation with 7700k @ 5GHz and a P4000 so TR made sense in my scenario.  I already had the two 1080FE's and I bought the 2080ti as an upgrade for gaming but since visualize can still use the cuda cores, I left the 1080's installed.  CPU should be fine to render still images.  The added GPU speed is nice for 2000+ frame animations (if they don't slow down or fail). 

                                                                                     

                                                                                    After spending this entire week trying to get my project to render..  I'm concluding that the "slowness" is due to terrible vram management & file efficiency by visualize.  My project is 2.5M polygons but it takes 10 mins to open the project and 15-20 mins for an animation render to start.  Luckily it fails in the first few frames but it still takes me another 30 mins to start another render just to see if it fails again.  Now I got it to not fail on "fast" setting and it starts off fast (1000 passes/frame in under 10 secs) and then slows way down (1000 passes/frame in 1.5-2 mins). CPU & GPU usage drops to almost nothing.  Temps are fine; vram usage is less than 75%; ram usage is at 26%; there's virtually zero disk usage; I cleaned the temp directory; no other programs are running; I updated to 2019 SP1; I tried redoing the entire animation in 2018.  I have no idea why it does this..  All I know is that it shouldn't and I'm very angry because it has wasted so much of my time. 

                                                                                     

                                                                                    If I find a solution I'll post it..

                                                                                  • Re: What's your render rig?
                                                                                    Jeffrey Model

                                                                                    More than anything, I think you're running into issues due to the GPUs overheating. The stock air coolers on the Nvidia cards are pretty bad when they're in open air, but your setup has them completely choked off from any cooling.

                                                                                     

                                                                                    The other thing is that, yes, Visualize is very unstable on the 20-series cards.