Many GPUs on one machine

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Many GPUs on one machine

Post by Peter_Hucker »

I've just purchased the following which (by my calculations, including CPU speed) should allow Folding to run flat out on up to 12 GPUs:

Esonic B250 BTC Gladiator motherboard (TWELVE PCI express sockets)
i3-6100 4-core CPU
256GB NVME.
(Borrowing RAM from another machine. Found case, cooler, power supply lying in garage).

I'm doing this because connecting 4 GPUs to one socket in my current machine causes folding to slow down. The CPU is not the limitation, sharing a PCI-express lane is. The GPUs are Tahitis (Radeon R9 280X) and will be connected with PCI-Express 2 USB risers. If you have faster cards, you can use ribbons to get PCI-Express 3.
Vester
Posts: 79
Joined: Sun Dec 02, 2007 4:29 pm
Hardware configuration: Gigabyte Aorus Z590 Pro AX, Intel i9-10850K, 32GB Crucial Ballistix DDR4-2600, Samsung NVMe EVO 980 Pro 256GB, CoolerMaster liquid cooler ML360, Nvidia Titan X (Pascal), Dell Nvidia RTX 3080 10GB 4Y12V, Pop!_OS.
Location: Fair Play, SC

Re: Many GPUs on one machine

Post by Vester »

I ran eight GPUs on a similar motherboard using PCIe x1 risers. In BIOS, hyperthreading should be disabled and you need to enable above 4k decoding.
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

I leave HT on, it gets more CPU work done. It only uses it if it needs to.

As for 4K decoding, I'll turn that on only if there's a problem, last time I put that on the computer wouldn't boot and I had to reset the CMOS. And that was a better machine - i5 8600K.

If it's too slow, I can get 1x to 16x ribbons, and just have to build a physical support structure to put the cards closer instead of just sitting on the shelf next to it. 3 foot USB cables make things much easier.
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

Correction, I've just read up on above 4G decoding (along with the newer "resizeable bar") and it sounds like they boost performance (about 15%). I'm gonna switch it on for all my machines. If you never see me again, it broke.
CarlHolmberg
Posts: 2
Joined: Mon May 18, 2020 1:20 am

Re: Many GPUs on one machine

Post by CarlHolmberg »

My pardons for jumping in months later. Re using various PCIe x1/x4/x8/x16 risers. My subjective experience suggests you may not want to get too carried away with the narrower connections.

My first setup used a Supermicro x8 dual socket, Extended ATX motherboard, mounted in a DIY plywood case. I had room for and used up to five GPU cards with risers attached wherever I could find room. Using NVTOP on Linux as my PCI performance monitor, I didn't appear to lose a noticeable amount of throughput with x8 risers compared to x16. Using x4, it noticeably sagged. Using the x1 riser via a USB cable, throughput was pathetic, this while I was using GTX1060s.

Unfortunately, I didn't write down any numbers back in 2020, and this was a 2012 mobo.
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

It depends on the card's power and the PCI express version. One lane of PCI Express version 2 will support 3 280X cards on Folding at full speed.

I'm putting more than one card per lane, not more than one lane per card.
nchowlett
Posts: 50
Joined: Tue Feb 28, 2023 4:05 am
Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
Location: Melbourne, Australia.

Re: Many GPUs on one machine

Post by nchowlett »

I'm having issues with Ubuntu recognising my last GPU (others are picked up by the OS fine). Has anyone had any issue of the GPU looking like it was physically working (lighting up and fans moving), but once in Ubuntu it didn't appear?
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

Frequently. But in Windows and with old cards. They do wear out. I dismantle them and give them a good clean, redo all the heatsink gunk, change any sticky fans, then they magically work. While I'm in there, I check for visible damage with a lit magnifying glass - blown up or burst or leaking transistors or capacitors.

If you fail to fix it or don't want to, they have a decent resale value on Ebay when broken. Just describe the fault and take photos.
nchowlett
Posts: 50
Joined: Tue Feb 28, 2023 4:05 am
Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
Location: Melbourne, Australia.

Re: Many GPUs on one machine

Post by nchowlett »

Thanks Peter! Just swapped in another GPU and my OS sees it, so I'll give your suggestions a go, thanks again.
nchowlett
Posts: 50
Joined: Tue Feb 28, 2023 4:05 am
Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
Location: Melbourne, Australia.

Re: Many GPUs on one machine

Post by nchowlett »

Peter_Hucker wrote: Thu Sep 28, 2023 3:14 am ... then they magically work.
I moved my case about 4 cm and suddenly it was booting (without infinitely rebooting). Go figure. :lol:
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

Are you saying you fixed the GPU by moving something?
nchowlett
Posts: 50
Joined: Tue Feb 28, 2023 4:05 am
Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
Location: Melbourne, Australia.

Re: Many GPUs on one machine

Post by nchowlett »

Peter_Hucker wrote: Thu Sep 28, 2023 6:05 am Are you saying you fixed the GPU by moving something?
Nope, but I was having issues where I couldn't boot into Ubuntu until I touched (moved) the motherboard (I think). Maybe a poor connection, they're off Ebay.
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

I had a machine where the SSD suddenly failed to be detected. I pushed the SATA lead's connection on the motherboard end (lightly, not violently), and then I couldn't boot either. The motherboard and the SSD are now broken. Assuming it wasn't me as I was careful, I'm guessing the disk controller was shorting out and blew the SSD, then managed to blow the motherboard when I touched it. Oh well, it was an old quad core board and I got a new one for £10, and an SSD for £7. And then found things were incompatible, continued to buy more pieces, and ended up with two machines. More GPU sockets!
nchowlett
Posts: 50
Joined: Tue Feb 28, 2023 4:05 am
Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
Location: Melbourne, Australia.

Re: Many GPUs on one machine

Post by nchowlett »

The motherboard and the SSD are now broken.
Dam, luckily I haven't had any such issues.

I'm trying to prototype a new configuration, your comments makes me think it needs to be almost production-quality at the same time (e.g. pre-designed and properly mounted to avoid human-induced errors). Annoying! But thanks for the insight, nonetheless.
Peter_Hucker
Posts: 306
Joined: Wed Feb 16, 2022 1:18 am

Re: Many GPUs on one machine

Post by Peter_Hucker »

No, my stuff is cobbled together from what's available and cheap. My GPUs aren't even inside the cases, I use mining risers on USB cables.
Post Reply