Linux Fahcore22/23/24 GLIBC issue resolution

[muziqaz]

Recently we had major issues with GLIBC version mismatch between libstdc++.so.6 bundled with the GPU fahcores and the one which comes with any Linux distro.
So basically latest distros would fail with core22 and core23 projects, but work with core24, while older distros would fail on core24 and sometimes core23, but work with core22 projects.
Solution is to remove libstdc++.so.6 inside of fahcore22/23/24 folder, and then symlink system provided libstdc++.so.6 to fahcore22/23/24 folders.
Finish any folding before doing this.

Core22 console commands:

• sudo rm /var/lib/fah-client/cores/openmm-core-22/fahcore-22-linux-64bit-release-0.0.20/libstdc++.so.6

• sudo ln -s /lib/x86_64-linux-gnu/libstdc++.so.6 /var/lib/fah-client/cores/openmm-core-22/fahcore-22-linux-64bit-release-0.0.20/libstdc++.so.6

Core23 console commands:

• sudo rm /var/lib/fah-client/cores/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3/libstdc++.so.6

• sudo ln -s /lib/x86_64-linux-gnu/libstdc++.so.6 /var/lib/fah-client/cores/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3/libstdc++.so.6

Core24 console commands:

• sudo rm /var/lib/fah-client/cores/openmm-core-24/centos-7.9.2009-64bit/release/fahcore-24-centos-7.9.2009-64bit-release-8.1.4/libstdc++.so.6

• sudo ln -s /lib/x86_64-linux-gnu/libstdc++.so.6 /var/lib/fah-client/cores/openmm-core-24/centos-7.9.2009-64bit/release/fahcore-24-centos-7.9.2009-64bit-release-8.1.4/libstdc++.so.6

After this try folding on your GPU

[arisu]

I don't think you need to symlink it. The system will look for libstdc++ in the system library path if it can't find it in LD_LIBRARY_PATH. That is what I had done and it worked with core22. I assume it should work with the other cores, shouldn't it?

[muziqaz]

It doesn't work like that

[arisu]

I just tested it now with core26, and after removing a library in its current directory, it tries to find it in the system directories and uses it if it finds it, confirmed with ldd and strace. Maybe there's another reason why it needs it in its own directory? But it works with core26 at least.

Not saying that symlinking isn't needed, but trying to find out why it is, because when I tested with core22 (and 26 just now), it wasn't.

[muziqaz]

Core26 has been developed to fix that exact thing
So it has a fix

[muziqaz]

The reason why it is is known, older cores look for shared libraries bundles with the core, when GLIBC version of the library bundles with the core does not match that of the system one, problems arise.
Core22 will always fail on latest distro versions (Ubuntu 24.04)
Core23 might fail on old or latest distros, but might work on one gen old distros (Ubuntu 22.04)
Core24 will fail on old distros (Ubuntu 20.04)

[Nicolas_orleans]

I have two rigs on Ubuntu 24.04 that fold out of the box core 22/23/24 on Nvidia cards so I don't support the "Core 22 will always fail on Ubuntu 24.04". You may want to refer to bleeding edges distros with rolling releases but not to Ubuntu 24.04 ?

[muziqaz]

We never really determined what caused the actual mismatch.
Yes, there are few users who got lucky and something did not update on their systems to trigger the issue. But the rest of the world suffered really bad, FAH lost 100s of thousands of WUs due to this issue.
We had users who were generating failed WUs in 10s of thousands at some point

[DarkFoss]

I had a wu that was initially rejected on the 7th, 2 days later it was rejected 4 times 10 min apart before being assigned to my machine. It finished fine with a TPF of 1.14 tad slower than normal for that project but well within the expected range.

I'd say the issue it still ongoing at a bad rate, Despite iinstalling via the packet manager and clinfo working right after installation and rebooting I still had to symlink core 23.