Impossibly slow servers

enroscado · Post by **enroscado** » Wed May 22, 2024 9:14 am

I've been having impossibly slow downloads both for WUs as well as cores for a few weeks now.

Sample logs:

*********************** Log Started 2024-05-22T07:59:13Z ***********************
07:59:13:I1:*********************** Folding@home Client ***********************
07:59:13:I1: Version: 8.3.5
07:59:13:I1: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:59:13:I1: Org: foldingathome.org
07:59:13:I1: Copyright: 2023-2024, foldingathome.org
07:59:13:I1: Homepage: https://foldingathome.org/
07:59:13:I1: License: GPL-3.0-or-later
07:59:13:I1: URL: https://beta.foldingathome.org/
07:59:13:I1: Date: Feb 14 2024
07:59:13:I1: Time: 14:09:03
07:59:13:I1: Revision: 652c05f093c7d9542ec3d2effdfa693c80a77e8d
07:59:13:I1: Branch: master
07:59:13:I1: Compiler: GNU 8.3.0
07:59:13:I1: Options: -faligned-new -std=c++17 -fsigned-char -ffunction-sections
07:59:13:I1: -fdata-sections -O3 -funroll-loops -fno-pie
07:59:13:I1: Platform: linux 4.19.0-26-cloud-amd64
07:59:13:I1: Bits: 64
07:59:13:I1: Mode: Release
07:59:13:I1: Args: --config=/etc/fah-client/config.xml
07:59:13:I1: --log=/var/log/fah-client/log.txt
07:59:13:I1: --log-rotate-dir=/var/log/fah-client/
07:59:13:I1: Config: /etc/fah-client/config.xml
07:59:13:I1:****************************** CBang ******************************
07:59:13:I1: Version: 1.7.2
07:59:13:I1: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:59:13:I1: Org: Cauldron Development LLC
07:59:13:I1: Copyright: Cauldron Development LLC, 2003-2024
07:59:13:I1: Homepage: https://cauldrondevelopment.com/
07:59:13:I1: License: LGPL-2.1-or-later
07:59:13:I1: Date: Feb 14 2024
07:59:13:I1: Time: 13:50:56
07:59:13:I1: Revision: 8f6145ef68f2c46b23df4487130e8f7e4fc8c757
07:59:13:I1: Branch: master
07:59:13:I1: Compiler: GNU 8.3.0
07:59:13:I1: Options: -faligned-new -std=c++17 -fsigned-char -ffunction-sections
07:59:13:I1: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
07:59:13:I1: Platform: linux 4.19.0-26-cloud-amd64
07:59:13:I1: Bits: 64
07:59:13:I1: Mode: Release
07:59:13:I1:***************************** System ******************************
07:59:13:I1: CPU: Intel(R) Core(TM) i9-14900KF
07:59:13:I1: CPU ID: GenuineIntel Family 6 Model 183 Stepping 1
07:59:13:I1: CPUs: 16
07:59:13:I1: Memory: 62.64GiB
07:59:13:I1:Free Memory: 58.59GiB
07:59:13:I1: OS Version: 6.8
07:59:13:I1:Has Battery: false
07:59:13:I1: On Battery: false
07:59:13:I1: Hostname: Black
07:59:13:I1: UTC Offset: 7
07:59:13:I1: PID: 7683
07:59:13:I1: CWD: /var/lib/fah-client
07:59:13:I1: Exec: /usr/bin/fah-client
07:59:13:I1:*******************************************************************
07:59:13:I2:<config/>
07:59:13:I1:Opening Database
07:59:13:I1:Generating RSA key................................................................................................................................................................................................................................++++-..............................................................................................................................................................................++++-
07:59:14:I1:F@H ID = Sinziowc6BhZymCSlcmqo_XilhIKjsnfK_Iaolkf7H0
07:59:14:I3:Loading default group
07:59:14:I3:Loading default resource group
07:59:14:I1:Listening for HTTP on 127.0.0.1:7396
07:59:14:I3:Loaded 0 wus.
07:59:14:I3:Connecting to api.foldingathome.org:443
07:59:14:I1:OUT1:> GET https://api.foldingathome.org/gpus HTTP/1.1
07:59:17:I1:OUT1:< HTTP/1.1 200 HTTP_OK
07:59:17:E :OpenCL not supported: Failed to open dynamic library 'libOpenCL.so': libOpenCL.so: cannot open shared object file: No such file or directory
07:59:17:I3:gpus = {
07:59:17:I3: "gpu:01:00:00": {
07:59:17:I3: "vendor": 4318,
07:59:17:I3: "type": "nvidia",
07:59:17:I3: "description": "NVIDIA GeForce RTX 4070 Ti",
07:59:17:I3: "uuid": "4dcd79b8-35f2-ff94-bec8-6bd9f20b5931",
07:59:17:I3: "cuda": {"platform": 0, "device": 0, "compute": "8.9", "driver": "12.2"},
07:59:17:I3: "device": 10114,
07:59:17:I3: "supported": true
07:59:17:I3: },
07:59:17:I3: "gpu:02:00:00": {
07:59:17:I3: "vendor": 4318,
07:59:17:I3: "type": "nvidia",
07:59:17:I3: "description": "NVIDIA GeForce RTX 4070 Ti",
07:59:17:I3: "uuid": "fc3a249f-b165-e42e-17ea-9ca1907a1f38",
07:59:17:I3: "cuda": {"platform": 0, "device": 1, "compute": "8.9", "driver": "12.2"},
07:59:17:I3: "device": 10114,
07:59:17:I3: "supported": true
07:59:17:I3: }
07:59:17:I3:}
08:02:31:I1:Default:Added new work unit: cpus:16 gpus:gpu:01:00:00,gpu:02:00:00
08:02:31:I1:WU1:Requesting WU assignment for user Anonymous team 0
08:02:32:I3:Connecting to assign1.foldingathome.org:443
08:02:32:I1:OUT3:> POST https://assign1.foldingathome.org/api/assign HTTP/1.1
08:02:33:I1:OUT3:< HTTP/1.1 200 HTTP_OK
08:02:33:I1:WU1:Received WU assignment 80oh126x09ea0fujWwgRSSw0zH1qtU9mIiaahACPgAM
08:02:33:I1:WU1:Downloading WU
08:02:33:I3:Connecting to highland3.seas.upenn.edu:443
08:02:33:I1:OUT4:> POST https://highland3.seas.upenn.edu/api/assign HTTP/1.1
08:02:33:E :Exception: Failed to prevent sleep: Permission denied
08:02:58:I3:Connecting to api.foldingathome.org:443
08:02:58:I1:OUT6:> PUT https://api.foldingathome.org/machine/Sinziowc6BhZymCSlcmqo_XilhIKjsnfK_Iaolkf7H0 HTTP/1.1
08:02:58:I1:WU1:DOWNLOAD 1% 310.17KiB of 31.23MiB
08:03:00:I1:OUT6:< HTTP/1.1 200 HTTP_OK
08:03:00:I1:Account linked
08:03:00:I3:Connecting to api.foldingathome.org:443
08:03:00:I1:OUT7:> GET https://api.foldingathome.org/machine/Sinziowc6BhZymCSlcmqo_XilhIKjsnfK_Iaolkf7H0 HTTP/1.1
08:03:01:I1:OUT7:< HTTP/1.1 200 HTTP_OK
08:03:01:I3:Connecting to node1.foldingathome.org:443
08:03:02:I1:OUT8:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
08:03:03:I1:OUT8:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
08:03:03:I1:Logging into node account
08:03:29:I1:WU1:DOWNLOAD 2% 628.30KiB of 31.23MiB
08:04:10:I1:WU1:DOWNLOAD 3% 954.38KiB of 31.23MiB
08:04:41:I1:WU1:DOWNLOAD 4% 1.23MiB of 31.23MiB
08:05:07:I1:WU1:DOWNLOAD 5% 1.57MiB of 31.23MiB
08:05:35:I1:WU1:DOWNLOAD 6% 1.86MiB of 31.23MiB
08:06:13:I1:WU1:DOWNLOAD 7% 2.18MiB of 31.23MiB
08:06:46:I1:WU1:DOWNLOAD 8% 2.49MiB of 31.23MiB
08:07:25:I1:WU1:DOWNLOAD 9% 2.80MiB of 31.23MiB
08:08:03:I1:WU1:DOWNLOAD 10% 3.11MiB of 31.23MiB
08:08:39:I1:WU1:DOWNLOAD 11% 3.43MiB of 31.23MiB
08:09:23:I1:WU1:DOWNLOAD 12% 3.74MiB of 31.23MiB
08:09:52:I1:WU1:DOWNLOAD 13% 4.05MiB of 31.23MiB
08:10:23:I1:WU1:DOWNLOAD 14% 4.36MiB of 31.23MiB
08:10:59:I1:WU1:DOWNLOAD 15% 4.68MiB of 31.23MiB
08:11:34:I1:WU1:DOWNLOAD 16% 4.99MiB of 31.23MiB
08:12:07:I1:WU1:DOWNLOAD 17% 5.30MiB of 31.23MiB
08:12:45:I1:WU1:DOWNLOAD 18% 5.61MiB of 31.23MiB
08:13:23:I1:WU1:DOWNLOAD 19% 5.92MiB of 31.23MiB
08:14:03:I1:WU1:DOWNLOAD 20% 6.24MiB of 31.23MiB
08:14:37:I1:WU1:DOWNLOAD 21% 6.56MiB of 31.23MiB
08:15:07:I1:WU1:DOWNLOAD 22% 6.86MiB of 31.23MiB
08:15:38:I1:WU1:DOWNLOAD 23% 7.17MiB of 31.23MiB
08:16:08:I1:WU1:DOWNLOAD 24% 7.50MiB of 31.23MiB
08:16:44:I1:WU1:DOWNLOAD 25% 7.80MiB of 31.23MiB
08:17:17:I1:WU1:DOWNLOAD 26% 8.12MiB of 31.23MiB
08:17:53:I1:WU1:DOWNLOAD 27% 8.43MiB of 31.23MiB
08:18:25:I1:WU1:DOWNLOAD 28% 8.75MiB of 31.23MiB
08:19:01:I1:WU1:DOWNLOAD 29% 9.09MiB of 31.23MiB
08:19:30:I1:WU1:DOWNLOAD 30% 9.37MiB of 31.23MiB
08:20:06:I1:WU1:DOWNLOAD 31% 9.67MiB of 31.23MiB
08:20:31:I1:WU1:DOWNLOAD 32% 9.98MiB of 31.23MiB
08:21:01:I1:WU1:DOWNLOAD 33% 10.29MiB of 31.23MiB
08:21:35:I1:WU1:DOWNLOAD 34% 10.61MiB of 31.23MiB
08:22:10:I1:WU1:DOWNLOAD 35% 10.92MiB of 31.23MiB
08:22:53:I1:WU1:DOWNLOAD 36% 11.23MiB of 31.23MiB
08:23:28:I1:WU1:DOWNLOAD 37% 11.55MiB of 31.23MiB
08:24:03:I1:WU1:DOWNLOAD 38% 11.85MiB of 31.23MiB
08:24:36:I1:WU1:DOWNLOAD 39% 12.17MiB of 31.23MiB
08:25:13:I1:WU1:DOWNLOAD 40% 12.48MiB of 31.23MiB
08:25:50:I1:WU1:DOWNLOAD 41% 12.79MiB of 31.23MiB
08:26:29:I1:WU1:DOWNLOAD 42% 13.10MiB of 31.23MiB
08:27:04:I1:WU1:DOWNLOAD 43% 13.42MiB of 31.23MiB
08:27:37:I1:WU1:DOWNLOAD 44% 13.73MiB of 31.23MiB
08:28:10:I1:WU1:DOWNLOAD 45% 14.04MiB of 31.23MiB
08:28:38:I1:WU1:DOWNLOAD 46% 14.35MiB of 31.23MiB
08:29:02:I1:WU1:DOWNLOAD 47% 14.66MiB of 31.23MiB
08:29:47:I1:WU1:DOWNLOAD 48% 14.99MiB of 31.23MiB
08:30:19:I1:WU1:DOWNLOAD 49% 15.30MiB of 31.23MiB
08:30:57:I1:WU1:DOWNLOAD 50% 15.61MiB of 31.23MiB
08:31:31:I1:WU1:DOWNLOAD 51% 15.92MiB of 31.23MiB
08:32:03:I1:WU1:DOWNLOAD 52% 16.22MiB of 31.23MiB
08:32:37:I1:WU1:DOWNLOAD 53% 16.54MiB of 31.23MiB
08:33:11:I1:WU1:DOWNLOAD 54% 16.86MiB of 31.23MiB
08:33:46:I1:WU1:DOWNLOAD 55% 17.16MiB of 31.23MiB
08:34:19:I1:WU1:DOWNLOAD 56% 17.48MiB of 31.23MiB
08:34:59:I1:WU1:DOWNLOAD 57% 17.82MiB of 31.23MiB
08:35:31:I1:WU1:DOWNLOAD 58% 18.14MiB of 31.23MiB
08:36:03:I1:WU1:DOWNLOAD 59% 18.42MiB of 31.23MiB
08:36:40:I1:WU1:DOWNLOAD 60% 18.73MiB of 31.23MiB
08:37:06:I1:WU1:DOWNLOAD 61% 19.04MiB of 31.23MiB
08:37:44:I1:WU1:DOWNLOAD 62% 19.35MiB of 31.23MiB
08:38:16:I1:WU1:DOWNLOAD 63% 19.67MiB of 31.23MiB
08:38:57:I1:WU1:DOWNLOAD 64% 19.98MiB of 31.23MiB
08:39:31:I1:WU1:DOWNLOAD 65% 20.29MiB of 31.23MiB
08:40:06:I1:WU1:DOWNLOAD 66% 20.61MiB of 31.23MiB
08:40:36:I1:WU1:DOWNLOAD 67% 20.92MiB of 31.23MiB
08:41:11:I1:WU1:DOWNLOAD 68% 21.23MiB of 31.23MiB
08:41:45:I1:WU1:DOWNLOAD 69% 21.54MiB of 31.23MiB
08:42:21:I1:WU1:DOWNLOAD 70% 21.86MiB of 31.23MiB
08:42:53:I1:WU1:DOWNLOAD 71% 22.16MiB of 31.23MiB
08:43:21:I1:WU1:DOWNLOAD 72% 22.48MiB of 31.23MiB
08:43:55:I1:WU1:DOWNLOAD 73% 22.80MiB of 31.23MiB
08:44:27:I1:WU1:DOWNLOAD 74% 23.10MiB of 31.23MiB
08:45:02:I1:WU1:DOWNLOAD 75% 23.42MiB of 31.23MiB
08:45:38:I1:WU1:DOWNLOAD 76% 23.72MiB of 31.23MiB
08:46:14:I1:WU1:DOWNLOAD 77% 24.05MiB of 31.23MiB
08:46:43:I1:WU1:DOWNLOAD 78% 24.36MiB of 31.23MiB
08:47:10:I1:WU1:DOWNLOAD 79% 24.66MiB of 31.23MiB
08:47:38:I1:WU1:DOWNLOAD 80% 24.97MiB of 31.23MiB
08:48:11:I1:WU1:DOWNLOAD 81% 25.28MiB of 31.23MiB
08:48:51:I1:WU1:DOWNLOAD 82% 25.60MiB of 31.23MiB
08:49:40:I1:WU1:DOWNLOAD 83% 25.91MiB of 31.23MiB
08:50:15:I1:WU1:DOWNLOAD 84% 26.23MiB of 31.23MiB
08:50:47:I1:WU1:DOWNLOAD 85% 26.53MiB of 31.23MiB
08:51:13:I1:WU1:DOWNLOAD 86% 26.85MiB of 31.23MiB
08:51:43:I1:WU1:DOWNLOAD 87% 27.16MiB of 31.23MiB
08:52:15:I1:WU1:DOWNLOAD 88% 27.47MiB of 31.23MiB
08:52:40:I1:WU1:DOWNLOAD 89% 27.80MiB of 31.23MiB
08:53:18:I1:WU1:DOWNLOAD 90% 28.11MiB of 31.23MiB
08:53:45:I1:WU1:DOWNLOAD 91% 28.42MiB of 31.23MiB
08:54:21:I1:WU1:DOWNLOAD 92% 28.72MiB of 31.23MiB
08:54:54:I1:WU1:DOWNLOAD 93% 29.03MiB of 31.23MiB
08:55:25:I1:WU1:DOWNLOAD 94% 29.34MiB of 31.23MiB
08:56:05:I1:WU1:DOWNLOAD 95% 29.65MiB of 31.23MiB
08:56:47:I1:WU1:DOWNLOAD 96% 29.97MiB of 31.23MiB
08:57:24:I1:WU1:DOWNLOAD 97% 30.28MiB of 31.23MiB
08:57:59:I1:WU1:DOWNLOAD 98% 30.59MiB of 31.23MiB
08:58:35:I1:WU1:DOWNLOAD 99% 30.90MiB of 31.23MiB
08:59:08:I1:WU1:DOWNLOAD 100% 31.23MiB of 31.23MiB
08:59:08:I1:OUT4:< HTTP/1.1 200 HTTP_OK
08:59:08:I1:WU1:Received WU
08:59:08:I1:Downloading https://cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3.tar.bz2.crt
08:59:09:I3:Connecting to cores.foldingathome.org:443
08:59:09:I1:OUT13:> GET https://cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3.tar.bz2.crt HTTP/1.1
08:59:16:I1:OUT13:< HTTP/1.1 200 OK
08:59:16:I1:Downloading https://cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3.tar.bz2.sig
08:59:16:I3:Connecting to cores.foldingathome.org:443
08:59:16:I1:OUT14:> GET https://cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3.tar.bz2.sig HTTP/1.1
08:59:20:I1:OUT14:< HTTP/1.1 200 OK
08:59:20:I1:Downloading https://cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3.tar.bz2
08:59:20:I3:Connecting to cores.foldingathome.org:443
08:59:20:I1:OUT15:> GET https://cores.foldingathome.org/openmm-core-23/centos-7.9.2009-64bit/release/fahcore-23-centos-7.9.2009-64bit-release-8.0.3.tar.bz2 HTTP/1.1
09:04:21:I1:WU1:CORE 1% 1.53MiB of 159.89MiB
09:09:42:I1:WU1:CORE 2% 3.12MiB of 159.89MiB

That is one out of 3 machines experiencing the same issues.

My internet connection is rock solid, see below test I just ran:

The only thing I can think of is the fact that I am in Indonesia, and there may be some geo-restrictions?

Ideas / thoughts / knowledge will be highly appreciated.

enroscado · Post by **enroscado** » Wed May 22, 2024 10:30 am

UPDATE: Indeed, there seems to be some limitation with location.

Running a VPN on all 3 machines, trying different countries (US, Singapore, Thailand, etc.) and the problem is gone.

For now it should do, however, any more details on why this restriction?

Post by **jjmiller** » Wed May 22, 2024 2:06 pm

Hi enroscado,

Are you seeing this just with the highlandX servers or more broadly across all servers? Happy to bring this up to the folks who manage our (UPenn's) networks if it's exclusively happening with our servers.

Thanks!

enroscado · Post by **enroscado** » Wed May 22, 2024 2:14 pm

Hi jjmiller,

The way it has been behaving lately, it seems to be across the board. Regardless, I'll keep an eye on specific servers and report back here.

Thank you.

Post by **jjmiller** » Tue Jun 04, 2024 6:32 pm

Hi enroscado! Wanted to check back - has this issue resolved? Also wanted to see, has this been a persistent issue or is it relatively recent e.g. since you first posted (4/22/24)?

Thanks!

enroscado · Post by **enroscado** » Wed Jun 05, 2024 1:55 am

Hi jjmiller,

The issue remains. It seems to be due to me being in Indonesia, and it applies to any server, not just Upenn's.

Look at the screenshot below:

It is always the same setup.

May 23rd I started using a VPN, and you can see the consistency that came with it; basically WUs download and upload in seconds, as it should be.

(May 30th for some reason one of the VPNs disconnected, hence the drop in points).

So, conclusion, it seems there is some geo-restriction (or problem) with Indonesian IPs.

Post by **jjmiller** » Wed Jun 05, 2024 2:44 pm

Thanks! Totally agree with you that it seems like an issue in coming from an Indonesian IP. I checked in with our software engineer(/networking guru) and we don't have any geo-restrictions in place on our end, maybe there are issues along the way somewhere though?

It seems like the VPN is doing a good job in circumventing the problem!

Folding Forum

Impossibly slow servers

Impossibly slow servers

Re: Impossibly slow servers

Re: Impossibly slow servers

Re: Impossibly slow servers

Re: Impossibly slow servers

Re: Impossibly slow servers

Re: Impossibly slow servers