Project 18201 not uploading

Moderators: Site Moderators, FAHC Science Team

smurfcorpse
Posts: 16
Joined: Wed Dec 16, 2020 3:30 am

Project 18201 not uploading

Post by smurfcorpse »

I'm not sure if this is a server issue as the status page shows the servers 128.252.203.11 and 128.252.203.14 are accepting WUs. This has been trying to upload since yesterday.

Code: Select all

13:47:05:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18201 run:1140 clone:2 gen:10 core:0x22 unit:0x000000020000000a0000471900000474
13:47:05:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
13:47:05:WU02:FS01:Connecting to 128.252.203.11:8080
13:47:14:WU02:FS01:Upload 0.45%
13:47:33:WU02:FS01:Upload 1.14%
13:47:33:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:47:33:WU02:FS01:Trying to send results to collection server
13:47:33:WU02:FS01:Uploading 27.50MiB to 128.252.203.14
13:47:33:WU02:FS01:Connecting to 128.252.203.14:8080
13:47:53:WU02:FS01:Upload 1.14%
13:40:55:ERROR:WU02:FS01:Exception: Transfer failed
Other work units are being turned in to other servers fine. I have tried to disable the firewall and I have tried a VPN. This WU took a long time to complete and I hate loosing points on it.
smurfcorpse
Posts: 16
Joined: Wed Dec 16, 2020 3:30 am

Re: Project 18201 not uploading

Post by smurfcorpse »

I thought I would have heard something back by now. So far 42 failed attempts to upload. I did a ping test from home to the two IP addresses and those were successful. Am I the only one having this issue?
toTOW
Site Moderator
Posts: 6296
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:

Re: Project 18201 not uploading

Post by toTOW »

The error message isn't really helpful ... and there's not other reports here, so I guess you're the only one having issues ...

Is there anything weird in your network that could interfere ?
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
psaam0001
Posts: 383
Joined: Mon May 18, 2020 2:02 am
Location: Ruckersville, Virginia, USA

Re: Project 18201 not uploading

Post by psaam0001 »

Try checking your internet security software/router firewall settings to verify that the F@H (Folding at Home) client is not restricted from using ports 80 and 8080. These appear to be the two primary TCP/IP ports that the client software needs to access the internet. Should you be using your employers internet bandwidth, confirm with their IT Manager that they have not restricted use of the F@H software for any reason.

Someone with more experience will be able to help you further. But we will need a little more information from your system log to help us. So please post the most recent 200 lines, starting with what is shown below:

Code: Select all

19:58:11:WU02:FS00:Starting
19:58:11:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit-avx2-256/a8-0.0.12/Core_a8.fah/FahCore_a8.exe -dir 02 -suffix 01 -version 706 -lifeline 3524 -checkpoint 30 -np 2
19:58:11:WU02:FS00:Started FahCore on PID 7516
19:58:11:WU02:FS00:Core PID:6116
19:58:11:WU02:FS00:FahCore 0xa8 started
19:58:12:WU02:FS00:0xa8:*********************** Log Started 2021-09-17T19:58:11Z ***********************
19:58:12:WU02:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
19:58:12:WU02:FS00:0xa8:       Core: Gromacs
19:58:12:WU02:FS00:0xa8:       Type: 0xa8
19:58:12:WU02:FS00:0xa8:    Version: 0.0.12
19:58:12:WU02:FS00:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:58:12:WU02:FS00:0xa8:  Copyright: 2020 foldingathome.org
19:58:12:WU02:FS00:0xa8:   Homepage: https://foldingathome.org/
19:58:12:WU02:FS00:0xa8:       Date: Jan 16 2021
19:58:12:WU02:FS00:0xa8:       Time: 12:29:40
19:58:12:WU02:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
19:58:12:WU02:FS00:0xa8:     Branch: master
19:58:12:WU02:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
19:58:12:WU02:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:58:12:WU02:FS00:0xa8:   Platform: win32 10
19:58:12:WU02:FS00:0xa8:       Bits: 64
19:58:12:WU02:FS00:0xa8:       Mode: Release
19:58:12:WU02:FS00:0xa8:       SIMD: avx2_256
19:58:12:WU02:FS00:0xa8:     OpenMP: ON
19:58:12:WU02:FS00:0xa8:       CUDA: OFF
19:58:12:WU02:FS00:0xa8:       Args: -dir 02 -suffix 01 -version 706 -lifeline 7516 -checkpoint 30 -np 2
19:58:12:WU02:FS00:0xa8:************************************ libFAH ************************************
19:58:12:WU02:FS00:0xa8:       Date: Jan 16 2021
19:58:12:WU02:FS00:0xa8:       Time: 11:24:13
19:58:12:WU02:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
19:58:12:WU02:FS00:0xa8:     Branch: master
19:58:12:WU02:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
19:58:12:WU02:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:58:12:WU02:FS00:0xa8:   Platform: win32 10
19:58:12:WU02:FS00:0xa8:       Bits: 64
19:58:12:WU02:FS00:0xa8:       Mode: Release
19:58:12:WU02:FS00:0xa8:************************************ CBang *************************************
19:58:12:WU02:FS00:0xa8:       Date: Jan 16 2021
19:58:12:WU02:FS00:0xa8:       Time: 11:23:53
19:58:12:WU02:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
19:58:12:WU02:FS00:0xa8:     Branch: master
19:58:12:WU02:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
19:58:12:WU02:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:58:12:WU02:FS00:0xa8:   Platform: win32 10
19:58:12:WU02:FS00:0xa8:       Bits: 64
19:58:12:WU02:FS00:0xa8:       Mode: Release
19:58:12:WU02:FS00:0xa8:************************************ System ************************************
19:58:12:WU02:FS00:0xa8:        CPU: AMD Ryzen 3 3200G with Radeon Vega Graphics
19:58:12:WU02:FS00:0xa8:     CPU ID: AuthenticAMD Family 23 Model 24 Stepping 1
19:58:12:WU02:FS00:0xa8:       CPUs: 4
19:58:12:WU02:FS00:0xa8:     Memory: 13.93GiB
19:58:12:WU02:FS00:0xa8:Free Memory: 9.20GiB
19:58:12:WU02:FS00:0xa8:    Threads: WINDOWS_THREADS
19:58:12:WU02:FS00:0xa8: OS Version: 6.2
19:58:12:WU02:FS00:0xa8:Has Battery: false
19:58:12:WU02:FS00:0xa8: On Battery: false
19:58:12:WU02:FS00:0xa8: UTC Offset: -4
19:58:12:WU02:FS00:0xa8:        PID: 6116
19:58:12:WU02:FS00:0xa8:        CWD: C:\ProgramData\FAHClient\work
19:58:12:WU02:FS00:0xa8:********************************************************************************
19:58:12:WU02:FS00:0xa8:Project: 18206 (Run 3099, Clone 6, Gen 5)
19:58:12:WU02:FS00:0xa8:Unit: 0x00000000000000000000000000000000
19:58:12:WU02:FS00:0xa8:Reading tar file core.xml
19:58:12:WU02:FS00:0xa8:Reading tar file frame5.tpr
19:58:12:WU02:FS00:0xa8:Digital signatures verified
19:58:12:WU02:FS00:0xa8:Calling: mdrun -c frame5.gro -s frame5.tpr -x frame5.xtc -cpt 30 -nt 2 -ntmpi 1
19:58:12:WU02:FS00:0xa8:Steps: first=625000 total=750000
19:58:14:WU02:FS00:0xa8:Completed 1 out of 125000 steps (0%)
20:06:19:WU02:FS00:0xa8:Completed 1250 out of 125000 steps (1%)
20:13:52:WU02:FS00:0xa8:Completed 2500 out of 125000 steps (2%)
20:16:28:WU00:FS02:0x22:Completed 50000 out of 2500000 steps (2%)
20:21:14:WU02:FS00:0xa8:Completed 3750 out of 125000 steps (3%)
20:21:52:WU03:FS01:0x22:Completed 650000 out of 5000000 steps (13%)
20:28:37:WU02:FS00:0xa8:Completed 5000 out of 125000 steps (4%)
20:36:00:WU02:FS00:0xa8:Completed 6250 out of 125000 steps (5%)
20:43:23:WU02:FS00:0xa8:Completed 7500 out of 125000 steps (6%)
20:47:11:WU03:FS01:0x22:Completed 700000 out of 5000000 steps (14%)
20:50:06:WU00:FS02:0x22:Completed 75000 out of 2500000 steps (3%)
20:50:46:WU02:FS00:0xa8:Completed 8750 out of 125000 steps (7%)
Do not filter by slot/unit or warnings/errors.

Paul
jjmiller
Scientist
Posts: 81
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project 18201 not uploading

Post by jjmiller »

Hi smurfcorpse,

I took a peek through the logs on our collection and work servers- I see the work server assigning the WU to you, but none of the servers have recorded any return attempts. Both servers are currently recording other WUs being returned. I've issued a restart to both servers in the event something is stuck for some reason.
smurfcorpse
Posts: 16
Joined: Wed Dec 16, 2020 3:30 am

Re: Project 18201 not uploading

Post by smurfcorpse »

Hopefully that restart resolves the issue. I haven't made any changes to the network in years and folding has always worked. The PC giving me the error is also running a CPU client and it uploads OK. I have 2 PCs folding on the same network and the other hasn't had any issues uploading. I'm kinda stumped. I will report back once I'm at home.
psaam0001
Posts: 383
Joined: Mon May 18, 2020 2:02 am
Location: Ruckersville, Virginia, USA

Re: Project 18201 not uploading

Post by psaam0001 »

My apologies to smurfcorpse, for my earlier presumptions. Given that I saw the low post count.

Though, I am glad that someone with access to the relevant servers could help.

Paul
smurfcorpse
Posts: 16
Joined: Wed Dec 16, 2020 3:30 am

Re: Project 18201 not uploading

Post by smurfcorpse »

Thanks for issuing a restart for me but it did not help. I even went so far as to use my cell phones as a hot spot to see of it would make any difference.

@ psaam0001 No worries. I am looking for anything that would cause this issue. Any help is always appreciated.
psaam0001
Posts: 383
Joined: Mon May 18, 2020 2:02 am
Location: Ruckersville, Virginia, USA

Re: Project 18201 not uploading

Post by psaam0001 »

One last quick thing to check is for O/S updates for the computer that is not uploading...

Do you have a spare network card or wireless adapter that has drivers for your O/S on the computer that is not uploading? If so, give it a try. Maybe try a known working Cat. 5 cable if it is cable connected to the modem. It might be something simple that gets overlooked.

Paul
smurfcorpse
Posts: 16
Joined: Wed Dec 16, 2020 3:30 am

Re: Project 18201 not uploading

Post by smurfcorpse »

The PC has a WiFi adapter that is normally disabled. I enabled it to test with my phones hot spot with same result. The PC is Windows 7 so no updates. Disabled anti virus and firewall temporarily with no luck. Seems like the fact it uploads 1.14% means that it makes a connection to the server and the server disconnects... maybe? Do you think older OS is related?
psaam0001
Posts: 383
Joined: Mon May 18, 2020 2:02 am
Location: Ruckersville, Virginia, USA

Re: Project 18201 not uploading

Post by psaam0001 »

I would have to let one of the more experienced members of the forum answer that. As my current folding machines are running either the most current Fedora Linux, or Windows 10 Home.

Paul
jjmiller
Scientist
Posts: 81
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project 18201 not uploading

Post by jjmiller »

Hi smurfcorpse,

Drat- the restart was my last trick on our end given the paucity of troubleshooting information in the log files. It seems like we have some weird interactions between Win7 and the highland servers when users are utilizing certain AV programs. However, I don't think this falls exactly into that phenotype as other AV associated errors as those tend to have their WUs upload but not receive a notification that the WU has been uploaded. Have you had issues with 18201/18202 WUs in the past, or is this the first that has given you an issue?

I'll raise this to more experienced members on our end.

Apologies,
Last edited by jjmiller on Mon Sep 20, 2021 9:48 pm, edited 1 time in total.
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Project 18201 not uploading

Post by Neil-B »

If the WU actually uploaded on the first attempt (check the WU Status web app) which it may well have done but the client didn't receive the correct response (can be caused by packet inspection by various A/V) it sometimes keeps retrying ... Check to see if it has been uploaded and if so then delete the WU folder - if you need help with that ask.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
smurfcorpse
Posts: 16
Joined: Wed Dec 16, 2020 3:30 am

Re: Project 18201 not uploading

Post by smurfcorpse »

I gave it the weekend to see but the project still hasn't uploaded.

@ Neil-B, Based on the logs, the project ended and the results never uploaded.

Just to note. I searched the old logs for that project number. That PC has folded that project before and returned the results successfully. I saw a few entries going back about a week from the 16th. No hardware changes, network changes or antivirus (Avira) changes in that time. The points are probably lost at this point. Oh well stuff happens I guess but I wish we knew the cause so it doesn't happen again.
Neil-B
Posts: 2027
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Project 18201 not uploading

Post by Neil-B »

@smurfcorpse ... the logs can lie - that is why I suggested you check the WU Status App which shows what the server has received :) ... https://apps.foldingathome.org/wu

A/V are constantly changing - they are designed to ... all it takes is a signature in the A/V update for the heuristics/deep packet inspection to match an element of the content in a specific WU and this type of thing could happen - it might even just be delaying the response long enough that the client assumes it isn't getting one ... It is possible it was a transient glitch with the server or indeed any part of the comms path between the server and your kit ... but the pattern tends to be (from the perspective of the logs) that the first upload gets to a high 90 something percent and then seems to not complete with a short response message (or some such) in the logs - this doesn't necessarily mean the WU wasn't received by the Work/Collection Server just that the client on your kit hasn't/didn't receive the correct response so that it knows it has been received - it therefore keeps trying.

If you get these type of issues then a number of A/V products have been strongly associated with them (but really hard to prove) and the DPI part seems to be the one that causes issues (again very hard to prove ... A quick check of the WU Status App will confirm (allowing time for the stats to update) whether the upload was actually successful - if it was then the WU folder for the WU in question can be deleted (ask for help if you need on this).

These cases happen sporadically in clusters - another folder has reported similar in the last few days ... if "loads" of people report the issue is leans to there being a problem with the server ... If only a few then it can be ISP, A/V (I deliberately don't mention the two regular culprits), "any other guess" ... but whatever the cause it is worth checking WU status app as if the WU has uploaded then there is no point in keeping th wu trying to return it as the server simply wo't accept it as it already has it !!

If the WU hasn't uploaded from yourself then this issue may be the same issue in reverse with the av/dpi or any of the intermediaries altering the packets on their way to the server - where this happens a lot then the issues tend to get diagnosed - when it is simply the occasional wu then tracking down and resolving the issues becomes much harder.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Post Reply