Page 1 of 2
Project 18201 not uploading
Posted: Fri Sep 17, 2021 2:20 pm
by smurfcorpse
I'm not sure if this is a server issue as the status page shows the servers 128.252.203.11 and 128.252.203.14 are accepting WUs. This has been trying to upload since yesterday.
Code: Select all
13:47:05:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18201 run:1140 clone:2 gen:10 core:0x22 unit:0x000000020000000a0000471900000474
13:47:05:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
13:47:05:WU02:FS01:Connecting to 128.252.203.11:8080
13:47:14:WU02:FS01:Upload 0.45%
13:47:33:WU02:FS01:Upload 1.14%
13:47:33:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:47:33:WU02:FS01:Trying to send results to collection server
13:47:33:WU02:FS01:Uploading 27.50MiB to 128.252.203.14
13:47:33:WU02:FS01:Connecting to 128.252.203.14:8080
13:47:53:WU02:FS01:Upload 1.14%
13:40:55:ERROR:WU02:FS01:Exception: Transfer failed
Other work units are being turned in to other servers fine. I have tried to disable the firewall and I have tried a VPN. This WU took a long time to complete and I hate loosing points on it.
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 8:03 pm
by smurfcorpse
I thought I would have heard something back by now. So far 42 failed attempts to upload. I did a ping test from home to the two IP addresses and those were successful. Am I the only one having this issue?
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 8:53 pm
by toTOW
The error message isn't really helpful ... and there's not other reports here, so I guess you're the only one having issues ...
Is there anything weird in your network that could interfere ?
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 8:57 pm
by psaam0001
Try checking your internet security software/router firewall settings to verify that the F@H (Folding at Home) client is not restricted from using ports 80 and 8080. These appear to be the two primary TCP/IP ports that the client software needs to access the internet. Should you be using your employers internet bandwidth, confirm with their IT Manager that they have not restricted use of the F@H software for any reason.
Someone with more experience will be able to help you further. But we will need a little more information from your system log to help us. So please post the most recent 200 lines, starting with what is shown below:
Code: Select all
19:58:11:WU02:FS00:Starting
19:58:11:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit-avx2-256/a8-0.0.12/Core_a8.fah/FahCore_a8.exe -dir 02 -suffix 01 -version 706 -lifeline 3524 -checkpoint 30 -np 2
19:58:11:WU02:FS00:Started FahCore on PID 7516
19:58:11:WU02:FS00:Core PID:6116
19:58:11:WU02:FS00:FahCore 0xa8 started
19:58:12:WU02:FS00:0xa8:*********************** Log Started 2021-09-17T19:58:11Z ***********************
19:58:12:WU02:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
19:58:12:WU02:FS00:0xa8: Core: Gromacs
19:58:12:WU02:FS00:0xa8: Type: 0xa8
19:58:12:WU02:FS00:0xa8: Version: 0.0.12
19:58:12:WU02:FS00:0xa8: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:58:12:WU02:FS00:0xa8: Copyright: 2020 foldingathome.org
19:58:12:WU02:FS00:0xa8: Homepage: https://foldingathome.org/
19:58:12:WU02:FS00:0xa8: Date: Jan 16 2021
19:58:12:WU02:FS00:0xa8: Time: 12:29:40
19:58:12:WU02:FS00:0xa8: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
19:58:12:WU02:FS00:0xa8: Branch: master
19:58:12:WU02:FS00:0xa8: Compiler: Visual C++ 2019 16.7
19:58:12:WU02:FS00:0xa8: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:58:12:WU02:FS00:0xa8: Platform: win32 10
19:58:12:WU02:FS00:0xa8: Bits: 64
19:58:12:WU02:FS00:0xa8: Mode: Release
19:58:12:WU02:FS00:0xa8: SIMD: avx2_256
19:58:12:WU02:FS00:0xa8: OpenMP: ON
19:58:12:WU02:FS00:0xa8: CUDA: OFF
19:58:12:WU02:FS00:0xa8: Args: -dir 02 -suffix 01 -version 706 -lifeline 7516 -checkpoint 30 -np 2
19:58:12:WU02:FS00:0xa8:************************************ libFAH ************************************
19:58:12:WU02:FS00:0xa8: Date: Jan 16 2021
19:58:12:WU02:FS00:0xa8: Time: 11:24:13
19:58:12:WU02:FS00:0xa8: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
19:58:12:WU02:FS00:0xa8: Branch: master
19:58:12:WU02:FS00:0xa8: Compiler: Visual C++ 2019 16.7
19:58:12:WU02:FS00:0xa8: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:58:12:WU02:FS00:0xa8: Platform: win32 10
19:58:12:WU02:FS00:0xa8: Bits: 64
19:58:12:WU02:FS00:0xa8: Mode: Release
19:58:12:WU02:FS00:0xa8:************************************ CBang *************************************
19:58:12:WU02:FS00:0xa8: Date: Jan 16 2021
19:58:12:WU02:FS00:0xa8: Time: 11:23:53
19:58:12:WU02:FS00:0xa8: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
19:58:12:WU02:FS00:0xa8: Branch: master
19:58:12:WU02:FS00:0xa8: Compiler: Visual C++ 2019 16.7
19:58:12:WU02:FS00:0xa8: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:58:12:WU02:FS00:0xa8: Platform: win32 10
19:58:12:WU02:FS00:0xa8: Bits: 64
19:58:12:WU02:FS00:0xa8: Mode: Release
19:58:12:WU02:FS00:0xa8:************************************ System ************************************
19:58:12:WU02:FS00:0xa8: CPU: AMD Ryzen 3 3200G with Radeon Vega Graphics
19:58:12:WU02:FS00:0xa8: CPU ID: AuthenticAMD Family 23 Model 24 Stepping 1
19:58:12:WU02:FS00:0xa8: CPUs: 4
19:58:12:WU02:FS00:0xa8: Memory: 13.93GiB
19:58:12:WU02:FS00:0xa8:Free Memory: 9.20GiB
19:58:12:WU02:FS00:0xa8: Threads: WINDOWS_THREADS
19:58:12:WU02:FS00:0xa8: OS Version: 6.2
19:58:12:WU02:FS00:0xa8:Has Battery: false
19:58:12:WU02:FS00:0xa8: On Battery: false
19:58:12:WU02:FS00:0xa8: UTC Offset: -4
19:58:12:WU02:FS00:0xa8: PID: 6116
19:58:12:WU02:FS00:0xa8: CWD: C:\ProgramData\FAHClient\work
19:58:12:WU02:FS00:0xa8:********************************************************************************
19:58:12:WU02:FS00:0xa8:Project: 18206 (Run 3099, Clone 6, Gen 5)
19:58:12:WU02:FS00:0xa8:Unit: 0x00000000000000000000000000000000
19:58:12:WU02:FS00:0xa8:Reading tar file core.xml
19:58:12:WU02:FS00:0xa8:Reading tar file frame5.tpr
19:58:12:WU02:FS00:0xa8:Digital signatures verified
19:58:12:WU02:FS00:0xa8:Calling: mdrun -c frame5.gro -s frame5.tpr -x frame5.xtc -cpt 30 -nt 2 -ntmpi 1
19:58:12:WU02:FS00:0xa8:Steps: first=625000 total=750000
19:58:14:WU02:FS00:0xa8:Completed 1 out of 125000 steps (0%)
20:06:19:WU02:FS00:0xa8:Completed 1250 out of 125000 steps (1%)
20:13:52:WU02:FS00:0xa8:Completed 2500 out of 125000 steps (2%)
20:16:28:WU00:FS02:0x22:Completed 50000 out of 2500000 steps (2%)
20:21:14:WU02:FS00:0xa8:Completed 3750 out of 125000 steps (3%)
20:21:52:WU03:FS01:0x22:Completed 650000 out of 5000000 steps (13%)
20:28:37:WU02:FS00:0xa8:Completed 5000 out of 125000 steps (4%)
20:36:00:WU02:FS00:0xa8:Completed 6250 out of 125000 steps (5%)
20:43:23:WU02:FS00:0xa8:Completed 7500 out of 125000 steps (6%)
20:47:11:WU03:FS01:0x22:Completed 700000 out of 5000000 steps (14%)
20:50:06:WU00:FS02:0x22:Completed 75000 out of 2500000 steps (3%)
20:50:46:WU02:FS00:0xa8:Completed 8750 out of 125000 steps (7%)
Do not filter by slot/unit or warnings/errors.
Paul
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 9:07 pm
by jjmiller
Hi smurfcorpse,
I took a peek through the logs on our collection and work servers- I see the work server assigning the WU to you, but none of the servers have recorded any return attempts. Both servers are currently recording other WUs being returned. I've issued a restart to both servers in the event something is stuck for some reason.
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 9:15 pm
by smurfcorpse
Hopefully that restart resolves the issue. I haven't made any changes to the network in years and folding has always worked. The PC giving me the error is also running a CPU client and it uploads OK. I have 2 PCs folding on the same network and the other hasn't had any issues uploading. I'm kinda stumped. I will report back once I'm at home.
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 10:15 pm
by psaam0001
My apologies to smurfcorpse, for my earlier presumptions. Given that I saw the low post count.
Though, I am glad that someone with access to the relevant servers could help.
Paul
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 10:27 pm
by smurfcorpse
Thanks for issuing a restart for me but it did not help. I even went so far as to use my cell phones as a hot spot to see of it would make any difference.
@ psaam0001 No worries. I am looking for anything that would cause this issue. Any help is always appreciated.
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 11:13 pm
by psaam0001
One last quick thing to check is for O/S updates for the computer that is not uploading...
Do you have a spare network card or wireless adapter that has drivers for your O/S on the computer that is not uploading? If so, give it a try. Maybe try a known working Cat. 5 cable if it is cable connected to the modem. It might be something simple that gets overlooked.
Paul
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 11:24 pm
by smurfcorpse
The PC has a WiFi adapter that is normally disabled. I enabled it to test with my phones hot spot with same result. The PC is Windows 7 so no updates. Disabled anti virus and firewall temporarily with no luck. Seems like the fact it uploads 1.14% means that it makes a connection to the server and the server disconnects... maybe? Do you think older OS is related?
Re: Project 18201 not uploading
Posted: Fri Sep 17, 2021 11:56 pm
by psaam0001
I would have to let one of the more experienced members of the forum answer that. As my current folding machines are running either the most current Fedora Linux, or Windows 10 Home.
Paul
Re: Project 18201 not uploading
Posted: Sun Sep 19, 2021 7:18 pm
by jjmiller
Hi smurfcorpse,
Drat- the restart was my last trick on our end given the paucity of troubleshooting information in the log files. It seems like we have some weird interactions between Win7 and the highland servers when users are utilizing certain AV programs. However, I don't think this falls exactly into that phenotype as other AV associated errors as those tend to have their WUs upload but not receive a notification that the WU has been uploaded. Have you had issues with 18201/18202 WUs in the past, or is this the first that has given you an issue?
I'll raise this to more experienced members on our end.
Apologies,
Re: Project 18201 not uploading
Posted: Sun Sep 19, 2021 7:22 pm
by Neil-B
If the WU actually uploaded on the first attempt (check the WU Status web app) which it may well have done but the client didn't receive the correct response (can be caused by packet inspection by various A/V) it sometimes keeps retrying ... Check to see if it has been uploaded and if so then delete the WU folder - if you need help with that ask.
Re: Project 18201 not uploading
Posted: Mon Sep 20, 2021 1:51 pm
by smurfcorpse
I gave it the weekend to see but the project still hasn't uploaded.
@ Neil-B, Based on the logs, the project ended and the results never uploaded.
Just to note. I searched the old logs for that project number. That PC has folded that project before and returned the results successfully. I saw a few entries going back about a week from the 16th. No hardware changes, network changes or antivirus (Avira) changes in that time. The points are probably lost at this point. Oh well stuff happens I guess but I wish we knew the cause so it doesn't happen again.
Re: Project 18201 not uploading
Posted: Mon Sep 20, 2021 2:49 pm
by Neil-B
@smurfcorpse ... the logs can lie - that is why I suggested you check the WU Status App which shows what the server has received
...
https://apps.foldingathome.org/wu
A/V are constantly changing - they are designed to ... all it takes is a signature in the A/V update for the heuristics/deep packet inspection to match an element of the content in a specific WU and this type of thing could happen - it might even just be delaying the response long enough that the client assumes it isn't getting one ... It is possible it was a transient glitch with the server or indeed any part of the comms path between the server and your kit ... but the pattern tends to be (from the perspective of the logs) that the first upload gets to a high 90 something percent and then seems to not complete with a short response message (or some such) in the logs - this doesn't necessarily mean the WU wasn't received by the Work/Collection Server just that the client on your kit hasn't/didn't receive the correct response so that it knows it has been received - it therefore keeps trying.
If you get these type of issues then a number of A/V products have been strongly associated with them (but really hard to prove) and the DPI part seems to be the one that causes issues (again very hard to prove ... A quick check of the WU Status App will confirm (allowing time for the stats to update) whether the upload was actually successful - if it was then the WU folder for the WU in question can be deleted (ask for help if you need on this).
These cases happen sporadically in clusters - another folder has reported similar in the last few days ... if "loads" of people report the issue is leans to there being a problem with the server ... If only a few then it can be ISP, A/V (I deliberately don't mention the two regular culprits), "any other guess" ... but whatever the cause it is worth checking WU status app as if the WU has uploaded then there is no point in keeping th wu trying to return it as the server simply wo't accept it as it already has it !!
If the WU hasn't uploaded from yourself then this issue may be the same issue in reverse with the av/dpi or any of the intermediaries altering the packets on their way to the server - where this happens a lot then the issues tend to get diagnosed - when it is simply the occasional wu then tracking down and resolving the issues becomes much harder.