Project: 10001 (Run 86, Clone 4, Gen 22)

[Tynat]

Returned to the computer and found that 41 minutes ago the CPU Console client had choked on project 10001 (Run 86, Clone 4, Gen 22) and was awaiting me to click OK on an error box with bad punctuation:

"Folding@home has run into a serious error running the core. and will shutdown."

Why that's rather sporting of it. However, my attention to the error wasn't necessary and it would have been more helpful to not kill the client and instead go download a different WU. Especially in light of the following:

Code: Select all

[18:48:20] Project: 10001 (Run 86, Clone 4, Gen 22)
[18:48:20] Reading tar file par_all27_prot_lipid.inp
[18:48:20] Reading tar file scpismQuartic.inp
[18:48:20] Reading tar file ww_exteq_nowater1.pdb
[18:48:20] Reading tar file ww_exteq_nowater1.psf
[18:48:20] Reading tar file checkpt
[18:48:20] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[18:48:20] Folding@home Core Shutdown: EARLY_UNIT_END
[18:48:24] CoreStatus = 79 (121)
[18:48:24] Client-core communications error: ERROR 0x79
[18:48:24] This is a sign of more serious problems, shutting down.

Closed off error dialog box and restarted.

Code: Select all

[19:29:42] Project: 10001 (Run 86, Clone 4, Gen 22)
[19:29:42] Reading tar file par_all27_prot_lipid.inp
[19:29:42] Reading tar file scpismQuartic.inp
[19:29:42] Reading tar file ww_exteq_nowater1.pdb
[19:29:42] Reading tar file ww_exteq_nowater1.psf
[19:29:42] Reading tar file checkpt
[19:29:42] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[19:29:42] Folding@home Core Shutdown: EARLY_UNIT_END
[19:29:46] CoreStatus = 79 (121)
[19:29:46] Client-core communications error: ERROR 0x79
[19:29:46] This is a sign of more serious problems, shutting down.

Closed off error dialog box and restarted.

Code: Select all

[19:36:15] Project: 10001 (Run 86, Clone 4, Gen 22)
[19:36:15] Reading tar file par_all27_prot_lipid.inp
[19:36:15] Reading tar file scpismQuartic.inp
[19:36:16] Reading tar file ww_exteq_nowater1.pdb
[19:36:16] Reading tar file ww_exteq_nowater1.psf
[19:36:16] Reading tar file checkpt
[19:36:16] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[19:36:16] Folding@home Core Shutdown: EARLY_UNIT_END
[19:36:19] CoreStatus = 79 (121)
[19:36:19] Client-core communications error: ERROR 0x79
[19:36:19] This is a sign of more serious problems, shutting down.

Closed off error dialog box and restarted.

Code: Select all

[19:37:48] Project: 10001 (Run 86, Clone 4, Gen 22)
[19:37:48] Reading tar file par_all27_prot_lipid.inp
[19:37:49] Reading tar file scpismQuartic.inp
[19:37:49] Reading tar file ww_exteq_nowater1.pdb
[19:37:49] Reading tar file ww_exteq_nowater1.psf
[19:37:49] Reading tar file checkpt
[19:37:49] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[19:37:49] Folding@home Core Shutdown: EARLY_UNIT_END
[19:37:52] CoreStatus = 79 (121)
[19:37:52] Client-core communications error: ERROR 0x79
[19:37:52] This is a sign of more serious problems, shutting down.

Closed off error dialog box and restarted. In infinitum.

Closed off error dialog box, deleted folder 08 and 08 locks.

Code: Select all

[19:58:39] Project: 10001 (Run 86, Clone 4, Gen 22)
[19:58:39] Reading tar file par_all27_prot_lipid.inp
[19:58:39] Reading tar file scpismQuartic.inp
[19:58:39] Reading tar file ww_exteq_nowater1.pdb
[19:58:39] Reading tar file ww_exteq_nowater1.psf
[19:58:39] Reading tar file checkpt
[19:58:39] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[19:58:39] Folding@home Core Shutdown: EARLY_UNIT_END
[19:58:43] CoreStatus = 79 (121)
[19:58:43] Client-core communications error: ERROR 0x79
[19:58:43] This is a sign of more serious problems, shutting down.

Closed off error dialog box, deleted folder 09 and 09 locks.

Code: Select all

[20:02:15] Project: 10001 (Run 86, Clone 4, Gen 22)
[20:02:15] Reading tar file par_all27_prot_lipid.inp
[20:02:15] Reading tar file scpismQuartic.inp
[20:02:15] Reading tar file ww_exteq_nowater1.pdb
[20:02:15] Reading tar file ww_exteq_nowater1.psf
[20:02:15] Reading tar file checkpt
[20:02:15] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[20:02:15] Folding@home Core Shutdown: EARLY_UNIT_END
[20:02:19] CoreStatus = 79 (121)
[20:02:19] Client-core communications error: ERROR 0x79
[20:02:19] This is a sign of more serious problems, shutting down.

Closed off error dialog box, deleted entire work folder.

Code: Select all

[20:04:48] Project: 10001 (Run 86, Clone 4, Gen 22)
[20:04:48] Reading tar file par_all27_prot_lipid.inp
[20:04:49] Reading tar file scpismQuartic.inp
[20:04:49] Reading tar file ww_exteq_nowater1.pdb
[20:04:49] Reading tar file ww_exteq_nowater1.psf
[20:04:49] Reading tar file checkpt
[20:04:49] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[20:04:49] Folding@home Core Shutdown: EARLY_UNIT_END
[20:04:53] CoreStatus = 79 (121)
[20:04:53] Client-core communications error: ERROR 0x79
[20:04:53] This is a sign of more serious problems, shutting down.

Closed off error dialog box, deleted entire work folder.

Code: Select all

[20:06:11] Project: 10001 (Run 86, Clone 4, Gen 22)
[20:06:11] Reading tar file par_all27_prot_lipid.inp
[20:06:11] Reading tar file scpismQuartic.inp
[20:06:11] Reading tar file ww_exteq_nowater1.pdb
[20:06:11] Reading tar file ww_exteq_nowater1.psf
[20:06:11] Reading tar file checkpt
[20:06:11] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[20:06:11] Folding@home Core Shutdown: EARLY_UNIT_END
[20:06:15] CoreStatus = 79 (121)
[20:06:15] Client-core communications error: ERROR 0x79
[20:06:15] This is a sign of more serious problems, shutting down.

Shutdown all clients and stopped all folding at home until further notice. This WU is filling one diaper after another.


P.S. If anyone is reading this post and has a hand in developing the client, I suggest that the user should be able to blacklist certain WUs. For example, open the client in -configonly mode and you can enter the following:

Blacklist (use comma): 10001 (Run 86, Clone 4, Gen 22)

This way you can manually reject projects that refuse to work on your computer and move on in a matter of minutes.

[toTOW]

Could you post the core startup part of the log that shows useful details about your OS and CPU ?

[Grendel]

Bet it's the same problem reported here and here.

From the 2nd link:

From your report I was able to track down the bug and fix it in the next release of the ProtoMol core.

When will that core be released ?

[Tynat]

As requested, here is the last attempt with extended core information.

Code: Select all

[20:06:02] + Received work.
[20:06:02] + Closed connections
[20:06:07] 
[20:06:07] + Processing work unit
[20:06:07] Core required: FahCore_b4.exe
[20:06:07] Core found.
[20:06:07] Working on queue slot 02 [January 13 20:06:07 UTC]
[20:06:07] + Working ...
[20:06:07] - Calling '.\FahCore_b4.exe -dir work/ -suffix 02 -checkpoint 15 -verbose -lifeline 568 -version 623'

[20:06:11] *********************** Log Started 13/Jan/2010 20:06:11 ***********************
[20:06:11] ************************** ProtoMol Folding@Home Core **************************
[20:06:11]   Version: 21
[20:06:11]      Type: 180
[20:06:11]      Core: ProtoMol
[20:06:11]   Website: http://folding.stanford.edu/
[20:06:11] Copyright: (c) 2009 Stanford University
[20:06:11]    Author: Joseph Coffland <joseph@cauldrondevelopment.com>
[20:06:11]      Args: -dir work/ -suffix 02 -checkpoint 15 -verbose -lifeline 568 -version
[20:06:11]            623
[20:06:11] ************************************ Build *************************************
[20:06:11]      Date: Dec 24 2009
[20:06:11]      Time: 14:36:31
[20:06:11]  Revision: 1748
[20:06:11]  Compiler: Intel(R) C++ MSVC 1500 mode 1110
[20:06:11]   Options: /TP /nologo /EHsc /wd4297 /wd4103 /wd1786 /arch:IA32 /Ox
[20:06:11]            /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qrestrict /MT
[20:06:11]  Platform: Windows XP
[20:06:11]      Bits: 32
[20:06:11] ************************************ System ************************************
[20:06:11]        OS: Microsoft Windows XP Professional
[20:06:11]       CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz
[20:06:11]    CPU ID: GenuineIntel Family 15 Model 2 Stepping 9
[20:06:11]      CPUs: 2 Logical, 1 Physical
[20:06:11]    Memory: 2.00 GB
[20:06:11] ********************************************************************************
[20:06:11] Project: 10001 (Run 86, Clone 4, Gen 22)
[20:06:11] Reading tar file par_all27_prot_lipid.inp
[20:06:11] Reading tar file scpismQuartic.inp
[20:06:11] Reading tar file ww_exteq_nowater1.pdb
[20:06:11] Reading tar file ww_exteq_nowater1.psf
[20:06:11] Reading tar file checkpt
[20:06:11] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[20:06:11] Folding@home Core Shutdown: EARLY_UNIT_END
[20:06:15] CoreStatus = 79 (121)
[20:06:15] Client-core communications error: ERROR 0x79
[20:06:15] This is a sign of more serious problems, shutting down.

[Tynat]

When will that core be released ?

If this is indeed the same problem and the core has been fixed days ago, why hasn't it been released?

The client will not download anything other than this project, which repeatedly results in the same "serious error" failure.

There needs to be a client side blacklist. Period. Let those who don't have a problem with this WU deal with it, meanwhile, my client can go on it's merry way downloading something different and contribute. Right now it's closed and contributing nothing. Without a means to blacklist a WU, problems such as this continue to waste enormous amounts of time.

[Tynat]

So, next comes this "No appropriate work server was available" problem that lasts for most of the day, and wouldn't you know it, the very first WU that my client tries to get is "Project: 10001 (Run 86, Clone 4, Gen 22)"!

Of course it's EUE'd without a fixed core being available for download.

Sure makes it seem like there isn't a different WU available on the planet.

If only there was some way to pick a different WU. Like, I don't know, if there was a client side WU blacklist option available.

[bollix47]

In order to get a different WU sometimes deleting queue.dat, unitinfo.txt and the contents of the work folder is not enough. When that happens I also delete machineindependent.dat which will change your client's ID and the server should switch to a different WU.

[Tynat]

In order to get a different WU sometimes deleting queue.dat, unitinfo.txt and the contents of the work folder is not enough. When that happens I also delete machineindependent.dat which will change your client's ID and the server should switch to a different WU.

A few minutes ago I deleted the Work folder with this result:

Code: Select all

[20:42:45] *********************** Log Started 15/Jan/2010 20:42:45 ***********************
[20:42:45] ************************** ProtoMol Folding@Home Core **************************
[20:42:45]   Version: 21
[20:42:45]      Type: 180
[20:42:45]      Core: ProtoMol
[20:42:45]   Website: http://folding.stanford.edu/
[20:42:45] Copyright: (c) 2009 Stanford University
[20:42:45]    Author: Joseph Coffland <joseph@cauldrondevelopment.com>
[20:42:45]      Args: -dir work/ -suffix 08 -checkpoint 15 -verbose -lifeline 5776 -version
[20:42:45]            623
[20:42:45] ************************************ Build *************************************
[20:42:45]      Date: Dec 24 2009
[20:42:45]      Time: 14:36:31
[20:42:45]  Revision: 1748
[20:42:45]  Compiler: Intel(R) C++ MSVC 1500 mode 1110
[20:42:45]   Options: /TP /nologo /EHsc /wd4297 /wd4103 /wd1786 /arch:IA32 /Ox
[20:42:45]            /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qrestrict /MT
[20:42:45]  Platform: Windows XP
[20:42:45]      Bits: 32
[20:42:45] ************************************ System ************************************
[20:42:45]        OS: Microsoft Windows XP Professional
[20:42:45]       CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz
[20:42:45]    CPU ID: GenuineIntel Family 15 Model 2 Stepping 9
[20:42:45]      CPUs: 2 Logical, 1 Physical
[20:42:45]    Memory: 2.00 GB
[20:42:45] ********************************************************************************
[20:42:45] Project: 10001 (Run 86, Clone 4, Gen 22)
[20:42:45] Reading tar file par_all27_prot_lipid.inp
[20:42:45] Reading tar file scpismQuartic.inp
[20:42:45] Reading tar file ww_exteq_nowater1.pdb
[20:42:45] Reading tar file ww_exteq_nowater1.psf
[20:42:45] Reading tar file checkpt
[20:42:45] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[20:42:45] Folding@home Core Shutdown: EARLY_UNIT_END
[20:42:49] CoreStatus = 79 (121)
[20:42:49] Client-core communications error: ERROR 0x79
[20:42:49] This is a sign of more serious problems, shutting down.

I then deleted queue.dat, unitinfo.txt and the Work folder with this result:

Code: Select all

[20:44:08] *********************** Log Started 15/Jan/2010 20:44:08 ***********************
[20:44:08] ************************** ProtoMol Folding@Home Core **************************
[20:44:08]   Version: 21
[20:44:08]      Type: 180
[20:44:08]      Core: ProtoMol
[20:44:08]   Website: http://folding.stanford.edu/
[20:44:08] Copyright: (c) 2009 Stanford University
[20:44:08]    Author: Joseph Coffland <joseph@cauldrondevelopment.com>
[20:44:08]      Args: -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 4620 -version
[20:44:08]            623
[20:44:08] ************************************ Build *************************************
[20:44:08]      Date: Dec 24 2009
[20:44:08]      Time: 14:36:31
[20:44:08]  Revision: 1748
[20:44:08]  Compiler: Intel(R) C++ MSVC 1500 mode 1110
[20:44:08]   Options: /TP /nologo /EHsc /wd4297 /wd4103 /wd1786 /arch:IA32 /Ox
[20:44:08]            /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qrestrict /MT
[20:44:08]  Platform: Windows XP
[20:44:08]      Bits: 32
[20:44:08] ************************************ System ************************************
[20:44:08]        OS: Microsoft Windows XP Professional
[20:44:08]       CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz
[20:44:08]    CPU ID: GenuineIntel Family 15 Model 2 Stepping 9
[20:44:08]      CPUs: 2 Logical, 1 Physical
[20:44:08]    Memory: 2.00 GB
[20:44:08] ********************************************************************************
[20:44:08] Project: 10001 (Run 86, Clone 4, Gen 22)
[20:44:08] Reading tar file par_all27_prot_lipid.inp
[20:44:08] Reading tar file scpismQuartic.inp
[20:44:08] Reading tar file ww_exteq_nowater1.pdb
[20:44:08] Reading tar file ww_exteq_nowater1.psf
[20:44:08] Reading tar file checkpt
[20:44:08] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[20:44:08] Folding@home Core Shutdown: EARLY_UNIT_END
[20:44:11] CoreStatus = 79 (121)
[20:44:11] Client-core communications error: ERROR 0x79
[20:44:11] This is a sign of more serious problems, shutting down.

There is no "machineindependent.dat" file on this computer. Maybe that is for a different client or OS?

The way the client keeps downloading the same WU makes it look like Project: 10001 (Run 86, Clone 4, Gen 22) is the last WU that Stanford needs before calling the Folding At Home project finished. Unfortunately for the project, it's insisting on my computer's client, using a buggy core, to be the one to do it.

I don't understand why Project: 10001 (Run 86, Clone 4, Gen 22) hasn't been pulled off the servers until the new core is released. It's been nearly 48 hours and now the weekend is upon us. It's bad enough that the core throws up a error dialog box which closes the client every time, but it is really silly that in order to get around problems such as these you have to manually delete the queue.dat, unitinfo.txt and the Work folder repeatedly. Which is why I believe being able to blacklist a WU is a much more elegant solution when problems such as these are not handled in a timely manor. Instead, there will be days (weeks?) of lost production.

[bollix47]

There is no "machineindependent.dat" file on this computer. Maybe that is for a different client or OS?

Yes, you are correct. In Windows XP the info is stored in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Fold.....

If you search the forum using registry and machineID you will find posts discussing where and how to change with warnings about the dangers of editing the registry.

[Tynat]

There is no "machineindependent.dat" file on this computer. Maybe that is for a different client or OS?

Yes, you are correct. In Windows XP the info is stored in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Fold.....

If you search the forum using registry and machineID you will find posts discussing where and how to change with warnings about the dangers of editing the registry.

Thanks for the information. However, the registry entry you provided above is for services and I prefer to have access to the client and see what it's doing, so I don't run it as a service.

On the off chance that a miracle had taken place and the new, fixed, core had finally been released, I deleted the B4 core file. Also, on the off chance that another miracle had taken place and the Project: 10001 (Run 86, Clone 4, Gen 22) had been pulled, I restarted the client. Both attempts to fix the problem resulted in the following:

Code: Select all

[09:32:14] *********************** Log Started 16/Jan/2010 09:32:14 ***********************
[09:32:14] ************************** ProtoMol Folding@Home Core **************************
[09:32:14]   Version: 21
[09:32:14]      Type: 180
[09:32:14]      Core: ProtoMol
[09:32:14]   Website: http://folding.stanford.edu/
[09:32:14] Copyright: (c) 2009 Stanford University
[09:32:14]    Author: Joseph Coffland <joseph@cauldrondevelopment.com>
[09:32:14]      Args: -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 5452 -version
[09:32:14]            623
[09:32:14] ************************************ Build *************************************
[09:32:14]      Date: Dec 24 2009
[09:32:14]      Time: 14:36:31
[09:32:14]  Revision: 1748
[09:32:14]  Compiler: Intel(R) C++ MSVC 1500 mode 1110
[09:32:14]   Options: /TP /nologo /EHsc /wd4297 /wd4103 /wd1786 /arch:IA32 /Ox
[09:32:14]            /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qrestrict /MT
[09:32:14]  Platform: Windows XP
[09:32:14]      Bits: 32
[09:32:14] ************************************ System ************************************
[09:32:14]        OS: Microsoft Windows XP Professional
[09:32:14]       CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz
[09:32:14]    CPU ID: GenuineIntel Family 15 Model 2 Stepping 9
[09:32:14]      CPUs: 2 Logical, 1 Physical
[09:32:14]    Memory: 2.00 GB
[09:32:14] ********************************************************************************
[09:32:14] Project: 10001 (Run 86, Clone 4, Gen 22)
[09:32:14] Reading tar file par_all27_prot_lipid.inp
[09:32:15] Reading tar file scpismQuartic.inp
[09:32:15] Reading tar file ww_exteq_nowater1.pdb
[09:32:15] Reading tar file ww_exteq_nowater1.psf
[09:32:15] Reading tar file checkpt
[09:32:15] ERROR: @ fah\tar\TarHeader.cpp:184:<unknown> 0: Error converting number '026193'
[09:32:15] Folding@home Core Shutdown: EARLY_UNIT_END
[09:32:18] CoreStatus = 79 (121)
[09:32:18] Client-core communications error: ERROR 0x79
[09:32:18] This is a sign of more serious problems, shutting down.

As you can see from the above; re-downloading the core resulted in the same core as before and Project: 10001 (Run 86, Clone 4, Gen 22) wasn't pulled from the server prior to the weekend.

Clearly this demonstrates there wasn't any forethought put into the client to deal with these types of common problems. Having the user manually deleting files, directories, and editing the registry is a helluva way to run a project. Where as a WU blacklist option would be trivial by comparison and would have already solved this predicament a long time ago.

[bollix47]

Have you tried re-configuring the client? Things like switching the advmethods option or changing the machineID. Another option might be to uninstall and re-install. Of course you should delete the aformentioned files before restarting with a different option.

I doubt your proposal will ever be taken seriously as it could lead to "cherry picking" and that is not allowed.

[Tynat]

Have you tried re-configuring the client? Things like switching the advmethods option or changing the machineID. Another option might be to uninstall and re-install. Of course you should delete the aformentioned files before restarting with a different option.

I don't know if a miracle took place or not between the last time I tried starting the client, but I changed the machineID (not sure why that would make a difference) and it finally downloaded a different WU. Sheesh, the hoops one has to jump through to get something to work.

I doubt your proposal will ever be taken seriously as it could lead to "cherry picking" and that is not allowed.

How can one "cherry pick" a WU when they are calculated on the same test machine to achieve equality in PPD? Unless of course the test machine method doesn't work.

[Tobit]

Tynat, if you really are this aggravated.. ahh, nevermind.. you wouldn't understand anyway.

If one was able to "blacklist", as you say, certain WUs, it would be easy to abuse so that one would only receive, for instance, 353 point work units which, on the NVIDIA GPUs, produce the most points per day. I would merely have to plug in all other projects to said "blacklist" and thus only receive the 353 point units I want.. hence "cherry pickin'".

[bollix47]

I don't know if a miracle took place or not between the last time I tried starting the client, but I changed the machineID (not sure why that would make a difference) and it finally downloaded a different WU. Sheesh, the hoops one has to jump through to get something to work.

Very basically what was happening is that a server sent you a "bad" WU. It died and didn't send anything back to the server to let it know the client wanted a different WU. So when the client went back to the server for a new WU the server checks and says "hold on there I didn't receive the last WU you were sent so I'm sending it again". This behaviour is normal to avoid problems in transmission etc. but is supposed to stop after 3 or 4 tries and move on to a different WU. Not sure if you gave it 3 or 4 chances but if it gave you the same bad WU more times than that then maybe something is amiss at the server end.
By changing the machineID you have identified a different client to the server and as a result it has no record of ever sending that bad WU to this new client and so has nothing to repeat and merely assigns you a new/different WU.

Glad you got that working again.