Project: 5767 (Run 14, Clone 33, Gen 275)

[Amaruk]

WU died immediately, taking the core and client with it.

Card is 8800 GT 256MB G92 at vendor clocks - 700/750/1700

X2 5600 @ 2.9 Ghz, 4 gb PC2-6400, XP Pro32 SP2, MSI K9A2 Platinum, 6.23 Systray, 181.20 drivers

Code: Select all

[20:32:43] - Preparing to get new work unit...
[20:32:43] + Attempting to get work packet
[20:32:43] - Will indicate memory of 2815 MB
[20:32:43] - Connecting to assignment server
[20:32:43] Connecting to http://assign-GPU.stanford.edu:8080/
[20:32:44] Posted data.
[20:32:44] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[20:32:44] + News From Folding@Home: GPU folding beta
[20:32:44] Loaded queue successfully.
[20:32:44] Connecting to http://171.67.108.11:8080/
[20:32:44] Posted data.
[20:32:44] Initial: 0000; - Receiving payload (expected size: 47282)
[20:32:44] Conversation time very short, giving reduced weight in bandwidth avg
[20:32:44] - Downloaded at ~92 kB/s
[20:32:44] - Averaged speed for that direction ~121 kB/s
[20:32:44] + Received work.
[20:32:44] Trying to send all finished work units
[20:32:44] + No unsent completed units remaining.
[20:32:44] + Closed connections
[20:32:49] 
[20:32:49] + Processing work unit
[20:32:49] Core required: FahCore_11.exe
[20:32:49] Core found.
[20:32:49] Working on queue slot 08 [May 3 20:32:49 UTC]
[20:32:49] + Working ...
[20:32:49] - Calling '.\FahCore_11.exe -dir work/ -suffix 08 -checkpoint 15 -verbose -lifeline 2876 -version 623'

[20:32:49] 
[20:32:49] *------------------------------*
[20:32:49] Folding@Home GPU Core - Beta
[20:32:49] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[20:32:49] 
[20:32:49] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[20:32:49] Build host: amoeba
[20:32:49] Board Type: Nvidia
[20:32:49] Core      : 
[20:32:49] Preparing to commence simulation
[20:32:49] - Looking at optimizations...
[20:32:49] - Created dyn
[20:32:49] - Files status OK
[20:32:49] - Expanded 46770 -> 252912 (decompressed 540.7 percent)
[20:32:49] Called DecompressByteArray: compressed_data_size=46770 data_size=252912, decompressed_data_size=252912 diff=0
[20:32:49] - Digital signature verified
[20:32:49] 
[20:32:49] Project: 5767 (Run 14, Clone 33, Gen 275)
[20:32:49] 
[20:32:49] Assembly optimizations on if available.
[20:32:49] Entering M.D.
[20:32:56] mdrun_gpu returned 
[20:32:56] Going to send back what have done -- stepsTotalG=0
[20:32:56] Work fraction=0.0000 steps=0.
[20:33:00] logfile size=5000 infoLength=5000 edr=0 trr=25
[20:33:00] - Writing 5538 bytes of core data to disk...
[20:33:00] Done: 5026 -> 1885 (compressed to 37.5 percent)
[20:33:00]   ... Done.
[20:33:00] 
[20:33:00] Folding@home Core Shutdown: UNSTABLE_MACHINE

At this point a pop-up message appeared with the heading Microsoft Visual C++ Runtime Library saying:

Runtime Error!

Program: C:\Documents and Settings\ect...

This application has requested the Runtime to terminate in an unusual way.
Please contact the application's support team for more information.

Clicking OK brought up another popup with heading FahCore_11.exe - Application error saying:

The exception unknown software exception (0x40000015) occurred in the application at location 0x005236ee.

Click on OK to terminate program

After clicking OK, the client added the following lines to the log regarding the core's untimely demise:

Code: Select all

[22:19:04] CoreStatus = 3 (3)
[22:19:04] Client-core communications error: ERROR 0x3
[22:19:04] This is a sign of more serious problems, shutting down.

At that point the following popup titled Folding@home appeared saying:

Folding@home has run into a serious error running the core, and will shutdown.


Clicking OK killed the client. Here is the log after restart.

Code: Select all

--- Opening Log file [May 3 22:21:53 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.23

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Documents and Settings\Hammersmythe\Application Data\Folding@home-gpu
Arguments: -gpu 0 -verbosity 9 -forcegpu nvidia_g80 

[22:21:53] - Ask before connecting: No
[22:21:53] - User name: Amaruk (Team 50625)
[22:21:53] - User ID: C*************5
[22:21:53] - Machine ID: 2
[22:21:53] 
[22:21:53] Loaded queue successfully.
[22:21:53] Initialization complete
[22:21:53] 
[22:21:53] + Processing work unit
[22:21:53] Core required: FahCore_11.exe
[22:21:53] Core found.
[22:21:53] - Autosending finished units... [May 3 22:21:53 UTC]
[22:21:53] Trying to send all finished work units
[22:21:53] + No unsent completed units remaining.
[22:21:53] - Autosend completed
[22:21:53] Working on queue slot 08 [May 3 22:21:53 UTC]
[22:21:53] + Working ...
[22:21:53] - Calling '.\FahCore_11.exe -dir work/ -suffix 08 -checkpoint 15 -verbose -lifeline 2868 -version 623'

[22:21:53] 
[22:21:53] *------------------------------*
[22:21:53] Folding@Home GPU Core - Beta
[22:21:53] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[22:21:53] 
[22:21:53] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[22:21:53] Build host: amoeba
[22:21:53] Board Type: Nvidia
[22:21:53] Core      : 
[22:21:53] Preparing to commence simulation
[22:21:53] - Looking at optimizations...
[22:21:53] - Created dyn
[22:21:53] - Files status OK
[22:21:53] Error: Missing work file=<>
[22:21:53] 
[22:21:53] Folding@home Core Shutdown: MISSING_WORK_FILES
[22:21:58] CoreStatus = 74 (116)
[22:21:58] The core could not find the work files specified. Removing from queue
[22:21:58] Deleting current work unit & continuing...
[22:22:02] Trying to send all finished work units
[22:22:02] + No unsent completed units remaining.
[22:22:02] - Preparing to get new work unit...
[22:22:02] + Attempting to get work packet
[22:22:02] - Will indicate memory of 2815 MB
[22:22:02] - Detect CPU. Vendor: AuthenticAMD, Family: 15, Model: 11, Stepping: 2
[22:22:02] - Connecting to assignment server
[22:22:02] Connecting to http://assign-GPU.stanford.edu:8080/
[22:22:02] Posted data.
[22:22:02] Initial: 40AB; - Successful: assigned to (171.64.122.70).
[22:22:02] + News From Folding@Home: GPU folding beta
[22:22:02] Loaded queue successfully.
[22:22:02] Connecting to http://171.64.122.70:8080/
[22:22:02] Posted data.
[22:22:02] Initial: 0000; - Receiving payload (expected size: 70834)
[22:22:03] - Downloaded at ~69 kB/s
[22:22:03] - Averaged speed for that direction ~111 kB/s
[22:22:03] + Received work.
[22:22:03] + Closed connections
[22:22:08] 
[22:22:08] + Processing work unit
[22:22:08] Core required: FahCore_14.exe
[22:22:08] Core found.
[22:22:08] Working on queue slot 09 [May 3 22:22:08 UTC]
[22:22:08] + Working ...
[22:22:08] - Calling '.\FahCore_14.exe -dir work/ -suffix 09 -checkpoint 15 -verbose -lifeline 2868 -version 623'

[22:22:08] 
[22:22:08] *------------------------------*
[22:22:08] Folding@Home GPU Core - Beta
[22:22:08] Version 1.25 (Mon Mar 2 19:49:32 PST 2009)
[22:22:08] 
[22:22:08] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[22:22:08] Build host: vspm46
[22:22:08] Board Type: Nvidia
[22:22:08] Core      : 
[22:22:08] Preparing to commence simulation
[22:22:08] - Looking at optimizations...
[22:22:08] - Created dyn
[22:22:08] - Files status OK
[22:22:08] - Expanded 70322 -> 360060 (decompressed 512.0 percent)
[22:22:08] Called DecompressByteArray: compressed_data_size=70322 data_size=360060, decompressed_data_size=360060 diff=0
[22:22:08] - Digital signature verified
[22:22:08] 
[22:22:08] Project: 5900 (Run 7, Clone 869, Gen 3)
[22:22:08] 
[22:22:08] Assembly optimizations on if available.
[22:22:08] Entering M.D.
[22:22:14] Tpr hash work/wudata_09.tpr:  1210930608 966887911 2741145792 396630749 1428891250
[22:22:14] Working on Protein
[22:22:15] Client config found, loading data.
[22:22:15] Starting GUI Server
[22:23:37] Completed 1%

Just curious if someone else has completed this yet.

If so I will most likely wipe everything and reinstall.

I might do that anyway, since that looks a bit like a registry error and nuking it will likely be faster than fixing it...

[bruce]

Yes, somebody else completed it successfully
Your WU (P5767 R14 C33 G275) was added to the stats database on 2009-05-03 20:11:06 for 353 points of credit

When was the WU downloaded?

Here is a similar report where a WU passed the deadline and the client attempted to abort the current WU but that code failed with the same error code (0x40000015). viewtopic.php?t=5228 Clearly the client should be able to tell the core that the current WU has expired without such a cryptic message, but as far as I know, they have not cleaned up that part of the code yet.

[Amaruk]

Thanks bruce.

Downloaded today per the log:

Code: Select all

[20:32:44] + Received work.
[20:32:44] Trying to send all finished work units
[20:32:44] + No unsent completed units remaining.
[20:32:44] + Closed connections
[20:32:49] 
[20:32:49] + Processing work unit
[20:32:49] Core required: FahCore_11.exe
[20:32:49] Core found.
[20:32:49] Working on queue slot 08 [May 3 20:32:49 UTC]

Restarted client at 22:21:53, less than two hours later.

Your WU (P5767 R14 C33 G275) was added to the stats database on 2009-05-03 20:11:06 for 353 points of credit

Let's see...PDT + 7 = UTC...so that's May 4 03:11:06 UTC.

That means I was assigned his/her WU 6 hours 38 minutes 22 seconds before then.

Either they missed the 2 day deadline (THAT's a slow GPU ) or the server assigned it twice. (got mine from 171.67.108.11)

[toTOW]

I think bruce didn't read carefully ... you completed the WU successfully :

Hi Amaruk (team 50625),
Your WU (P5767 R14 C33 G275) was added to the stats database on 2009-05-03 20:11:06 for 353 points of credit.

[Amaruk]

Wilber To Frank, say again your last, over.

has my gpu turned into a DMC-12?

~~~~EDIT~~~~

Thanks toTow. Sorted. Never thought to check the good WUs.

When I restarted the client I was assigned to 171.64.122.70, which gave me P5900 R7 C869 G3.

After completing it I was assigned to 171.67.108.11 at 01:27:36. Since I had not returned P5767 R14 C33 G275 it was assigned again.

Code: Select all

[01:27:36] - Connecting to assignment server
[01:27:36] Connecting to http://assign-GPU.stanford.edu:8080/
[01:27:36] Posted data.
[01:27:36] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[01:27:36] + News From Folding@Home: GPU folding beta
[01:27:36] Loaded queue successfully.
[01:27:36] Connecting to http://171.67.108.11:8080/
[01:27:36] Posted data.
[01:27:36] Initial: 0000; - Receiving payload (expected size: 47282)
[01:27:37] - Downloaded at ~46 kB/s
[01:27:37] - Averaged speed for that direction ~98 kB/s
[01:27:37] + Received work.
[01:27:37] Trying to send all finished work units
[01:27:37] + No unsent completed units remaining.
[01:27:37] + Closed connections
[01:27:37] 
[01:27:37] + Processing work unit
[01:27:37] Core required: FahCore_11.exe
[01:27:37] Core found.
[01:27:37] Working on queue slot 00 [May 4 01:27:37 UTC]
[01:27:37] + Working ...
[01:27:37] - Calling '.\FahCore_11.exe -dir work/ -suffix 00 -checkpoint 15 -verbose -lifeline 2868 -version 623'

[01:27:37] 
[01:27:37] *------------------------------*
[01:27:37] Folding@Home GPU Core - Beta
[01:27:37] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[01:27:37] 
[01:27:37] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[01:27:37] Build host: amoeba
[01:27:37] Board Type: Nvidia
[01:27:37] Core      : 
[01:27:37] Preparing to commence simulation
[01:27:37] - Looking at optimizations...
[01:27:37] - Created dyn
[01:27:37] - Files status OK
[01:27:37] - Expanded 46770 -> 252912 (decompressed 540.7 percent)
[01:27:37] Called DecompressByteArray: compressed_data_size=46770 data_size=252912, decompressed_data_size=252912 diff=0
[01:27:37] - Digital signature verified
[01:27:37] 
[01:27:37] Project: 5767 (Run 14, Clone 33, Gen 275)
[01:27:37] 
[01:27:37] Assembly optimizations on if available.
[01:27:37] Entering M.D.
[01:27:43] Working on Protein
[01:27:44] Client config found, loading data.
[01:27:44] Starting GUI Server
[01:28:56] Completed 1%
[01:30:08] Completed 2%
[01:31:20] Completed 3%
[01:32:32] Completed 4%
[01:33:44] Completed 5%
[01:34:56] Completed 6%
[01:36:09] Completed 7%
[01:37:21] Completed 8%
[01:38:34] Completed 9%
[01:39:46] Completed 10%
[01:40:59] Completed 11%
[01:42:11] Completed 12%
[01:43:24] Completed 13%
[01:44:37] Completed 14%
[01:45:50] Completed 15%
[01:47:03] Completed 16%
[01:48:16] Completed 17%
[01:49:29] Completed 18%
[01:50:42] Completed 19%
[01:51:55] Completed 20%
[01:53:08] Completed 21%
[01:54:22] Completed 22%
[01:55:35] Completed 23%
[01:56:48] Completed 24%
[01:58:02] Completed 25%
[01:59:15] Completed 26%
[02:00:29] Completed 27%
[02:01:42] Completed 28%
[02:02:56] Completed 29%
[02:04:09] Completed 30%
[02:05:23] Completed 31%
[02:06:37] Completed 32%
[02:07:51] Completed 33%
[02:09:05] Completed 34%
[02:10:19] Completed 35%
[02:11:33] Completed 36%
[02:12:47] Completed 37%
[02:14:01] Completed 38%
[02:15:20] Completed 39%
[02:16:35] Completed 40%
[02:17:50] Completed 41%
[02:19:05] Completed 42%
[02:20:20] Completed 43%
[02:21:35] Completed 44%
[02:22:50] Completed 45%
[02:24:06] Completed 46%
[02:25:21] Completed 47%
[02:26:36] Completed 48%
[02:27:52] Completed 49%
[02:29:07] Completed 50%
[02:30:22] Completed 51%
[02:31:37] Completed 52%
[02:32:53] Completed 53%
[02:34:08] Completed 54%
[02:35:23] Completed 55%
[02:36:39] Completed 56%
[02:37:41] Completed 57%
[02:38:38] Completed 58%
[02:39:35] Completed 59%
[02:40:32] Completed 60%
[02:41:30] Completed 61%
[02:42:27] Completed 62%
[02:43:24] Completed 63%
[02:44:22] Completed 64%
[02:45:19] Completed 65%
[02:46:16] Completed 66%
[02:47:13] Completed 67%
[02:48:11] Completed 68%
[02:49:08] Completed 69%
[02:50:05] Completed 70%
[02:51:03] Completed 71%
[02:52:10] Completed 72%
[02:53:26] Completed 73%
[02:54:41] Completed 74%
[02:55:57] Completed 75%
[02:57:12] Completed 76%
[02:58:28] Completed 77%
[02:59:42] Completed 78%
[03:00:44] Completed 79%
[03:01:46] Completed 80%
[03:02:47] Completed 81%
[03:03:49] Completed 82%
[03:04:51] Completed 83%
[03:05:53] Completed 84%
[03:06:55] Completed 85%
[03:07:56] Completed 86%
[03:08:58] Completed 87%
[03:10:00] Completed 88%
[03:11:02] Completed 89%
[03:12:04] Completed 90%
[03:13:05] Completed 91%
[03:14:07] Completed 92%
[03:15:09] Completed 93%
[03:16:11] Completed 94%
[03:17:13] Completed 95%
[03:18:15] Completed 96%
[03:19:17] Completed 97%
[03:20:18] Completed 98%
[03:21:22] Completed 99%
[03:22:25] Completed 100%
[03:22:25] Successful run
[03:22:25] DynamicWrapper: Finished Work Unit: sleep=10000
[03:22:35] Reserved 75788 bytes for xtc file; Cosm status=0
[03:22:35] Allocated 75788 bytes for xtc file
[03:22:35] - Reading up to 75788 from "work/wudata_00.xtc": Read 75788
[03:22:35] Read 75788 bytes from xtc file; available packet space=786354676
[03:22:35] xtc file hash check passed.
[03:22:35] Reserved 15168 15168 786354676 bytes for arc file=<work/wudata_00.trr> Cosm status=0
[03:22:35] Allocated 15168 bytes for arc file
[03:22:35] - Reading up to 15168 from "work/wudata_00.trr": Read 15168
[03:22:35] Read 15168 bytes from arc file; available packet space=786339508
[03:22:35] trr file hash check passed.
[03:22:35] Allocated 560 bytes for edr file
[03:22:35] Read bedfile
[03:22:35] edr file hash check passed.
[03:22:35] Allocated 33342 bytes for logfile
[03:22:35] Read logfile
[03:22:35] GuardedRun: success in DynamicWrapper
[03:22:35] GuardedRun: done
[03:22:35] Run: GuardedRun completed.
[03:22:37] - Writing 125370 bytes of core data to disk...
[03:22:37] Done: 124858 -> 99172 (compressed to 79.4 percent)
[03:22:37]   ... Done.
[03:22:37] - Shutting down core 
[03:22:37] 
[03:22:37] Folding@home Core Shutdown: FINISHED_UNIT
[03:22:40] CoreStatus = 64 (100)
[03:22:40] Unit 0 finished with 97 percent of time to deadline remaining.
[03:22:40] Updated performance fraction: 0.958249
[03:22:40] Sending work to server
[03:22:40] Project: 5767 (Run 14, Clone 33, Gen 275)
[03:22:40] - Read packet limit of 540015616... Set to 524286976.


[03:22:40] + Attempting to send results [May 4 03:22:40 UTC]
[03:22:40] - Reading file work/wuresults_00.dat from core
[03:22:40]   (Read 99684 bytes from disk)
[03:22:40] Connecting to http://171.67.108.11:8080/
[03:22:41] Posted data.
[03:22:41] Initial: 0000; - Uploaded at ~98 kB/s
[03:22:41] - Averaged speed for that direction ~66 kB/s
[03:22:41] + Results successfully sent
[03:22:41] Thank you for your contribution to Folding@Home.

Funny thing is it looks like I got credit 10 minutes before I finished it. (2009-05-03 20:11:06 = May 4 03:11:06 UTC) But my GPU isn't really a Delorean - the log times are 1 hour fast, so I actually uploaded at 02:22:41 UTC.

Now the question is what caused the core to die an immediate ignominious death the first time, taking the client with it. Then it runs fine a few hours later.

[bruce]

I think bruce didn't read carefully ... you completed the WU successfully :

Hi Amaruk (team 50625),
Your WU (P5767 R14 C33 G275) was added to the stats database on 2009-05-03 20:11:06 for 353 points of credit.

Oops. Sorry about that.

The times are confusing to me, too. According to the database, that WU was uploaded at 2009-05-03 20:06:59 and credited a few minutes later at 20:11:06 which would have been 03:07 and 03:11 UTC on May 4 which is before the time in your log: [May 4 03:22:40 UTC]. I don't see any record that the WU was uploaded twice, so we can probably exclude the possibility that the WU was assigned to two different clients. Is your local clock set correctly?

[Amaruk]

No worries bruce.

I was given the WU twice. The next time I connected to the server (171.67.108.11) it assigned it again because it knew I had not returned it yet, which is exactly what it's supposed to do.

My system time is (was) 1m45s ahead of real (local) time. It's set manually, and I think this tricks the clients into adding an hour to adjust for DST/Summer time. Thus 02:45:10 local = 01:43:25 UTC for total offset of 1:01:45.

I've resolved this by setting my system clock to GMT.

Code: Select all

[03:37:44] Completed 4%
[03:39:42] ***** Got a SIGTERM signal (2)
[03:39:42] Killing all core threads

Folding@Home Client Shutdown.

**IMMEDIATE RESTART**

--- Opening Log file [May 5 02:48:48 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.23

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Documents and Settings\Hammersmythe\Application Data\Folding@home-gpu2
Arguments: -gpu 1 -verbosity 9 -forcegpu nvidia_g80 

[02:48:48] - Ask before connecting: No
[02:48:48] - User name: Amaruk (Team 50625)
[02:48:48] - User ID: C*************5
[02:48:48] - Machine ID: 3
[02:48:48] 
[02:48:48] Loaded queue successfully.
[02:48:48] Initialization complete
[02:48:48] 
[02:48:48] + Processing work unit
[02:48:48] Core required: FahCore_11.exe
[02:48:48] Core found.
[02:48:48] - Autosending finished units... [May 5 02:48:48 UTC]
[02:48:48] Trying to send all finished work units
[02:48:48] + No unsent completed units remaining.
[02:48:48] - Autosend completed
[02:48:48] Working on queue slot 07 [May 5 02:48:48 UTC]
[02:48:48] + Working ...
[02:48:48] - Calling '.\FahCore_11.exe -dir work/ -suffix 07 -priority 96 -checkpoint 15 -verbose -lifeline 2352 -version 623'

[02:48:48] 
[02:48:48] *------------------------------*
[02:48:48] Folding@Home GPU Core - Beta
[02:48:48] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[02:48:48] 
[02:48:48] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[02:48:48] Build host: amoeba
[02:48:48] Board Type: Nvidia
[02:48:48] Core      : 
[02:48:48] Preparing to commence simulation
[02:48:48] - Looking at optimizations...
[02:48:48] - Files status OK
[02:48:49] - Expanded 67885 -> 350980 (decompressed 517.0 percent)
[02:48:49] Called DecompressByteArray: compressed_data_size=67885 data_size=350980, decompressed_data_size=350980 diff=0
[02:48:49] - Digital signature verified
[02:48:49] 
[02:48:49] Project: 5773 (Run 4, Clone 130, Gen 3)
[02:48:49] 
[02:48:49] Assembly optimizations on if available.
[02:48:49] Entering M.D.
[02:48:54] Will resume from checkpoint file
[02:48:55] Working on Protein
[02:48:56] Client config found, loading data.
[02:48:56] Resuming from checkpoint
[02:48:56] Verified work/wudata_07.log
[02:48:56] Verified work/wudata_07.edr
[02:48:56] Starting GUI Server
[02:48:56] Verified work/wudata_07.xtc
[02:48:56] Completed 4%

Current offset = 0 This not only fixes the log times, it also syncs XP logs with F@H's logs for easier troubleshooting.

The same thing happened to another one of my GPUs

viewtopic.php?f=19&t=9840

My fingers were quicker than my brain - I restarted the client before I thought about copying the work and related files.

Would it be helpful at all if I saved the work folder and/or other files from the App Data folder before restarting the client so they could be compared to the files created when the WU is reissued? Could this help the with troubleshooting? Or is that question one the project owner would have the answer? (Vince?)