Page 1 of 1
Project: 4754 (Run 4, Clone 357, Gen 19)
Posted: Thu Apr 23, 2009 7:08 am
by shiryunaga
NANs detected on GPU, then i get the same WU again but nothing wrong
, WHY?
Code: Select all
[06:19:58] + Closed connections
[06:20:03]
[06:20:03] + Processing work unit
[06:20:03] Core required: FahCore_11.exe
[06:20:03] Core found.
[06:20:03] Working on queue slot 03 [April 23 06:20:03 UTC]
[06:20:03] + Working ...
[06:20:03] - Calling '.\FahCore_11.exe -dir work/ -suffix 03 -checkpoint 30 -verbose -lifeline 3232 -version 623'
[06:20:04]
[06:20:04] *------------------------------*
[06:20:04] Folding@Home GPU Core - Beta
[06:20:04] Version 1.24 (Mon Feb 9 11:00:12 PST 2009)
[06:20:04]
[06:20:04] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[06:20:04] Build host: amoeba
[06:20:04] Board Type: AMD
[06:20:04] Core :
[06:20:04] Preparing to commence simulation
[06:20:04] - Looking at optimizations...
[06:20:04] - Created dyn
[06:20:04] - Files status OK
[06:20:04] - Expanded 85694 -> 444252 (decompressed 518.4 percent)
[06:20:04] Called DecompressByteArray: compressed_data_size=85694 data_size=444252, decompressed_data_size=444252 diff=0
[06:20:04] - Digital signature verified
[06:20:04]
[06:20:04] Project: 4754 (Run 4, Clone 357, Gen 19)
[06:20:04]
[06:20:04] Assembly optimizations on if available.
[06:20:04] Entering M.D.
[06:20:10] Tpr hash work/wudata_03.tpr: 3016136433 4003022685 778188496 2204497586 1905765869
[06:20:11] Working on 1254 p4754_lam5w_300K_g91
[06:20:11] Client config found, loading data.
[06:20:11] Starting GUI Server
[06:26:56] Completed 1%
[06:33:25] Completed 2%
[06:38:01] mdrun_gpu returned
[06:38:01] NANs detected on GPU
[06:38:01]
[06:38:01] Folding@home Core Shutdown: UNSTABLE_MACHINE
[06:38:04] CoreStatus = 7A (122)
[06:38:04] Sending work to server
[06:38:04] Project: 4754 (Run 4, Clone 357, Gen 19)
[06:38:04] - Read packet limit of 540015616... Set to 524286976.
[06:38:04] - Error: Could not get length of results file work/wuresults_03.dat
[06:38:04] - Error: Could not read unit 03 file. Removing from queue.
[06:38:04] Trying to send all finished work units
[06:38:04] + No unsent completed units remaining.
[06:38:04] - Preparing to get new work unit...
[06:38:04] + Attempting to get work packet
[06:38:04] - Will indicate memory of 510 MB
[06:38:04] - Connecting to assignment server
[06:38:04] Connecting to http://assign-GPU.stanford.edu:8080/
[06:38:08] Posted data.
[06:38:08] Initial: 40AB; - Successful: assigned to (171.64.65.103).
[06:38:08] + News From Folding@Home: GPU folding beta
[06:38:09] Loaded queue successfully.
[06:38:09] Connecting to http://171.64.65.103:8080/
[06:38:11] Posted data.
[06:38:12] Initial: 0000; - Receiving payload (expected size: 86206)
[06:38:22] - Downloaded at ~8 kB/s
[06:38:22] - Averaged speed for that direction ~11 kB/s
[06:38:22] + Received work.
[06:38:22] Trying to send all finished work units
[06:38:22] + No unsent completed units remaining.
[06:38:22] + Closed connections
[06:38:27]
[06:38:27] + Processing work unit
[06:38:27] Core required: FahCore_11.exe
[06:38:27] Core found.
[06:38:27] Working on queue slot 04 [April 23 06:38:27 UTC]
[06:38:27] + Working ...
[06:38:27] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -checkpoint 30 -verbose -lifeline 3232 -version 623'
[06:38:27]
[06:38:27] *------------------------------*
[06:38:27] Folding@Home GPU Core - Beta
[06:38:27] Version 1.24 (Mon Feb 9 11:00:12 PST 2009)
[06:38:27]
[06:38:27] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[06:38:27] Build host: amoeba
[06:38:27] Board Type: AMD
[06:38:27] Core :
[06:38:27] Preparing to commence simulation
[06:38:27] - Looking at optimizations...
[06:38:28] - Created dyn
[06:38:28] - Files status OK
[06:38:28] - Expanded 85694 -> 444252 (decompressed 518.4 percent)
[06:38:28] Called DecompressByteArray: compressed_data_size=85694 data_size=444252, decompressed_data_size=444252 diff=0
[06:38:28] - Digital signature verified
[06:38:28]
[06:38:28] Project: 4754 (Run 4, Clone 357, Gen 19)
[06:38:28]
[06:38:28] Assembly optimizations on if available.
[06:38:28] Entering M.D.
[06:38:34] Tpr hash work/wudata_04.tpr: 3016136433 4003022685 778188496 2204497586 1905765869
[06:38:34] Working on 1254 p4754_lam5w_300K_g91
[07:06:59] ***** Got a SIGTERM signal (2)
[07:06:59] Killing all core threads
Folding@Home Client Shutdown.
[07:06:59] Got kill signal -- issuing INTERRUPTED core shutdown
--- Opening Log file [April 23 07:07:04 UTC]
# Windows GPU Console Edition #################################################
###############################################################################
Folding@Home Client Version 6.23
http://folding.stanford.edu
###############################################################################
###############################################################################
Launch directory: C:\Documents and Settings\shiryunaga\Application Data\Folding@home-gpu
Arguments: -advmethods -verbosity 9
[07:07:04] - Ask before connecting: No
[07:07:04] - User name: shiryunaga (Team 153760)
[07:07:04] - User ID: 73178E15B3AB94B
[07:07:04] - Machine ID: 3
[07:07:04]
[07:07:04] Loaded queue successfully.
[07:07:04] Initialization complete
[07:07:04]
[07:07:04] + Processing work unit
[07:07:04] - Autosending finished units... [April 23 07:07:04 UTC]
[07:07:04] Trying to send all finished work units
[07:07:04] + No unsent completed units remaining.
[07:07:04] - Autosend completed
[07:07:04] Core required: FahCore_11.exe
[07:07:04] Core found.
[07:07:04] Working on queue slot 04 [April 23 07:07:04 UTC]
[07:07:04] + Working ...
[07:07:04] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -checkpoint 30 -verbose -lifeline 996 -version 623'
[07:07:04]
[07:07:04] *------------------------------*
[07:07:04] Folding@Home GPU Core - Beta
[07:07:04] Version 1.24 (Mon Feb 9 11:00:12 PST 2009)
[07:07:04]
[07:07:04] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[07:07:04] Build host: amoeba
[07:07:04] Board Type: AMD
[07:07:04] Core :
[07:07:04] Preparing to commence simulation
[07:07:04] - Looking at optimizations...
[07:07:04] - Files status OK
[07:07:05] - Expanded 85694 -> 444252 (decompressed 518.4 percent)
[07:07:05] Called DecompressByteArray: compressed_data_size=85694 data_size=444252, decompressed_data_size=444252 diff=0
[07:07:05] - Digital signature verified
[07:07:05]
[07:07:05] Project: 4754 (Run 4, Clone 357, Gen 19)
[07:07:05]
[07:07:05] Assembly optimizations on if available.
[07:07:05] Entering M.D.
[07:07:11] Tpr hash work/wudata_04.tpr: 3016136433 4003022685 778188496 2204497586 1905765869
[07:07:11] Working on 1254 p4754_lam5w_300K_g91
[07:07:11] Client config found, loading data.
[07:07:12] Starting GUI Server
[07:10:56] Completed 1%
[07:14:43] Completed 2%
[07:20:18] Completed 3%
[07:26:05] Completed 4%
[07:31:47] Completed 5%
[07:36:48] Completed 6%
[07:41:08] Completed 7%
[07:45:27] Completed 8%
[07:49:13] Completed 9%
[07:53:18] Completed 10%
Re: Project: 4754 (Run 4, Clone 357, Gen 19)
Posted: Thu Apr 23, 2009 11:56 am
by toTOW
Some WUs are unstable in certain condition, or on some hardware. Sometimes, that might also be a random instability (interection with anoter application, overheating, voltage variations ...) ...
If this often happens, it might be good to check your hardware ...
There's no data for this WU in the DB yet ...
Re: Project: 4754 (Run 4, Clone 357, Gen 19)
Posted: Thu Apr 23, 2009 2:47 pm
by shiryunaga
update report
Code: Select all
[07:10:56] Completed 1%
[07:14:43] Completed 2%
[07:20:18] Completed 3%
[07:26:05] Completed 4%
[07:31:47] Completed 5%
[07:36:48] Completed 6%
[07:41:08] Completed 7%
[07:45:27] Completed 8%
[07:49:13] Completed 9%
[07:53:18] Completed 10%
[07:58:10] Completed 11%
[08:02:38] Completed 12%
[08:07:37] Completed 13%
[08:12:57] Completed 14%
[08:18:25] Completed 15%
[08:23:49] Completed 16%
[08:29:04] Completed 17%
[08:34:20] Completed 18%
[08:39:13] Completed 19%
[08:43:28] Completed 20%
[08:47:34] Completed 21%
--- Opening Log file [April 23 08:49:00 UTC]
# Windows GPU Console Edition #################################################
###############################################################################
Folding@Home Client Version 6.23
http://folding.stanford.edu
###############################################################################
###############################################################################
Launch directory: C:\Documents and Settings\shiryunaga\Application Data\Folding@home-gpu
Arguments: -advmethods -verbosity 9
[08:49:00] - Ask before connecting: No
[08:49:00] - User name: shiryunaga (Team 153760)
[08:49:00] - User ID: 73178E15B3AB94B
[08:49:00] - Machine ID: 3
[08:49:00]
[08:49:00] Loaded queue successfully.
[08:49:00] Initialization complete
[08:49:00]
[08:49:00] + Processing work unit
[08:49:00] - Autosending finished units... [April 23 08:49:00 UTC]
[08:49:00] Trying to send all finished work units
[08:49:00] + No unsent completed units remaining.
[08:49:00] - Autosend completed
[08:49:01] Core required: FahCore_11.exe
[08:49:01] Core found.
[08:49:02] Working on queue slot 04 [April 23 08:49:02 UTC]
[08:49:02] + Working ...
[08:49:06] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -checkpoint 30 -verbose -lifeline 1836 -version 623'
[08:49:13]
[08:49:13] *------------------------------*
[08:49:13] Folding@Home GPU Core - Beta
[08:49:13] Version 1.24 (Mon Feb 9 11:00:12 PST 2009)
[08:49:13]
[08:49:13] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[08:49:13] Build host: amoeba
[08:49:13] Board Type: AMD
[08:49:13] Core :
[08:49:13] Preparing to commence simulation
[08:49:13] - Ensuring status. Please wait.
[08:49:23] - Looking at optimizations...
[08:49:23] - Working with standard loops on this execution.
[08:49:23] - Previous termination of core was improper.
[08:49:23] - Files status OK
[08:49:24] - Expanded 85694 -> 444252 (decompressed 518.4 percent)
[08:49:24] Called DecompressByteArray: compressed_data_size=85694 data_size=444252, decompressed_data_size=444252 diff=0
[08:49:24] - Digital signature verified
[08:49:24]
[08:49:24] Project: 4754 (Run 4, Clone 357, Gen 19)
[08:49:24]
[08:49:24] Entering M.D.
[08:49:30] Will resume from checkpoint file
[08:49:30] Tpr hash work/wudata_04.tpr: 3016136433 4003022685 778188496 2204497586 1905765869
[08:49:39] Working on 1254 p4754_lam5w_300K_g91
[08:49:41] Client config found, loading data.
[08:49:41] Starting GUI Server
--- Opening Log file [April 23 09:03:08 UTC]
# Windows GPU Console Edition #################################################
###############################################################################
Folding@Home Client Version 6.23
http://folding.stanford.edu
###############################################################################
###############################################################################
Launch directory: C:\Documents and Settings\shiryunaga\Application Data\Folding@home-gpu
Arguments: -advmethods -verbosity 9
[09:03:08] - Ask before connecting: No
[09:03:08] - User name: shiryunaga (Team 153760)
[09:03:08] - User ID: 73178E15B3AB94B
[09:03:08] - Machine ID: 3
[09:03:08]
[09:03:09] Loaded queue successfully.
[09:03:09] Initialization complete
[09:03:09]
[09:03:09] + Processing work unit
[09:03:09] Core required: FahCore_11.exe
[09:03:09] Core found.
[09:03:09] - Autosending finished units... [April 23 09:03:09 UTC]
[09:03:09] Trying to send all finished work units
[09:03:09] + No unsent completed units remaining.
[09:03:09] - Autosend completed
[09:03:09] Working on queue slot 04 [April 23 09:03:09 UTC]
[09:03:09] + Working ...
[09:03:09] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -checkpoint 30 -verbose -lifeline 2344 -version 623'
[09:03:10]
[09:03:10] *------------------------------*
[09:03:10] Folding@Home GPU Core - Beta
[09:03:10] Version 1.24 (Mon Feb 9 11:00:12 PST 2009)
[09:03:10]
[09:03:10] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[09:03:10] Build host: amoeba
[09:03:10] Board Type: AMD
[09:03:10] Core :
[09:03:10] Preparing to commence simulation
[09:03:10] - Ensuring status. Please wait.
[09:03:20] - Looking at optimizations...
[09:03:20] - Working with standard loops on this execution.
[09:03:20] - Previous termination of core was improper.
[09:03:20] - Going to use standard loops.
[09:03:20] - Files status OK
[09:03:20] - Expanded 85694 -> 444252 (decompressed 518.4 percent)
[09:03:20] Called DecompressByteArray: compressed_data_size=85694 data_size=444252, decompressed_data_size=444252 diff=0
[09:03:20] - Digital signature verified
[09:03:20]
[09:03:20] Project: 4754 (Run 4, Clone 357, Gen 19)
[09:03:20]
[09:03:20] Entering M.D.
[09:03:27] Will resume from checkpoint file
[09:03:27] Tpr hash work/wudata_04.tpr: 3016136433 4003022685 778188496 2204497586 1905765869
[09:03:42] Working on 1254 p4754_lam5w_300K_g91
[09:03:43] Client config found, loading data.
[09:03:43] Starting GUI Server
[09:03:54] Resuming from checkpoint
[09:03:54] fcCheckPointResume: retreived and current tpr file hash:
[09:03:54] 0 3016136433 3016136433
[09:03:54] 1 4003022685 4003022685
[09:03:54] 2 778188496 778188496
[09:03:54] 3 2204497586 2204497586
[09:03:54] 4 1905765869 1905765869
[09:03:54] Verified work/wudata_04.log
[09:03:54] Verified work/wudata_04.edr
[09:03:54] Verified work/wudata_04.xtc
[09:03:54] Completed 21%
[09:06:59] Completed 22%
[09:10:05] Completed 23%
[09:13:55] Completed 24%
[09:17:48] Completed 25%
[09:21:22] Completed 26%
[09:25:22] Completed 27%
[09:29:39] Completed 28%
[09:33:11] Completed 29%
[09:37:01] Completed 30%
[09:41:35] Completed 31%
[09:47:00] Completed 32%
[09:50:54] Completed 33%
[09:54:14] Completed 34%
[09:58:11] Completed 35%
[10:02:21] Completed 36%
[10:06:31] Completed 37%
[10:10:53] Completed 38%
[10:14:56] Completed 39%
[10:19:03] Completed 40%
[10:23:00] Completed 41%
[10:27:06] Completed 42%
[10:31:36] Completed 43%
[10:36:14] Completed 44%
[10:40:34] Completed 45%
[10:44:54] Completed 46%
[10:49:02] Completed 47%
[10:52:45] Completed 48%
[12:37:43] ***** Got a SIGTERM signal (2)
[12:37:43] Killing all core threads
Folding@Home Client Shutdown.
--- Opening Log file [April 23 12:38:04 UTC]
# Windows GPU Console Edition #################################################
###############################################################################
Folding@Home Client Version 6.23
http://folding.stanford.edu
###############################################################################
###############################################################################
Launch directory: C:\Documents and Settings\shiryunaga\Application Data\Folding@home-gpu
Arguments: -advmethods -verbosity 9
[12:38:04] - Ask before connecting: No
[12:38:04] - User name: shiryunaga (Team 153760)
[12:38:04] - User ID: 73178E15B3AB94B
[12:38:04] - Machine ID: 3
[12:38:04]
[12:38:04] Loaded queue successfully.
[12:38:04] Initialization complete
[12:38:04]
[12:38:04] + Processing work unit
[12:38:04] - Autosending finished units... [April 23 12:38:04 UTC]
[12:38:04] Trying to send all finished work units
[12:38:04] + No unsent completed units remaining.
[12:38:04] - Autosend completed
[12:38:04] Core required: FahCore_11.exe
[12:38:04] Core found.
[12:38:04] Working on queue slot 04 [April 23 12:38:04 UTC]
[12:38:04] + Working ...
[12:38:04] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -checkpoint 30 -verbose -lifeline 3836 -version 623'
[12:38:05]
[12:38:05] *------------------------------*
[12:38:05] Folding@Home GPU Core - Beta
[12:38:05] Version 1.24 (Mon Feb 9 11:00:12 PST 2009)
[12:38:05]
[12:38:05] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[12:38:05] Build host: amoeba
[12:38:05] Board Type: AMD
[12:38:05] Core :
[12:38:05] Preparing to commence simulation
[12:38:05] - Looking at optimizations...
[12:38:05] - Files status OK
[12:38:05] - Expanded 85694 -> 444252 (decompressed 518.4 percent)
[12:38:05] Called DecompressByteArray: compressed_data_size=85694 data_size=444252, decompressed_data_size=444252 diff=0
[12:38:05] - Digital signature verified
[12:38:05]
[12:38:05] Project: 4754 (Run 4, Clone 357, Gen 19)
[12:38:05]
[12:38:05] Assembly optimizations on if available.
[12:38:05] Entering M.D.
[12:38:11] Will resume from checkpoint file
[12:38:11] Tpr hash work/wudata_04.tpr: 3016136433 4003022685 778188496 2204497586 1905765869
[12:38:12] Working on 1254 p4754_lam5w_300K_g91
[12:38:12] Client config found, loading data.
[12:38:12] Starting GUI Server
[12:38:22] Resuming from checkpoint
[12:38:22] fcCheckPointResume: retreived and current tpr file hash:
[12:38:22] 0 3016136433 3016136433
[12:38:22] 1 4003022685 4003022685
[12:38:22] 2 778188496 778188496
[12:38:22] 3 2204497586 2204497586
[12:38:22] 4 1905765869 1905765869
[12:38:22] Verified work/wudata_04.log
[12:38:22] Verified work/wudata_04.edr
[12:38:22] Verified work/wudata_04.xtc
[12:38:22] Completed 48%
[12:41:22] Completed 49%
[12:44:19] Completed 50%
[12:47:20] Completed 51%
[12:50:28] Completed 52%
[12:53:41] Completed 53%
[12:56:52] Completed 54%
[13:00:08] Completed 55%
[13:03:51] Completed 56%
[13:07:46] Completed 57%
[13:11:18] Completed 58%
[13:15:06] Completed 59%
[13:18:03] mdrun_gpu returned
[13:18:03] NANs detected on GPU
[13:18:03]
[13:18:03] Folding@home Core Shutdown: UNSTABLE_MACHINE
[13:18:05] CoreStatus = 7A (122)
[13:18:05] Sending work to server
[13:18:05] Project: 4754 (Run 4, Clone 357, Gen 19)
[13:18:05] - Read packet limit of 540015616... Set to 524286976.
[13:18:05] - Error: Could not get length of results file work/wuresults_04.dat
[13:18:05] - Error: Could not read unit 04 file. Removing from queue.
[13:18:05] Trying to send all finished work units
[13:18:05] + No unsent completed units remaining.
[13:18:05] - Preparing to get new work unit...
[13:18:05] + Attempting to get work packet
[13:18:05] - Will indicate memory of 1022 MB
[13:18:05] - Detect CPU. Vendor: GenuineIntel, Family: 15, Model: 4, Stepping: 7
[13:18:05] - Connecting to assignment server
[13:18:05] Connecting to http://assign-GPU.stanford.edu:8080/
[13:18:14] Posted data.
[13:18:14] Initial: 40AB; - Successful: assigned to (171.64.65.102).
[13:18:14] + News From Folding@Home: GPU folding beta
[13:18:14] Loaded queue successfully.
[13:18:14] Connecting to http://171.64.65.102:8080/
[13:18:17] Posted data.
[13:18:17] Initial: 0000; - Receiving payload (expected size: 97110)
[13:34:16] ***** Got a SIGTERM signal (2)
[13:34:16] Killing all core threads
Folding@Home Client Shutdown.
NANs detected on GPU again
