Project: 2671 (Run 20, Clone 52, Gen 11) seg fault

alpha754293 · Post by **alpha754293** » Wed Apr 22, 2009 4:59 am

console:

[19:49:12]
[19:49:12] *------------------------------*
[19:49:12] Folding@Home Gromacs SMP Core
[19:49:12] Version 2.06 (Tue Mar 31 08:29:45 PDT 2009)
[19:49:12]
[19:49:12] Preparing to commence simulation
[19:49:12] - Ensuring status. Please wait.
[19:49:21] - Looking at optimizations...
[19:49:21] - Working with standard loops on this execution.
[19:49:21] - Files status OK
[19:49:22] - Expanded 4837766 -> 24030157 (decompressed 496.7 percent)
[19:49:22] Called DecompressByteArray: compressed_data_size=4837766 data_size=24
030157, decompressed_data_size=24030157 diff=0
[19:49:22] - Digital signature verified
[19:49:22]
[19:49:22] Project: 2671 (Run 20, Clone 52, Gen 11)
[19:49:22]
[19:49:23] Entering M.D.
NNODES=4, MYRANK=1, HOSTNAME=computenode
NNODES=4, MYRANK=2, HOSTNAME=computenode
NNODES=4, MYRANK=3, HOSTNAME=computenode
NNODES=4, MYRANK=0, HOSTNAME=computenode
NODEID=2 argc=20
NODEID=1 argc=20
NODEID=0 argc=20
NODEID=3 argc=20
                         :-)  G  R  O  M  A  C  S  (-:

                   Groningen Machine for Chemical Simulation

                 :-)  VERSION 4.0.99_development_20090307  (-:


      Written by David van der Spoel, Erik Lindahl, Berk Hess, and others.
       Copyright (c) 1991-2000, University of Groningen, The Netherlands.
             Copyright (c) 2001-2008, The GROMACS development team,
            check out http://www.gromacs.org for more information.


                                :-)  mdrun  (-:

Reading file work/wudata_02.tpr, VERSION 3.3.99_development_20070618 (single pre
cision)
Note: tpx file_version 48, software version 64

NOTE: The tpr file used for this simulation is in an old format, for less memory
 usage and possibly more performance create a new tpr file with an up to date ve
rsion of grompp

Making 1D domain decomposition 1 x 1 x 4
starting mdrun '22911 system in water'
3000000 steps,   6000.0 ps (continuing from step 2750000,   5500.0 ps).
[19:49:32] Completed 0 out of 250000 steps  (0%)
[19:58:06] Completed 2500 out of 250000 steps  (1%)
[20:04:35] - Autosending finished units... [April 20 20:04:35 UTC]
[20:04:35] Trying to send all finished work units
[20:04:35] + No unsent completed units remaining.
[20:04:35] - Autosend completed
[20:06:39] Completed 5000 out of 250000 steps  (2%)
[20:15:12] Completed 7500 out of 250000 steps  (3%)
[20:23:47] Completed 10000 out of 250000 steps  (4%)
[20:32:31] Completed 12500 out of 250000 steps  (5%)
[20:41:12] Completed 15000 out of 250000 steps  (6%)
[20:49:54] Completed 17500 out of 250000 steps  (7%)
[20:58:35] Completed 20000 out of 250000 steps  (8%)
[21:07:16] Completed 22500 out of 250000 steps  (9%)
[21:15:57] Completed 25000 out of 250000 steps  (10%)
[21:24:37] Completed 27500 out of 250000 steps  (11%)
[21:33:18] Completed 30000 out of 250000 steps  (12%)
[21:41:59] Completed 32500 out of 250000 steps  (13%)
[21:50:40] Completed 35000 out of 250000 steps  (14%)
[21:59:20] Completed 37500 out of 250000 steps  (15%)
[22:08:00] Completed 40000 out of 250000 steps  (16%)
[22:16:40] Completed 42500 out of 250000 steps  (17%)
[22:25:23] Completed 45000 out of 250000 steps  (18%)
[22:34:07] Completed 47500 out of 250000 steps  (19%)
[22:42:50] Completed 50000 out of 250000 steps  (20%)
[22:51:34] Completed 52500 out of 250000 steps  (21%)
[23:00:18] Completed 55000 out of 250000 steps  (22%)
[23:09:01] Completed 57500 out of 250000 steps  (23%)
[23:17:43] Completed 60000 out of 250000 steps  (24%)
[23:26:25] Completed 62500 out of 250000 steps  (25%)
[23:35:06] Completed 65000 out of 250000 steps  (26%)
[23:43:47] Completed 67500 out of 250000 steps  (27%)
[23:52:29] Completed 70000 out of 250000 steps  (28%)
[00:01:10] Completed 72500 out of 250000 steps  (29%)
[00:09:51] Completed 75000 out of 250000 steps  (30%)
[00:18:33] Completed 77500 out of 250000 steps  (31%)
[00:27:17] Completed 80000 out of 250000 steps  (32%)
[00:36:01] Completed 82500 out of 250000 steps  (33%)
[00:44:45] Completed 85000 out of 250000 steps  (34%)
[00:53:29] Completed 87500 out of 250000 steps  (35%)
[01:02:13] Completed 90000 out of 250000 steps  (36%)
[01:10:57] Completed 92500 out of 250000 steps  (37%)
[01:19:41] Completed 95000 out of 250000 steps  (38%)
[01:28:25] Completed 97500 out of 250000 steps  (39%)
[01:37:08] Completed 100000 out of 250000 steps  (40%)
[01:45:50] Completed 102500 out of 250000 steps  (41%)
[01:54:32] Completed 105000 out of 250000 steps  (42%)
[02:03:13] Completed 107500 out of 250000 steps  (43%)
[02:04:35] - Autosending finished units... [April 21 02:04:35 UTC]
[02:04:35] Trying to send all finished work units
[02:04:35] + No unsent completed units remaining.
[02:04:35] - Autosend completed
[02:11:54] Completed 110000 out of 250000 steps  (44%)
[02:20:36] Completed 112500 out of 250000 steps  (45%)
[02:29:19] Completed 115000 out of 250000 steps  (46%)
[02:37:58] Completed 117500 out of 250000 steps  (47%)
[02:46:38] Completed 120000 out of 250000 steps  (48%)
[02:55:16] Completed 122500 out of 250000 steps  (49%)
[03:03:53] Completed 125000 out of 250000 steps  (50%)
[03:12:32] Completed 127500 out of 250000 steps  (51%)
[03:21:13] Completed 130000 out of 250000 steps  (52%)
[03:29:55] Completed 132500 out of 250000 steps  (53%)
[03:38:38] Completed 135000 out of 250000 steps  (54%)
[03:47:21] Completed 137500 out of 250000 steps  (55%)
[03:56:04] Completed 140000 out of 250000 steps  (56%)
[04:04:46] Completed 142500 out of 250000 steps  (57%)
[04:13:29] Completed 145000 out of 250000 steps  (58%)
[04:22:12] Completed 147500 out of 250000 steps  (59%)
[04:30:54] Completed 150000 out of 250000 steps  (60%)
[04:39:35] Completed 152500 out of 250000 steps  (61%)
[04:48:15] Completed 155000 out of 250000 steps  (62%)
[04:56:56] Completed 157500 out of 250000 steps  (63%)
[05:05:35] Completed 160000 out of 250000 steps  (64%)
[05:14:16] Completed 162500 out of 250000 steps  (65%)
[05:22:56] Completed 165000 out of 250000 steps  (66%)
[05:31:36] Completed 167500 out of 250000 steps  (67%)
[05:40:17] Completed 170000 out of 250000 steps  (68%)
[05:48:55] Completed 172500 out of 250000 steps  (69%)
[05:57:34] Completed 175000 out of 250000 steps  (70%)
[06:06:11] Completed 177500 out of 250000 steps  (71%)
[06:09:07]
[06:09:07] Folding@home Core Shutdown: INTERRUPTED
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 102) - process 0
[cli_1]: aborting job:
Fatal error in MPI_Sendrecv: Error message texts are not available
[cli_3]: aborting job:
Fatal error in MPI_Sendrecv: Error message texts are not available
[0]0:Return code = 102
[0]1:Return code = 1
[0]2:Return code = 0, signaled with Segmentation fault
[0]3:Return code = 1
[06:09:11] CoreStatus = 66 (102)
[06:09:11] + Shutdown requested by user. Exiting.***** Got a SIGTERM signal (15)
[06:09:11] Killing all core threads

Folding@Home Client Shutdown.

FAHlog:

Code: Select all

[19:49:12] 
[19:49:12] *------------------------------*
[19:49:12] Folding@Home Gromacs SMP Core
[19:49:12] Version 2.06 (Tue Mar 31 08:29:45 PDT 2009)
[19:49:12] 
[19:49:12] Preparing to commence simulation
[19:49:12] - Ensuring status. Please wait.
[19:49:21] - Looking at optimizations...
[19:49:21] - Working with standard loops on this execution.
[19:49:21] - Files status OK
[19:49:22] - Expanded 4837766 -> 24030157 (decompressed 496.7 percent)
[19:49:22] Called DecompressByteArray: compressed_data_size=4837766 data_size=24030157, decompressed_data_size=24030157 diff=0
[19:49:22] - Digital signature verified
[19:49:22] 
[19:49:22] Project: 2671 (Run 20, Clone 52, Gen 11)
[19:49:22] 
[19:49:23] Entering M.D.
[19:49:32] Completed 0 out of 250000 steps  (0%)
[19:58:06] Completed 2500 out of 250000 steps  (1%)
[20:04:35] - Autosending finished units... [April 20 20:04:35 UTC]
[20:04:35] Trying to send all finished work units
[20:04:35] + No unsent completed units remaining.
[20:04:35] - Autosend completed
[20:06:39] Completed 5000 out of 250000 steps  (2%)
[20:15:12] Completed 7500 out of 250000 steps  (3%)
[20:23:47] Completed 10000 out of 250000 steps  (4%)
[20:32:31] Completed 12500 out of 250000 steps  (5%)
[20:41:12] Completed 15000 out of 250000 steps  (6%)
[20:49:54] Completed 17500 out of 250000 steps  (7%)
[20:58:35] Completed 20000 out of 250000 steps  (8%)
[21:07:16] Completed 22500 out of 250000 steps  (9%)
[21:15:57] Completed 25000 out of 250000 steps  (10%)
[21:24:37] Completed 27500 out of 250000 steps  (11%)
[21:33:18] Completed 30000 out of 250000 steps  (12%)
[21:41:59] Completed 32500 out of 250000 steps  (13%)
[21:50:40] Completed 35000 out of 250000 steps  (14%)
[21:59:20] Completed 37500 out of 250000 steps  (15%)
[22:08:00] Completed 40000 out of 250000 steps  (16%)
[22:16:40] Completed 42500 out of 250000 steps  (17%)
[22:25:23] Completed 45000 out of 250000 steps  (18%)
[22:34:07] Completed 47500 out of 250000 steps  (19%)
[22:42:50] Completed 50000 out of 250000 steps  (20%)
[22:51:34] Completed 52500 out of 250000 steps  (21%)
[23:00:18] Completed 55000 out of 250000 steps  (22%)
[23:09:01] Completed 57500 out of 250000 steps  (23%)
[23:17:43] Completed 60000 out of 250000 steps  (24%)
[23:26:25] Completed 62500 out of 250000 steps  (25%)
[23:35:06] Completed 65000 out of 250000 steps  (26%)
[23:43:47] Completed 67500 out of 250000 steps  (27%)
[23:52:29] Completed 70000 out of 250000 steps  (28%)
[00:01:10] Completed 72500 out of 250000 steps  (29%)
[00:09:51] Completed 75000 out of 250000 steps  (30%)
[00:18:33] Completed 77500 out of 250000 steps  (31%)
[00:27:17] Completed 80000 out of 250000 steps  (32%)
[00:36:01] Completed 82500 out of 250000 steps  (33%)
[00:44:45] Completed 85000 out of 250000 steps  (34%)
[00:53:29] Completed 87500 out of 250000 steps  (35%)
[01:02:13] Completed 90000 out of 250000 steps  (36%)
[01:10:57] Completed 92500 out of 250000 steps  (37%)
[01:19:41] Completed 95000 out of 250000 steps  (38%)
[01:28:25] Completed 97500 out of 250000 steps  (39%)
[01:37:08] Completed 100000 out of 250000 steps  (40%)
[01:45:50] Completed 102500 out of 250000 steps  (41%)
[01:54:32] Completed 105000 out of 250000 steps  (42%)
[02:03:13] Completed 107500 out of 250000 steps  (43%)
[02:04:35] - Autosending finished units... [April 21 02:04:35 UTC]
[02:04:35] Trying to send all finished work units
[02:04:35] + No unsent completed units remaining.
[02:04:35] - Autosend completed
[02:11:54] Completed 110000 out of 250000 steps  (44%)
[02:20:36] Completed 112500 out of 250000 steps  (45%)
[02:29:19] Completed 115000 out of 250000 steps  (46%)
[02:37:58] Completed 117500 out of 250000 steps  (47%)
[02:46:38] Completed 120000 out of 250000 steps  (48%)
[02:55:16] Completed 122500 out of 250000 steps  (49%)
[03:03:53] Completed 125000 out of 250000 steps  (50%)
[03:12:32] Completed 127500 out of 250000 steps  (51%)
[03:21:13] Completed 130000 out of 250000 steps  (52%)
[03:29:55] Completed 132500 out of 250000 steps  (53%)
[03:38:38] Completed 135000 out of 250000 steps  (54%)
[03:47:21] Completed 137500 out of 250000 steps  (55%)
[03:56:04] Completed 140000 out of 250000 steps  (56%)
[04:04:46] Completed 142500 out of 250000 steps  (57%)
[04:13:29] Completed 145000 out of 250000 steps  (58%)
[04:22:12] Completed 147500 out of 250000 steps  (59%)
[04:30:54] Completed 150000 out of 250000 steps  (60%)
[04:39:35] Completed 152500 out of 250000 steps  (61%)
[04:48:15] Completed 155000 out of 250000 steps  (62%)
[04:56:56] Completed 157500 out of 250000 steps  (63%)
[05:05:35] Completed 160000 out of 250000 steps  (64%)
[05:14:16] Completed 162500 out of 250000 steps  (65%)
[05:22:56] Completed 165000 out of 250000 steps  (66%)
[05:31:36] Completed 167500 out of 250000 steps  (67%)
[05:40:17] Completed 170000 out of 250000 steps  (68%)
[05:48:55] Completed 172500 out of 250000 steps  (69%)
[05:57:34] Completed 175000 out of 250000 steps  (70%)
[06:06:11] Completed 177500 out of 250000 steps  (71%)
[06:09:07] 
[06:09:07] Folding@home Core Shutdown: INTERRUPTED
[06:09:11] CoreStatus = 66 (102)
[06:09:11] + Shutdown requested by user. Exiting.***** Got a SIGTERM signal (15)
[06:09:11] Killing all core threads

Folding@Home Client Shutdown.

restarting....(although it might actually be done by now, cuz I restarted it quite some time ago, but didn't get a chance to post this earlier)

Post by **toTOW** » Wed Apr 22, 2009 10:29 am

Hi alpha754293 (team 596),
Your WU (P2671 R20 C52 G11) was added to the stats database on 2009-04-21 09:19:06 for 1920 points of credit.

alpha754293 · Post by **alpha754293** » Wed Apr 22, 2009 7:37 pm

toTOW wrote:Hi alpha754293 (team 596),
Your WU (P2671 R20 C52 G11) was added to the stats database on 2009-04-21 09:19:06 for 1920 points of credit.

Curious question - what does the above actually really tell me? Or what should I be getting from it (other than the obvious)?

Is there supposed to be like something would tell me why the system or the WU is getting a seg fault error?

*confused*

Post by **toTOW** » Wed Apr 22, 2009 11:33 pm

It tells you that you returned a valid result that's been credited.

You seg fault issue is still a mystery for me (and none of our tools can tell why they happen ... they even don't see them).

alpha754293 · Post by **alpha754293** » Tue May 12, 2009 4:35 pm

toTOW wrote:It tells you that you returned a valid result that's been credited.

You seg fault issue is still a mystery for me (and none of our tools can tell why they happen ... they even don't see them).

But other than that, there's really not much more that I can do in order to chase down the root cause of the seg faults, right?

Folding Forum

Project: 2671 (Run 20, Clone 52, Gen 11) seg fault

Project: 2671 (Run 20, Clone 52, Gen 11) seg fault

Re: Project: 2671 (Run 20, Clone 52, Gen 11) seg fault

Re: Project: 2671 (Run 20, Clone 52, Gen 11) seg fault

Re: Project: 2671 (Run 20, Clone 52, Gen 11) seg fault

Re: Project: 2671 (Run 20, Clone 52, Gen 11) seg fault