Project 781 (Run 31 Clone 42 Gen 0) Very Slow.

[Owen_Huggard]

Hi All,

Have recently installed Folding@Home on an AMD 64 Sempron 3600+ with 512MB RAM running Ubuntu 8.04.

I started the FAH core using the -verbosity 9 flag
I've been assigned the following project which has been running for the past 10-11 hours:

Project 781 (Run 31 Clone 42 Gen 0)
500000 steps, 1000ps.

The problem is that the project seems to be progressing by 1% every 4.5 to 5 hours. If it continues at this rate it ought to take another 3 weeks assuming I leave my PC on at all times.

My questions are:

1) Is this normal for this particular project?
2) Is my PC a little under-powered for this particular job (if so should I set the verbosity to a value less than 9, or change the configuration to allow only small packets instead of normal?)
3) Whats the deadline for this particular project? If Idecide to leave it running and it ends up taking 2 Months (I plan on shutting it odwn from time to time), will I miss the deadline?

Regards,

Owen

[John Naylor]

Welcome to the forums and the fold!

Could you post the first 30 lines of the log file showing the startup with this unit inside

Code: Select all

tags? That may help to explain the slowness. To answer your questions:

1) No, it is not normal.
2) No, that CPU is plenty powerful enough to complete this unit well within the deadline. Also both those settings you mentioned do not affect Work Unit length: -verbosity x just changes the amount of information printed to the log file (using -verbosity 9 can help immensely when troubleshooting, like now), and the download size refers to just that: download size. Download size and unit time to process are not connected in any way.
3) The preferred deadline is 20 days (deadlines can be found here or by clicking "Project Summary" at the top of this forum), hence why taking so long for the unit to process is unusual. Whilst you will see a longer final deadline on that page, if the unit is not returned before the preferred deadline it is reissued to another computer, (to help speed the science) although you still get points until the final deadline.

One possible solution though - If you installed the graphical client, please keep the viewer closed as much as possible. It may or may not work for you, but regardless of whether it works it is a huge CPU hog while in use, and if you have been using it that would go a long way to explaining the slow processing times.

[Flathead74]

Screen savers can also be CPU hogs while they are in use, and cause long frame/step times.
Not sure if you use one or not, just a thought.

[Owen_Huggard]

Thanks for the help. Was afraid that I was going top have to leave my PC running an awful long time to get this done

To answer your questions:

No, I'm not running any graphical clients. I'm running the FAH from the command line shell. Essentially I just opened the linux bash and typed in the following:

Code: Select all

./fah6-verbosity 9 

while in my folding directory.

I'm not running any screen savers either. I've got all graphical settings on my PC set to the minimum.

I do however have an Squeezebox music server running in the background. Though this doesn't really chew up much resources, I might try killing it and re-starting the folding@home service to see what happens.

Here's my log. Sorry for not just posting the first 30 lines, but I wasn't sure which parts were relevant and the first 30 lines were just the client configuration, and I subsequently stopped and re-started the client a number of times. I've tried to shorten it a little by removing repetitive messages (where I've written etc.)

Code: Select all

--- Opening Log file [January 30 18:31:17] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -configonly 

[18:31:17] Configuring Folding@Home...


[18:31:58] - Ask before connecting: No
[18:31:58] - User name: Owen (Team 0)
[18:31:58] - User ID not found locally
[18:31:58] + Requesting User ID from server
[18:31:59] - Machine ID: 1
[18:31:59] 
[18:31:59] -configonly flag given, so exiting.


--- Opening Log file [January 30 18:32:25] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[18:32:25] - Ask before connecting: No
[18:32:25] - User name: Owen (Team 0)
[18:32:25] - User ID: C4075B9726D5CF7
[18:32:25] - Machine ID: 1
[18:32:25] 
[18:32:25] Work directory not found. Creating...
[18:32:25] Could not open work queue, generating new queue...
[18:32:25] - Autosending finished units...
[18:32:25] Trying to send all finished work units
[18:32:25] + No unsent completed units remaining.
[18:32:25] - Autosend completed
[18:32:25] - Preparing to get new work unit...
[18:32:25] + Attempting to get work packet
[18:32:25] - Will indicate memory of 502 MB
[18:32:25] - Detect CPU. Vendor: AuthenticAMD, Family: 15, Model: 15, Stepping: 2
[18:32:25] - Connecting to assignment server
[18:32:25] Connecting to http://assign.stanford.edu:8080/
[18:32:26] Posted data.
[18:32:26] Initial: 40AB; - Successful: assigned to (171.64.122.138).
[18:32:26] + News From Folding@Home: Welcome to Folding@Home
[18:32:26] Loaded queue successfully.
[18:32:26] Connecting to http://171.64.122.138:8080/
[18:32:28] Posted data.
[18:32:28] Initial: 0000; - Receiving payload (expected size: 961590)
[18:32:37] - Downloaded at ~104 kB/s
[18:32:37] - Averaged speed for that direction ~104 kB/s
[18:32:37] + Received work.
[18:32:37] + Closed connections
[18:32:37] 
[18:32:37] + Processing work unit
[18:32:37] Core required: FahCore_a0.exe
[18:32:37] Core not found.
[18:32:37] - Core is not present or corrupted.
[18:32:37] - Attempting to download new core...
[18:32:37] + Downloading new core: FahCore_a0.exe
[18:32:37] Downloading core (/~pande/Linux/x86/Core_a0.fah from www.stanford.edu)
[18:32:38] Initial: AFDE; + 10240 bytes downloaded
etc...
[18:33:31] Initial: AF9E; + 980952 bytes downloaded
[18:33:32] Verifying core Core_a0.fah...
[18:33:32] Signature is VALID
[18:33:32] 
[18:33:32] Trying to unzip core FahCore_a0.exe
[18:33:33] Decompressed FahCore_a0.exe (3055460 bytes) successfully
[18:33:33] + Core successfully engaged
[18:33:38] 
[18:33:38] + Processing work unit
[18:33:38] Core required: FahCore_a0.exe
[18:33:38] Core found.
[18:33:38] Working on Unit 01 [January 30 18:33:38]
[18:33:38] + Working ...
[18:33:38] - Calling './FahCore_a0.exe -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 15368 -version 602'

[18:33:38] 
[18:33:38] *------------------------------*
[18:33:38] Folding@Home Gromacs 3.3 Core
[18:33:38] Version 1.92 (April 17. 2007)
[18:33:38] 
[18:33:38] Preparing to commence simulation
[18:33:38] - Looking at optimizations...
[18:33:38] - Created dyn
[18:33:38] - Files status OK
[18:33:39] - Expanded 961078 -> 6252409 (decompressed 650.5 percent)
[18:33:39] - Starting from initial work packet
[18:33:39] 
[18:33:39] Project: 781 (Run 31, Clone 42, Gen 0)
[18:33:39] 
[18:33:39] Assembly optimizations on if available.
[18:33:39] Entering M.D.
[18:33:46] Protein: Mini chaperonin
[18:33:46] Writing local files
[18:33:47] Extra 3DNow boost OK.
[18:33:47] Extra SSE boost OK.
[18:33:48] Writing local files
[18:33:49] Completed 0 out of 500000 steps  (0 percent)
[18:48:50] Timered checkpoint triggered.
[18:58:40] ***** Got an Activate signal (2)
[18:58:40] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [January 30 19:06:50] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[19:06:50] - Ask before connecting: No
[19:06:50] - User name: Owen (Team 0)
[19:06:50] - User ID: C4075B9726D5CF7
[19:06:50] - Machine ID: 1
[19:06:50] 
[19:06:50] Loaded queue successfully.
[19:06:50] - Autosending finished units...
[19:06:50] Trying to send all finished work units
[19:06:50] + No unsent completed units remaining.
[19:06:50] - Autosend completed
[19:06:50] 
[19:06:50] + Processing work unit
[19:06:50] Core required: FahCore_a0.exe
[19:06:50] Core found.
[19:06:50] Working on Unit 01 [January 30 19:06:50]
[19:06:50] + Working ...
[19:06:50] - Calling './FahCore_a0.exe -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 15817 -version 602'

[19:06:50] 
[19:06:50] *------------------------------*
[19:06:50] Folding@Home Gromacs 3.3 Core
[19:06:50] Version 1.92 (April 17. 2007)
[19:06:50] 
[19:06:50] Preparing to commence simulation
[19:06:50] - Looking at optimizations...
[19:06:50] - Files status OK
[19:06:51] - Expanded 961078 -> 6252409 (decompressed 650.5 percent)
[19:06:51] 
[19:06:51] Project: 781 (Run 31, Clone 42, Gen 0)
[19:06:51] 
[19:06:51] Assembly optimizations on if available.
[19:06:51] Entering M.D.
[19:07:13] (Starting from checkpoint)
[19:07:13] Protein: Mini chaperonin
[19:07:13] Writing local files
[19:07:13] Completed 489 out of 500000 steps  (0%)
[19:07:14] Extra 3DNow boost OK.
[19:07:14] Extra SSE boost OK.


--- Opening Log file [January 30 19:12:27] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -configonly 

[19:12:27] - Ask before connecting: No
[19:12:27] - User name: Owen (Team 0)
[19:12:27] - User ID: C4075B9726D5CF7
[19:12:27] - Machine ID: 1
[19:12:27] 
[19:12:27] Configuring Folding@Home...


[19:12:48] - Ask before connecting: No
[19:12:48] - User name: Owen_Huggard (Team 0)
[19:12:48] - User ID: C4075B9726D5CF7
[19:12:48] - Machine ID: 1
[19:12:48] 
[19:12:48] -configonly flag given, so exiting.


--- Opening Log file [January 30 19:12:53] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[19:12:53] - Ask before connecting: No
[19:12:53] - User name: Owen_Huggard (Team 0)
[19:12:53] - User ID: C4075B9726D5CF7
[19:12:53] - Machine ID: 1
[19:12:53] 

A potential conflict was detected:

Process 15817 is currently running and may also be a client with Mach. ID 1.
Program will now exit. Upon restart, this check will not be done -- 
you may wish to check that no client is currently running in
/home/owen/folding before restarting.

Please press any key to exit.
[19:13:48] ***** Got a SIGTERM signal (15)
[19:13:48] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [January 30 19:17:37] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[19:17:37] - Ask before connecting: No
[19:17:37] - User name: Owen_Huggard (Team 0)
[19:17:37] - User ID: C4075B9726D5CF7
[19:17:37] - Machine ID: 1
[19:17:37] 
[19:17:37] Loaded queue successfully.
[19:17:37] - Autosending finished units...
[19:17:37] Trying to send all finished work units
[19:17:37] + No unsent completed units remaining.
[19:17:37] - Autosend completed
[19:17:37] 
[19:17:37] + Processing work unit
[19:17:37] Core required: FahCore_a0.exe
[19:17:37] Core found.
[19:17:37] Working on Unit 01 [January 30 19:17:37]
[19:17:37] + Working ...
[19:17:37] - Calling './FahCore_a0.exe -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 6104 -version 602'



--- Opening Log file [January 30 19:23:25] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[19:23:25] - Ask before connecting: No
[19:23:25] - User name: Owen_Huggard (Team 0)
[19:23:25] - User ID: C4075B9726D5CF7
[19:23:25] - Machine ID: 1
[19:23:25] 
[19:23:25] Loaded queue successfully.
[19:23:25] - Autosending finished units...
[19:23:25] Trying to send all finished work units
[19:23:25] + No unsent completed units remaining.
[19:23:25] - Autosend completed
[19:23:25] 
[19:23:25] + Processing work unit
[19:23:25] Core required: FahCore_a0.exe
[19:23:25] Core found.
[19:23:25] Working on Unit 01 [January 30 19:23:25]
[19:23:25] + Working ...
[19:23:25] - Calling './FahCore_a0.exe -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 6370 -version 602'



--- Opening Log file [January 30 19:38:17] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[19:38:17] - Ask before connecting: No
[19:38:17] - User name: Owen_Huggard (Team 0)
[19:38:17] - User ID: C4075B9726D5CF7
[19:38:17] - Machine ID: 1
[19:38:17] 
[19:38:18] Loaded queue successfully.
[19:38:18] - Autosending finished units...
[19:38:18] Trying to send all finished work units
[19:38:18] + No unsent completed units remaining.
[19:38:18] - Autosend completed
[19:38:18] 
[19:38:18] + Processing work unit
[19:38:18] Core required: FahCore_a0.exe
[19:38:18] Core found.
[19:38:18] Working on Unit 01 [January 30 19:38:18]
[19:38:18] + Working ...
[19:38:18] - Calling './FahCore_a0.exe -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 6599 -version 602'

[19:38:18] 
[19:38:18] *------------------------------*
[19:38:18] Folding@Home Gromacs 3.3 Core
[19:38:18] Version 1.92 (April 17. 2007)
[19:38:18] 
[19:38:18] Preparing to commence simulation
[19:38:18] - Ensuring status. Please wait.
[19:38:35] - Looking at optimizations...
[19:38:35] - Working with standard loops on this execution.
[19:38:35] - Previous termination of core was improper.
[19:38:35] - Going to use standard loops.
[19:38:35] - Files status OK
[19:38:36] - Expanded 961078 -> 6252409 (decompressed 650.5 percent)
[19:38:36] 
[19:38:36] Project: 781 (Run 31, Clone 42, Gen 0)
[19:38:36] 
[19:38:36] Entering M.D.
[19:38:58] (Starting from checkpoint)
[19:38:58] Protein: Mini chaperonin
[19:38:58] Writing local files
[19:38:58] Completed 489 out of 500000 steps  (0%)
[19:53:59] Timered checkpoint triggered.
[20:09:01] Timered checkpoint triggered.


--- Opening Log file [January 30 20:10:52] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[20:10:52] - Ask before connecting: No
[20:10:52] - User name: Owen_Huggard (Team 0)
[20:10:52] - User ID: C4075B9726D5CF7
[20:10:52] - Machine ID: 1
[20:10:52] 

A potential conflict was detected:

Process 6599 is currently running and may also be a client with Mach. ID 1.
Program will now exit. Upon restart, this check will not be done -- 
you may wish to check that no client is currently running in
/home/owen/folding before restarting.

Please press any key to exit.


--- Opening Log file [January 30 20:11:47] 


# Linux Console Edition #######################################################
###############################################################################

                       Folding@Home Client Version 6.02

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/owen/folding
Executable: ./fah6
Arguments: -verbosity 9 

[20:11:47] - Ask before connecting: No
[20:11:47] - User name: Owen_Huggard (Team 0)
[20:11:47] - User ID: C4075B9726D5CF7
[20:11:47] - Machine ID: 1
[20:11:47] 
[20:11:47] Loaded queue successfully.
[20:11:47] - Autosending finished units...
[20:11:47] Trying to send all finished work units
[20:11:47] + No unsent completed units remaining.
[20:11:47] - Autosend completed
[20:11:47] 
[20:11:47] + Processing work unit
[20:11:47] Core required: FahCore_a0.exe
[20:11:47] Core found.
[20:11:47] Working on Unit 01 [January 30 20:11:47]
[20:11:47] + Working ...
[20:11:47] - Calling './FahCore_a0.exe -dir work/ -suffix 01 -checkpoint 15 -verbose -lifeline 7302 -version 602'

[20:11:47] 
[20:11:47] *------------------------------*
[20:11:47] Folding@Home Gromacs 3.3 Core
[20:11:47] Version 1.92 (April 17. 2007)
[20:11:47] 
[20:11:47] Preparing to commence simulation
[20:11:47] - Ensuring status. Please wait.
[20:12:04] - Looking at optimizations...
[20:12:04] - Working with standard loops on this execution.
[20:12:04] - Previous termination of core was improper.
[20:12:04] - Going to use standard loops.
[20:12:04] - Files status OK
[20:12:05] - Expanded 961078 -> 6252409 (decompressed 650.5 percent)
[20:12:05] 
[20:12:05] Project: 781 (Run 31, Clone 42, Gen 0)
[20:12:05] 
[20:12:06] Entering M.D.
[20:12:28] (Starting from checkpoint)
[20:12:28] Protein: Mini chaperonin
[20:12:28] Writing local files
[20:12:28] Completed 998 out of 500000 steps  (0%)
[20:27:32] Timered checkpoint triggered.
etc...
[23:58:02] Timered checkpoint triggered.
[00:01:27] Writing local files
[00:01:27] Completed 5000 out of 500000 steps  (1 percent)
[00:16:28] Timered checkpoint triggered.
etc...

Please let me know if you see anything obviously wrong in here.

In the mean time I'll post back soon, to let you know if killing my music server had any positive results.

Thanks,

Owen

[John Naylor]

Gotcha

There seem to be a lot of non-graceful shutdowns. For instance when you shut down your client correctly you get this...

Code: Select all

[18:58:40] ***** Got an Activate signal (2)
[18:58:40] Killing all core threads

Folding@Home Client Shutdown.

Whereas many of your shutdowns just show a checkpoint and then client starting again which means the shutdown was not done correctly. How are you closing the client? The result of this is that the client disables calculation optimisations next time it starts up, which makes processing immensely slow...

Code: Select all

[20:11:47] - Ensuring status. Please wait.
[20:12:04] - Looking at optimizations...
[20:12:04] - Working with standard loops on this execution.
[20:12:04] - Previous termination of core was improper.
[20:12:04] - Going to use standard loops.

If you are shutting the client down in the recommended way I suggest adding a second flag to the client: -forceasm . This disables the client checking for a bad shutdown and forces the optimisations to be used. However only use that if you are sure your system is completely stable (if it can run StressCPU2 for an extended period of time then it is stable enough for F@H).

For that reason I think you can leave the music server running, it won't be the problem

EDIT: Make something clearer.

[John_Weatherman]

Check also how many clients are running - from your log,
"A potential conflict was detected:

Process 6599 is currently running and may also be a client with Mach. ID 1.
Program will now exit. Upon restart, this check will not be done --
you may wish to check that no client is currently running in
/home/owen/folding before restarting"

I would suggest checking everything that's running.

[bruce]

During your first hour I see a lot of starts and restarts, but that's not unexpected. At [18:48:50] you successfully completed 15 minutes of processing and nothing else permanent happened until [19:53:59]. Then at [20:12:28] you let it run for a while and made enough progress to get past 1%.

FAH is designed to run continuously at the lowest possible priority. It assumes that the computer is lightly loaded with other activities which are given priority to use whatever they want. I don't run a media server so I don't know what kind of load it puts on your CPU or RAM.

What does "ps" show? Is something called FahCore_a0 receiving an appreciable percentage of the CPU, or is something else taking it?

I'm going to assume that you have enough RAM so that when you add FAH to your normal processing load, it does not produce continual paging. I'm also going to assume that you do not have a power-savings feature that is slowing your CPU when it thinks you are not using the computer. Either of these assumptions may be wrong and if so, should be addressed in one way or another.

In the last portion of FAHlog.txt, you completed 1% in about 4½ hours which means you can complete that WU in something over 18 days. Project 781 has a Preferred Deadline of 20 days so that computer, with its present workload is a marginal contributor. I don't know if the 4½ hours that was documented is typical. If it is, and if the earlier assumptions are valid, you should consider anything which might lighten the processing load somewhat or upgrading your CPU to something slightly more powerful.

If everything that I see is typical, you can still contribute to FAH but if you are going to be consistently missing the Preferred Deadline, you really won't be helping enough to matter.

[Owen_Huggard]

Well thanks for the pointers everyone.

John Naylor:
Have tried your suggestion of loading with the -forceasm flag. It seems to have doubled the speed at which the fah client runs. It's still pretty slow but hopefully I'll make the deadline on this one.

John_Weatherman :
Concerning the issue with a "potential conflict being detected", that was admittedly me not killing the process correctly from the command line whe I wanted to stop and re-start it. I've since found out how do that properly.

bruce :
Concerning the RAM, I know 512MB is pretty measly these days, but my processes tab shows that my computers RAM isn't fully used with the FAH client open and the Swaps file is tiny. The music server is in sleep mode most of the time (though I've tried killing it to see if it makes an appreciable difference) and the CPU load is typically < 5% when I'm not using the PC and have the fah client stopped, while at 100% whenever it's running, so I'm pretty sure that there's nothing else running that's hogging resources.

So if anyone thinks it's still running suspiciously slow please let me know, though it looks like the only answer is upgrading to a faster PC.

Owen

[codysluder]

Have tried your suggestion of loading with the -forceasm flag. It seems to have doubled the speed at which the fah client runs. It's still pretty slow but hopefully I'll make the deadline on this one.

The key to that issue is that you've got a message :Extra SSE boost OK." or a message about "use standard loops" when the WU starts. The SSE is two to three times a fast, which changes bruce's calculations and his conclusion about being too slow.

Welcome to the fold.