Page 1 of 1
Upload compression
Posted: Tue Apr 01, 2008 4:10 pm
by butc8
Is it possible to implement upload compression especially when you getting 15MB result files for the SMP client p30xx.
Re: Upload compression
Posted: Tue Apr 01, 2008 4:13 pm
by uncle_fungus
Results are already compressed before upload.
Code: Select all
[06:39:15] Finished Work Unit:
[06:39:15] - Reading up to 230280 from "work/wudata_00.arc": Read 230280
[06:39:15] - Reading up to 173868 from "work/wudata_00.xtc": Read 173868
[06:39:15] goefile size: 0
[06:39:15] logfile size: 119813
[06:39:15] Leaving Run
[06:39:20] - Writing 594025 bytes of core data to disk...
[06:39:20] Done: 593513 -> 449135 (compressed to 75.6 percent)
[06:39:20] ... Done.
[06:39:21] - Shutting down core
[06:39:21]
[06:39:21] Folding@home Core Shutdown: FINISHED_UNIT
Re: Upload compression
Posted: Tue Apr 01, 2008 4:24 pm
by butc8
Does it still compress when it does this
Code: Select all
[06:44:27] Finished Work Unit:
[06:44:27] - Reading up to 3721632 from "work/wudata_02.arc": Read 3721632
[06:44:27] - Reading up to 1776020 from "work/wudata_02.xtc": Read 1776020
[06:44:27] goefile size: 0
[06:44:27] logfile size: 25510
[06:44:27] Leaving Run
[06:44:29] - Writing 5527562 bytes of core data to disk...
[06:44:29] ... Done.
[06:44:29] - Failed to delete work/wudata_02.sas
[06:44:29] - Failed to delete work/wudata_02.goe
[06:44:29] Warning: check for stray files
[06:44:29] - Shutting down core
[06:46:29]
[06:46:29] Folding@home Core Shutdown: FINISHED_UNIT
[06:46:29]
[06:46:29] Folding@home Core Shutdown: FINISHED_UNIT
[06:46:32] CoreStatus = 64 (100)
[06:46:32] Unit 2 finished with 80 percent of time to deadline remaining.
[06:46:32] Updated performance fraction: 0.750828
[06:46:32] Sending work to server
Re: Upload compression
Posted: Tue Apr 01, 2008 4:31 pm
by 7im
-verbosity 9 on in that example?
Re: Upload compression
Posted: Tue Apr 01, 2008 4:49 pm
by alancabler
butc8 wrote:Is it possible to implement upload compression especially when you getting 15MB result files for the SMP client p30xx.
Some "WUs greater than 5MB in size", or single- client Bonus WUs are known to be around 45MB in size... not for dial- up.
Re: Upload compression
Posted: Tue Apr 01, 2008 4:51 pm
by butc8
7im wrote:-verbosity 9 on in that example?
yip
When will Stanford start compressing results files?
Posted: Fri May 16, 2008 5:52 pm
by Reddog99
I have two computers running SMP units and the size of the results files returned is about to cause me to be penalized by my ISP because of the excess bandwidth being used. I'm on a satellite connection and they allow me 2.3 gb upload bandwidth, and I'm using about 3.0 gb.
I experimented with the smp result files and found that the 22 mb file sizes can be reduced to about 15 mb by compressing using "7zip". So why isn't Stanford compressing the files? With result file sizes seeming to grow, this could become a problem for more people than just me.
Pat
Merged new post with existing topic. PM Sent. 7im
Re: Upload compression
Posted: Fri May 16, 2008 8:35 pm
by David
I suspect they need to find a balance between computational power needed to compress units (and hence time to compress) vs file sizes. I'm not sure how old the compression method used is (ie is it the same as was used for v4? v3? v2? but it may be worth looking at alernatives).