Multiple Failures

[theteofscuba]

Lately, I've been getting quite a bit of MACHINE_UNSTABLE results. Infact, one time a work unit ("Project: 5736 (Run 1, Clone 43, Gen 485)" barfed at 12%, but consequently when the client restarted, the WU completed successfully! I don't know what is going on here, as I'm getting a mixed bag of failures resulting in EUE, failing then completing after restarting, and other work units that simply complete successfully without any trouble. Judging by the fact there were so many different EUEs for projects such as:

Project: 5735 (Run 3, Clone 114, Gen 513)
Project: 5737 (Run 3, Clone 9, Gen 550)
Project: 5741 (Run 3, Clone 35, Gen 629)
Project: 5742 (Run 0, Clone 19, Gen 689)
Project: 5743 (Run 0, Clone 98, Gen 539)
Project: 5744 (Run 3, Clone 95, Gen 249)


I am led to suspect that someone is intentionally feeding back corrupt data to the WU servers in an effort to get points at the expense of the science. On one hand I hope that is the case because it would mean that it is not due to a hardware problem. I am also hoping that this could just be a software bug in the GPU2 client. I am using -forcegpu ati_r700 because I am usng an HD 5870 which is currently unsupported.

[bruce]

I don't think your conspiracy theory is too likely. The servers are really very good at rejecting corrupt data. In fact, we get a number of complaints when the server rejects an upload as corrupt and the donor claims that it couldn't possibly be corrupt.

Nevertheless, whether the servers have an issue or not, you're saying that your hardware does not calculate the same results twice. The fact that a WU barfed at 12% and then restarted and it finished normally points to a problem in your hardware. The most common causes of non-repeatability is a GPU or CPU that is overclocked, overheated, or underpowered. What temperature is your GPU running? How are your 12v rails doing?

[theteofscuba]

I'm ruling CPU overheating out. my other 7 cpu cores are running normally without error. also, this system automatically shuts down and alarms when overheating so i'm fairly confident that its not the cpu, and they are not over clocked or in the red zone temperature wise. as for the gpu on the other hand, i'm pretty sure it is getting plenty of juice from this 1000w psu and is plugged in properly. i'm not sure how to get a temperature reading on it though. also, no overclocking on the gpu. any possibility it is a driver bug? I updated to catalyst 9.11 for windows 7 32bit.

edit:

ati catalyst control center reports a steady ~ 80 C temperature. the guage is pretty much half way in between super cool and max temp, which makes temperature appear to not be the issue at this time.

[theteofscuba]

Ah, found the problem. The GPU was not fully seated in its pci-e slot. I expected problems like that to be detected at boot time with some kind of alarm or obvious sign that something is wrong instead of these subtle, unpredictable and unreproducable errors that I was encountering.

[7im]

I'm glad to see a more realistic cause to the problem (and solution!) has been found.