CUDA_ERROR_LAUNCH_FAILED

It seems that a lot of GPU problems revolve around specific versions of drivers. Though NVidia has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: CUDA_ERROR_LAUNCH_FAILED

Post by bruce »

You're asking me to predict how Microsoft's basic logic and NVidia's drivers manage the timeout detection to achieve whatever they think is important. It's not possible to do that without actually looking at the internals or doing a lot of testing. Your question is valid, but i don't have an irrefutable answer ... and it might change in the next version of their code.
gunnarre
Posts: 567
Joined: Sun May 24, 2020 7:23 pm
Location: Norway

Re: CUDA_ERROR_LAUNCH_FAILED

Post by gunnarre »

Is hardware accellerated GPU scheduling switched on?
https://www.windowslatest.com/2020/06/2 ... cheduling/
Image
Online: GTX 1660 Super, GTX 1080, GTX 1050 Ti 4G OC, RX580 + occasional CPU folding in the cold.
Offline: Radeon HD 7770, GTX 960, GTX 950
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

gunnarre wrote:Is hardware accellerated GPU scheduling switched on?
https://www.windowslatest.com/2020/06/2 ... cheduling/
Nope. And, looking at my other hosts, for the ones that have 1080/2080 GPUs, it seems the default value is OFF, and I have not turned it on for any host.

Did you suspect that an ON setting could be the culprit? Or, that an ON setting would be a better choice to avoid this situation?

Thanks!
Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: CUDA_ERROR_LAUNCH_FAILED

Post by bruce »

The setting is new and there has been a lot of guessing about what it does. Hearing the results (if any) of changing that setting will add to the body of knowledge and maybe we'll come up with a recommendation.
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

bruce wrote:The setting is new and there has been a lot of guessing about what it does. Hearing the results (if any) of changing that setting will add to the body of knowledge and maybe we'll come up with a recommendation.
OK. I turned it ON on my 4 hosts that have 10XX/20XX/30XX GPUs (seems it is not supported on Titan Z nor on Titan X), and run a 2004 or 20H2 version of Win10. If one of them starts behaving badly, I'll know.

Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

PantherX wrote:As a first step, I would finish all Slots and then then only fold on 1 GPU. Assuming you initially encountered the CUDA_ERROR_LAUNCH_TIMEOUT (702) issue once a week, see if you can encounter this issue again while folding on a single GPU. If you can fold an entire week without having that issue, then start up a second GPU. Continue until you encounter that issue.
Tuna_Ertemalp wrote:Thanks for the tip about pause-on-start per GPU. Updated FAH, started GPU #1, muted others. I'll report back about the CUDA error here when I have something new...
So, for the last 5 days, I ran GPU#1 on its own for a few days. No problem. Then GPU#2 for a few days. No problem. Then GPU#3 for a few days, and the problem showed up.

Code: Select all

05:56:11:WU02:FS03:0x22:An exception occurred at step 61949: Error downloading array energySum: CUDA_ERROR_LAUNCH_TIMEOUT (702)
05:56:11:WU02:FS03:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
05:56:11:WU02:FS03:0x22:Folding@home Core Shutdown: CORE_RESTART
But, that doesn't necessarily mean that the hardware GPU#3 is at fault, since during my original reporting on this thread, this had happened on GPU #4, then on #1 and #2 in short order while GPU #4 was still in the failed state, therefore resulting 3 of 4 GPUs showing the crash report concurrently, while running under my usual QuadGPU config. Now #3 failed, in a SingleGPU config.

I grabbed both the old "#4 failed on Quad" and the new "#3 failed in Single" logs, both full and filtered for the relevant GPU, and placed them under https://1drv.ms/u/s!AvC041C64j0eyINf-i6 ... Q?e=Nratne.

I am leaving my host in the failure state ("FahCore_22.exe has stopped working" for GPU#3) in case someone needs to harvest something else.

Thanks
Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: CUDA_ERROR_LAUNCH_FAILED

Post by bruce »

Which FAHClient 7.6.xx are you running. Have you updated recently?
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

bruce wrote:Which FAHClient 7.6.xx are you running. Have you updated recently?
I try to use always the latest released. Therefore 7.6.21. But it had been happening under 7.6.20. Unfortunately, I don't remember if it was happening before 7.6.20.

Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

Tuna_Ertemalp wrote:I am leaving my host in the failure state ("FahCore_22.exe has stopped working" for GPU#3) in case someone needs to harvest something else.
Given no takers for harvesting :), I have restarted the machine with only the GPU#4 running.

Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

Tuna_Ertemalp wrote:I have restarted the machine with only the GPU#4 running.
After running #4 single for more than a day, I got no problem, restarted all GPUs in parallel, and again got it to crash overnight, this time for GPUs #1, #3, #4. If you look at the early parts of this thread, it was #1, #2, #4, at that time. And, during "single CPU" trials, it happened to happen on #3. So, it jumps around.

I placed the full log into the same place: https://1drv.ms/u/s!AvC041C64j0eyINf-i6 ... Q?e=Nratne

Again leaving the host in this state for a while to see if there is any interest of harvesting any other info.

I am thinking an instrumented debug binary given to me by a dev would serve much better in figuring this out. I don't think I am the only one running into this, although it might be rare...

Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

Tuna_Ertemalp wrote:I am thinking an instrumented debug binary given to me by a dev would serve much better in figuring this out. I don't think I am the only one running into this, although it might be rare...
Since, again, no takers, I OK'd all crash dialogs, and all GPUs are now continuing.

Since this will keep happening, my current thinking is to "Finish" all GPU projects, and then disable CUDA on all of them, just to see if that will work: "FAHControl -> Configure -> Expert -> Click Add under Extra Core Options -> -disable-cuda -> OK -> Save"

I will report my findings.

Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: CUDA_ERROR_LAUNCH_FAILED

Post by PantherX »

BTW, if you were to publish the PRCGs of the WUs that failed, we can potentially pass that on to the researchers for further inspection. That's assuming that all the information was captured in the upload. This is what a PRCG looks like in the log file:
Project: 16929 (Run 0, Clone 8, Gen 2)
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

Heading to bed in a minute, so I will do that tomorrow. I am guessing they are already in the logs I published, right? So, you'd like me to find them and post here?

Thanks
Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
PantherX
Site Moderator
Posts: 7020
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: CUDA_ERROR_LAUNCH_FAILED

Post by PantherX »

Ah, you're right. I grabbed the data from the OneDrive link you posted and have notified few people about this. Let's wait and see what happens :)
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Tuna_Ertemalp
Posts: 68
Joined: Sun Mar 22, 2020 8:54 pm
Hardware configuration: OS:Win10
GPUs: EVGA

CPU (cores), RAM, (GPU Core OC, Mem OC): GPU(s), Motherboard:

* AMD Ryzen 5 3600 (6C), 32G DDR4-2400, (+0,+0): 3090 FTW3 ULTRA, Gigabyte AB350M-D3H-CF
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+0,+0): 3090 XC3 ULTRA HYBRID, ASUS X99-M WS
* Intel Core i7 5960X (8C), 32G DDR4-2400, (+100,+200): 2x 3090 FTW3 ULTRA, ASUS X99-E WS/USB 3.1
* Intel Core i7 970 (6C), 24G DDR3-1333, (+0,+0): 2x 3080 FTW3 ULTRA HYBRID, ASUS RAMPAGE III GENE
* Intel Core i7 5960X (8C), 16G DDR4-2400, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, ASRock X99 OC Formula/3.1
* AMD Ryzen 7 2700X (8C), 16G DDR4-2666, (+100,+200): 3090 FTW3 ULTRA HYBRID, ASRock B450M Pro4
* AMD Ryzen TR 1950X (16C), 32G DDR4-2133, (+100,+200): 3x 3090 XC3 ULTRA HYBRID, ASRock X399 Taichi
* Intel Core i7 5960X (8C), 64G DDR4-2133, (+100,+0): 1080 Ti FTW3 + HYBRID KIT, 2x 1080 Ti SC2 HYBRID, MSI X99A XPOWER AC
Location: Seattle, WA, USA

Re: CUDA_ERROR_LAUNCH_FAILED

Post by Tuna_Ertemalp »

Awesome! Thank you for saving me the trouble... :)

In the meantime, for the last 7.5hrs, all 4 GPUs are ticking along without a crash, with CUDA disabled. Fingers crossed.

I'll report back IF there is a crash. Otherwise, it would be safe to assume it is working.

If there is anything from the developers/researchers, I am all ears and am open to testing anything they want to throw my way.

Tuna
Small things make quality, but quality is no small thing. (Adapted from Henry Royce talking about perfection, not quality)
8 Win10 PCs/22 slots: 8x CPUs (3xAMD+5xIntel=68C/122T), 14x NVIDIA EVGA GPUs (8x 3090, 2x 3080, 4x 1080Ti) [Details in my profile]
Image
Post Reply