FahCore_a7.exe fails on Windows Azure (hyper-v) server

Moderators: Site Moderators, FAHC Science Team

Post Reply
cs1818
Posts: 2
Joined: Thu Oct 20, 2016 11:17 pm

ERROR: rtdscp CPU instruction

Post by cs1818 »

Running as a service on Windows Server 2012 r2, clean install. Everything works initially, but ends up crashing every time. I can send the events logged in Windows, and the log file if needed. Please help! From the log file in program data directory:


ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212

The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
Last edited by cs1818 on Thu Oct 20, 2016 11:33 pm, edited 1 time in total.
cs1818
Posts: 2
Joined: Thu Oct 20, 2016 11:17 pm

Re: ERROR: rtdscp CPU instruction

Post by cs1818 »

Additional Info: it's running in Hyper-V with NUMA, 32 cores, 48GB RAM, on a Dell R620 physical machine.
JimboPalmer
Posts: 2573
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: ERROR: rtdscp CPU instruction

Post by JimboPalmer »

this is for some other hypervisor, but illustrates the problem and how they solved it. https://xenbits.xen.org/docs/4.3-testin ... scmode.txt

I do not know what options Hyper V may need to correctly emulate rdtscp.
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
alius
Posts: 5
Joined: Tue Sep 13, 2011 1:11 am

FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by alius »

Windows server 2012 R2 datacenter under Hyper-V. 2 CPU (Xeon E5-2673 v3)

Error as follows:

Code: Select all

18:32:14:WU00:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
18:32:14:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
18:32:14:WU00:FS00:0xa7:ERROR:For more information and tips for troubleshooting, please check the GROMACS
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:website at http://www.gromacs.org/Documentation/Errors
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:15:WARNING:WU00:FS00:FahCore returned an unknown error code which probably indicates that it crashed
18:32:15:WARNING:WU00:FS00:FahCore returned: UNKNOWN_ENUM (-1073741784 = 0xc0000028)
System info:

Code: Select all

18:32:12:******************************* System ********************************
18:32:12:          CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
18:32:12:       CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
18:32:12:         CPUs: 16
18:32:12:       Memory: 32.00GiB
18:32:12:  Free Memory: 30.44GiB
18:32:12:      Threads: WINDOWS_THREADS
18:32:12:   OS Version: 6.2
18:32:12:  Has Battery: false
18:32:12:   On Battery: false
18:32:12:   UTC Offset: 0
18:32:12:          PID: 4740
18:32:12:          CWD: C:/Windows/system32
18:32:12:           OS: Windows Server 2012 R2 Datacenter
18:32:12:      OS Arch: AMD64
18:32:12:         GPUs: 0
18:32:12:         CUDA: Not detected
18:32:12:Win32 Service: true
18:32:12:***********************************************************************
Head of the log where the core tries to start:

Code: Select all

18:32:12:WU00:FS00:0xa7:*********************** Log Started 2016-12-02T18:32:12Z ***********************
18:32:12:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
18:32:12:WU00:FS00:0xa7:       Type: 0xa7
18:32:12:WU00:FS00:0xa7:       Core: Gromacs
18:32:12:WU00:FS00:0xa7:    Website: http://folding.stanford.edu/
18:32:12:WU00:FS00:0xa7:  Copyright: (c) 2009-2016 Stanford University
18:32:12:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:32:12:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 704 -lifeline 3032 -checkpoint 15 -np
18:32:12:WU00:FS00:0xa7:             16 -service
18:32:12:WU00:FS00:0xa7:     Config: <none>
18:32:12:WU00:FS00:0xa7:************************************ Build *************************************
18:32:12:WU00:FS00:0xa7:    Version: 0.0.11
18:32:12:WU00:FS00:0xa7:       Date: Sep 21 2016
18:32:12:WU00:FS00:0xa7:       Time: 01:43:48
18:32:12:WU00:FS00:0xa7: Repository: Git
18:32:12:WU00:FS00:0xa7:   Revision: 957bd90e68d95ddcf1594dc15ff6c64cc4555146
18:32:12:WU00:FS00:0xa7:     Branch: master
18:32:12:WU00:FS00:0xa7:   Compiler: GNU 4.2.1 Compatible Clang 3.9.0 (trunk 274080)
18:32:12:WU00:FS00:0xa7:    Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
18:32:12:WU00:FS00:0xa7:             -fno-unsafe-math-optimizations -msse2 -I/mingw64/include
18:32:12:WU00:FS00:0xa7:             -Wno-inconsistent-dllimport -Wno-parentheses-equality
18:32:12:WU00:FS00:0xa7:             -Wno-deprecated-register -Wno-unused-local-typedef
18:32:12:WU00:FS00:0xa7:   Platform: linux2 4.6.0-1-amd64
18:32:12:WU00:FS00:0xa7:       Bits: 64
18:32:12:WU00:FS00:0xa7:       Mode: Release
18:32:12:WU00:FS00:0xa7:       SIMD: avx_256
18:32:12:WU00:FS00:0xa7:************************************ System ************************************
18:32:12:WU00:FS00:0xa7:        CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
18:32:12:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
18:32:12:WU00:FS00:0xa7:       CPUs: 16
18:32:12:WU00:FS00:0xa7:     Memory: 32.00GiB
18:32:12:WU00:FS00:0xa7:Free Memory: 30.46GiB
18:32:12:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
18:32:12:WU00:FS00:0xa7: OS Version: 6.2
18:32:12:WU00:FS00:0xa7:Has Battery: false
18:32:12:WU00:FS00:0xa7: On Battery: false
18:32:12:WU00:FS00:0xa7: UTC Offset: 0
18:32:12:WU00:FS00:0xa7:        PID: 2164
18:32:12:WU00:FS00:0xa7:        CWD: C:\Users\Alius\AppData\Roaming\FAHClient\work
18:32:12:WU00:FS00:0xa7:         OS: Windows Server 2012 R2 Datacenter
18:32:12:WU00:FS00:0xa7:    OS Arch: AMD64
18:32:12:WU00:FS00:0xa7:********************************************************************************
18:32:12:WU00:FS00:0xa7:Project: 11920 (Run 957, Clone 4, Gen 112)
18:32:12:WU00:FS00:0xa7:Unit: 0x00000089ab4041295809c36e95f41303
18:32:12:WU00:FS00:0xa7:Reading tar file core.xml
18:32:12:WU00:FS00:0xa7:Reading tar file frame112.tpr
18:32:12:WU00:FS00:0xa7:Digital signatures verified
18:32:12:WU00:FS00:0xa7:Calling: mdrun -s frame112.tpr -o frame112.trr -cpt 15 -nt 16
18:32:14:WU00:FS00:0xa7:Steps: first=8960000 total=80000
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:WARNING:Unexpected exit() call
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:-------------------------------------------------------
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20160919-669094a-unknown
18:32:14:WU00:FS00:0xa7:ERROR:
18:32:14:WU00:FS00:0xa7:ERROR:Fatal error:
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:Source code file: /host/windows-cross-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/gmxlib/gmx_detect_hardware.c, line: 212
18:32:14:WU00:FS00:0xa7:ERROR:Fatal error:
18:32:14:WU00:FS00:0xa7:ERROR:The GROMACS executable was compiled to use the rdtscp CPU instruction. However, this is not supported by the current hardware and continuing would lead to a crash. Please rebuild GROMACS with the GMX_USE_RDTSCP=OFF CMake option.
Every time this VM pulls down an a7 it fails the same way until it finds an a4. If there is any other information I can provide I would be happy to do so. Cheers
alius
Posts: 5
Joined: Tue Sep 13, 2011 1:11 am

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by alius »

Unfortunately, I do not have control over the hyper-v hypervisor settings for this host, as this is a Windows Azure cloud server. If the situation arises that I try this with Hyper-V running on hardware and a host OS that I control, I will definitely poke around and report back if I find the right setting(s). This may even be a flaw in hyper-v itself, but I thought the FAH developers should be made aware of this issue.

There should be a way to detect that the client is running under hyper-v, and utilize the alternate GROMACS compiled using the GMX_USE_RDTSCP=OFF flag, so that donors running under hyper-v can successfully complete these WUs once the core gets put on non-test projects. https://blogs.technet.microsoft.com/ton ... side-a-vm/ may be helpful if that is an avenue that might be pursued.
Last edited by alius on Sat Dec 03, 2016 7:55 pm, edited 1 time in total.
Joe_H
Site Admin
Posts: 7856
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by Joe_H »

Unfortunately the recent releases of the Gromacs code do not include the ability to compile with conditional execution paths depending on the CPU hardware being used. So a folding core is built and compiled for each target platform's capabilities. Right now that is two separate downloads for each OS type, one for CPU's that support AVX2 and the other for older hardware that does not and therefore uses SSE2 computations. So the detection could not be built into the folding core but would need to be in the client to download and use a core suitable for use in a VM without full access to the detected type of CPU hardware.

We can bring this issue to the attention of the developer of the A7 core. He may be able to add support for this type of configuration, or to at least not assign to it, but that might not be easy or quick.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
alius
Posts: 5
Joined: Tue Sep 13, 2011 1:11 am

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by alius »

Joe_H wrote: We can bring this issue to the attention of the developer of the A7 core. He may be able to add support for this type of configuration, or to at least not assign to it, but that might not be easy or quick.

These things rarely are! Just wanted to provide as much info as I could in the hope it might help. Thanks!
bruce
Posts: 20910
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by bruce »

Thanks for reporting this.

I can inform you that code has been added to the upcoming v4.7.16 client to detect the rtdscp CPU instruction and code has been added to an upcoming version of the Assignment Server to process that report. These changes will be rolled out "soon"© and once both are in place (i.e. - at some point in the future) it'll be fixed.
JimboPalmer
Posts: 2573
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by JimboPalmer »

Thank you for taking care of this, and of us!
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
alius
Posts: 5
Joined: Tue Sep 13, 2011 1:11 am

Re: FahCore_a7.exe fails on Windows Azure (hyper-v) server

Post by alius »

JimboPalmer wrote:Thank you for taking care of this, and of us!
Seconded, great news! Thank you!
Post Reply