loader: enable the AMD CPU instruction patcher on macOS (#1014)

* loader: enable AMD CPU instruction patching on macOS

The guest instruction patcher was compiled out on macOS and runtimeLinker forced emulate_amd off,
so under Rosetta every SSE4a EXTRQ/INSERTQ took the SIGILL path (about 18 us each). Build the
patcher on all hosts and let --amd-cpu enable it on macOS like on Windows and Linux. The host
feature probe already reports no SSE4a under Rosetta, so EXTRQ becomes a native trampoline and the
remaining fallbacks keep using the x64 instruction emulator through the host exception handler.

* tests: macos_sse4a covers the guest instruction patcher on macOS

Patch EXTRQ/INSERTQ blobs in an RWX area with a trampoline page behind it, with the host features of
Rosetta (no SSE4a/RDPID/CLWB), and compare against the SIGILL emulator: EXTRQ must become a native
trampoline without SIGILL, INSERTQ stays a trap handled by the emulator, and nothing is touched
without emulate_amd.

* loader: emulate the patcher's VRSQRTPS trap and MWAITX on macOS

The AMD CPU patcher marks VRSQRTPS it cannot relocate by clearing a reserved
VEX.vvvv bit, which raises #UD on every host including Rosetta. The macOS build
left TryEmulateReciprocalSquareRoot and TryEmulateMonitorxMwaitx out, so the
guest died with an unhandled SIGILL at the first such site.

Both are plain SSE2/libc, so enable them on macOS. The Darwin context also
gains ClearUpperYmm so the VEX.128 result zeroes YMM[255:128] when the thread
has the AVX context. macos_sse4a covers the trapped VRSQRTPS and MWAITX.

* tests: macos_sse4a runs a plain VEX.128 VRSQRTPS through the patcher

The existing case started from an instruction that was already marked as a trap. The
new cases feed the ordinary encoding to PatchGuestInstructions and compare the result
with the emulated reference, both through a native trampoline and, when the instruction
cannot be relocated, through the trap mark and the signal emulator.

* loader: compact macOS instruction patcher coverage

Share the emulator dispatch chain while keeping SHA emulation limited to non-Apple hosts and preserving handler order. Remove a duplicate VRSQRTPS signal test already covered by both native and trapped patcher cases, plus blank lines left by removed platform guards.

---------

Co-authored-by: nmzik <Nmzik@mail.ru>
This commit is contained in:
Bartłomiej Wołyńczyk
2026-10-06 19:55:11 +02:00
committed by GitHub
parent 4852721005
commit 35c6e8fff6
5 changed files with 244 additions and 43 deletions
+15 -8
View File
@@ -12,6 +12,7 @@
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
#include <windows.h> // IWYU pragma: keep
#elif defined(__APPLE__)
#include <sched.h>
#include <sys/ucontext.h>
#else
#include <sched.h>
@@ -468,6 +469,16 @@ struct Context {
default: return nullptr;
}
}
// The kernel hands AVX threads the longer context that carries YMM[255:128]; a plain SSE
// context has no such state to clear.
void ClearUpperYmm(uint8_t index) const {
if (index >= 16 || native->uc_mcsize < sizeof(_STRUCT_MCONTEXT_AVX64)) {
return;
}
auto* avx = reinterpret_cast<_STRUCT_MCONTEXT_AVX64*>(native->uc_mcontext);
auto* ymmh = &avx->__fs.__fpu_ymmh0;
ymmh[index] = {};
}
#else
ucontext_t* native;
@@ -703,8 +714,6 @@ static bool TryEmulateCpuExtensions(Context& context) {
return true;
}
#if !defined(__APPLE__)
static bool TryEmulateMonitorxMwaitx(Context& context) {
const auto* rip = reinterpret_cast<const uint8_t*>(context.Rip());
if (rip[0] != 0x0f || rip[1] != 0x01 || (rip[2] != 0xfa && rip[2] != 0xfb)) {
@@ -790,8 +799,6 @@ static bool TryEmulateReciprocalSquareRoot(Context& context) {
return true;
}
#endif
bool TryEmulate(void* native_context) {
if (native_context == nullptr) {
return false;
@@ -807,12 +814,12 @@ bool TryEmulate(void* native_context) {
#else
Context context {static_cast<ucontext_t*>(native_context)};
#endif
#if !defined(__APPLE__)
return TryEmulateReciprocalSquareRoot(context) || TryEmulateMonitorxMwaitx(context) ||
TryEmulateSse4a(context) || TryEmulateShaNi(context) || TryEmulateCpuExtensions(context);
#else
return TryEmulateSse4a(context) || TryEmulateCpuExtensions(context);
TryEmulateSse4a(context) ||
#if !defined(__APPLE__)
TryEmulateShaNi(context) ||
#endif
TryEmulateCpuExtensions(context);
}
} // namespace Loader::X64InstructionEmulator