fix: keep hint-less guest mappings at the canonical PS5 base (fixes the #135 macOS regression) (#138)

* fix: keep hint-less guest mappings at the canonical PS5 base

FindGuestFreeRange searched the low system-managed range first for
mappings with no address hint, so the first hint-less direct-memory map
could land as low as 0x200000. The PS5 kernel never places hint-less
user mappings below 0x200000000 and guest code relies on that: Sony's
libc maps 4 MiB of direct memory for its internal heap, fails its
mspace setup when the returned address is that low, and the first
malloc then dereferences a null mspace (a read at 0x38, the mspace
magic check). On macOS this made Raiden III crash on the main guest
thread a couple of seconds after boot, 100 percent reproducible with
--printf-direction Silent.

Search from the canonical base first, fall back to the user range, and
keep the low system-managed range only as a last resort. The mmap path
already anchored hint-less searches at 0x200000000; this aligns the
shared search helper with it.

Adds two regression tests: the libc-shaped allocation must come back at
or above the canonical base and hold writes, and direct-memory content
must survive an unmap and remap of the same physical range.

* macos: make the fatal-report memory dumps fault-safe

IsReadableRange returned true for any nonzero address on macOS, so the
fatal report's guest memory dumps dereferenced whatever the crashed
thread had in its registers. A fault inside the reporter re-enters the
signal handler and wedges the reporting thread, which hid real guest
crashes whenever logging was enabled: the game kept running with a dead
thread and the report was never completed.

Walk the Mach regions covering the range and require read permission
before dumping, the same contract the Linux implementation provides.

* do not fallthrough HOST_SYSTEM_MANAGED_MIN

---------

Co-authored-by: nmzik <Nmzik@mail.ru>

src/kernel/memory.cpp

@@ -818,6 +818,11 @@ static void                               MemoryPoolSubtractCommitted(uint64_t l
 // Keep host mappings, physical blocks, placeholders, and virtual ranges in step.
 static std::recursive_mutex g_memory_operation_mutex;
 
+// The base address the PS5 kernel hands out for hint-less user mappings. Guest code can
+// assume mappings it did not place explicitly are at or above this (Sony's libc rejects a
+// heap below it), so hint-less searches must not fall back to the low system-managed range.
+static constexpr uint64_t GUEST_DEFAULT_MAP_BASE = 0x200000000ull;
+
 static uint64_t FindGuestFreeRange(uint64_t search_addr, uint64_t size, uint64_t alignment) {
 	EXIT_IF(g_guest_address_space == nullptr || g_virtual_ranges == nullptr);
 
@@ -845,8 +850,11 @@ static uint64_t FindGuestFreeRange(uint64_t search_addr, uint64_t size, uint64_t
 	if (search_addr != 0) {
 		return find_in(search_addr, HOST_USER_MAX + 1u);
 	}
-	auto addr = find_in(HOST_SYSTEM_MANAGED_MIN, HOST_SYSTEM_MANAGED_MAX + 1u);
-	return addr != 0 ? addr : find_in(HOST_USER_MIN, HOST_USER_MAX + 1u);
+	auto addr = find_in(GUEST_DEFAULT_MAP_BASE, HOST_SYSTEM_MANAGED_MAX + 1u);
+	if (addr == 0) {
+		addr = find_in(HOST_USER_MIN, HOST_USER_MAX + 1u);
+	}
+	return addr;
 }
 
 bool TryWriteBacking(uint64_t vaddr, const void* data, uint64_t size) {

src/loader/runtimeLinker.cpp

@@ -38,7 +38,10 @@
 #include <windows.h>
 #else
 #include <dlfcn.h>
-#if KYTY_PLATFORM == KYTY_PLATFORM_LINUX && !defined(__APPLE__)
+#if defined(__APPLE__)
+#include <mach/mach.h>
+#include <mach/mach_vm.h>
+#elif KYTY_PLATFORM == KYTY_PLATFORM_LINUX
 #include <sys/uio.h>
 #include <unistd.h>
 #endif
@@ -722,7 +725,26 @@ static bool IsReadableRange(uint64_t addr, uint64_t size) {
 		}
 		current = std::min(region_end, end);
 	}
-#elif KYTY_PLATFORM == KYTY_PLATFORM_LINUX && !defined(__APPLE__)
+#elif defined(__APPLE__)
+	// Walk the Mach regions covering the range and require read permission. The fatal
+	// report dumps memory behind raw register values, and a fault inside the reporter
+	// re-enters the signal handler and wedges the reporting thread.
+	uint64_t current = addr;
+	while (current < end) {
+		mach_vm_address_t              region_addr = current;
+		mach_vm_size_t                 region_size = 0;
+		vm_region_basic_info_data_64_t info {};
+		mach_msg_type_number_t         count       = VM_REGION_BASIC_INFO_COUNT_64;
+		mach_port_t                    object_name = MACH_PORT_NULL;
+		if (mach_vm_region(mach_task_self(), &region_addr, &region_size, VM_REGION_BASIC_INFO_64,
+		                   reinterpret_cast<vm_region_info_t>(&info), &count,
+		                   &object_name) != KERN_SUCCESS ||
+		    region_addr > current || (info.protection & VM_PROT_READ) == 0) {
+			return false;
+		}
+		current = region_addr + region_size;
+	}
+#elif KYTY_PLATFORM == KYTY_PLATFORM_LINUX
 	const auto page_size = static_cast<uint64_t>(sysconf(_SC_PAGESIZE));
 	if (page_size == 0) {
 		return false;
@@ -752,7 +774,7 @@ static bool IsReadableRange(uint64_t addr, uint64_t size) {
 }
 
 static bool IsDumpableRange(uint64_t addr, uint64_t size) {
-#if KYTY_PLATFORM == KYTY_PLATFORM_LINUX && !defined(__APPLE__)
+#if KYTY_PLATFORM == KYTY_PLATFORM_LINUX
 	return IsReadableRange(addr, size);
 #else
 	(void)size;