[Bug]: MacOS Sonoma crashes on boot with more than 1 CPU

[jbramburek]

Operating system

Ubuntu 24.04

Description

Running docker image 1.10 (see #83) with 1 CPU I can get Sonoma installed and it seems to work. Changing to 4 or 2 CPUs however makes the masOS crash on boot. After changing back to 1 CPU the system starts again. The bug report macOS generates is attached.

Docker compose

services:
  macos:
    image: dockurr/macos
    container_name: macos
    environment:
      VERSION: "sonoma"
      RAM_SIZE: "10G"
      CPU_CORES: "4"
      DISK_SIZE: "256G"
    devices:
      - /dev/kvm
    cap_add:
      - NET_ADMIN
    ports:
      - 8006:8006
      - 5900:5900/tcp
      - 5900:5900/udp
    stop_grace_period: 2m
    volumes:
      - /storage/osx:/storage

Docker log

docker compose logs

macos  | ❯ Starting macOS for Docker v1.10...
macos  | ❯ For support visit https://github.com/dockur/macos
macos  | ❯ CPU: AMD Ryzen 5 4600G | RAM: 11/12 GB | DISK: 462 GB (ext4) | HOST: 6.8.0-45...
macos  |
macos  | ❯ Warning: unexpected clocksource: kvm-clock
macos  | ❯ Warning: your configured RAM_SIZE of 10 GB is very close to the 11 GB of memory available, please consider a lower value.
macos  | ❯ Booting macOS 1920x1080 using QEMU v9.0.2...
macos  | qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
macos  | qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.erms [bit 9]
macos  | qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
macos  | qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
macos  |
BdsDxe: failed to load Boot0080 "Mac OS X" from PciRoot(0x0)/Pci(0xA,0x0)/VenMedia(BE74FCF7-0B7C-49F3-9147-01F4042E6842,F407ECD909D6DF4288953882CAB087A7)/\B9ACD089-962F-4937-9CD0-F429E3B3CD41\System\Library\CoreServices\boot.efi: Not Found
macos  | BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0xA,0x0): Not Found
macos  | BdsDxe: loading Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x5,0x0)
macos  | BdsDxe: starting Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x5,0x0)
macos  | #[EB|LOG:EXITBS:END] _
macos  | #[EB.BST.FBS|-]
macos  | #[EB|B:BOOT]
macos  | #[EB|LOG:HANDOFF TO XNU] _
macos  | ======== End of efiboot serial output. ========

MasOS bug report

panic(cpu 1 caller 0xffffff8001e54b4e): Non-monotonic time: invoke at 0x6ad95c891, runnable at 0x6ad95d658 @sched_prim.c:3031
Panicked task 0xffffff91fb327bd8: 5 threads: pid 147: cfprefsd
Backtrace (CPU 1), panicked thread: 0xffffffa06085c598, Frame : Return Address
0xffffff8a537f7950 : 0xffffff8001e265a1 mach_kernel : _handle_debugger_trap + 0x4b1
0xffffff8a537f79a0 : 0xffffff8001f8bcd0 mach_kernel : _kdp_i386_trap + 0x110
0xffffff8a537f79e0 : 0xffffff8001f7b28c mach_kernel : _kernel_trap + 0x55c
0xffffff8a537f7a60 : 0xffffff8001dc0971 mach_kernel : _return_from_trap + 0xc1
0xffffff8a537f7a80 : 0xffffff8001e2688d mach_kernel : _DebuggerTrapWithState + 0x5d
0xffffff8a537f7b70 : 0xffffff8001e25f33 mach_kernel : _panic_trap_to_debugger + 0x1e3
0xffffff8a537f7bd0 : 0xffffff80025db78b mach_kernel : _panic + 0x84
0xffffff8a537f7cc0 : 0xffffff8001e54b4e mach_kernel : _thread_block_reason + 0x21ce
0xffffff8a537f7d50 : 0xffffff8001e52a70 mach_kernel : _thread_block_reason + 0xf0
0xffffff8a537f7da0 : 0xffffff80022cbedf mach_kernel : _workq_thread_set_max_qos + 0x31bf
0xffffff8a537f7e10 : 0xffffff80022ca7b1 mach_kernel : _workq_thread_set_max_qos + 0x1a91
0xffffff8a537f7ec0 : 0xffffff80022cf1b9 mach_kernel : _workq_kernreturn + 0x489
0xffffff8a537f7f40 : 0xffffff8002455170 mach_kernel : _unix_syscall64 + 0x1f0
0xffffff8a537f7fa0 : 0xffffff8001dc0db6 mach_kernel : _hndl_unix_scall64 + 0x16

Process name corresponding to current thread (0xffffffa06085c598): cfprefsd
Boot args: keepsyms=1 

Mac OS version:
23H124

Kernel version:
Darwin Kernel Version 23.6.0: Wed Jul 31 20:48:44 PDT 2024; root:xnu-10063.141.1.700.5~1/RELEASE_X86_64
Kernel UUID: 67760982-25DC-34FC-A09E-E5624B8B4D62
roots installed: 0
KernelCache slide: 0x0000000001a00000
KernelCache base:  0xffffff8001c00000
Kernel slide:      0x0000000001ae4000
Kernel text base:  0xffffff8001ce4000
__HIB  text base: 0xffffff8001b00000
System model name: iMacPro1,1 (Mac-7BA5B2D9E42DDD94)
System shutdown begun: NO
Panic diags file available: NO (0xe00002bc)
Hibernation exit count: 0

System uptime in nanoseconds: 28682746701
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x00000006ad9ffe96
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000000c097e2e4d 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff8b93c26000 - 0xffffffab93c26000
  . PGZ   : 0xffffff8b93c26000 - 0xffffff8b95027000
  . VM    : 0xffffff8b95027000 - 0xffffff90619f3000
  . RO    : 0xffffff90619f3000 - 0xffffff91fb28c000
  . GEN0  : 0xffffff91fb28c000 - 0xffffff96c7c58000
  . GEN1  : 0xffffff96c7c58000 - 0xffffff9b94625000
  . GEN2  : 0xffffff9b94625000 - 0xffffffa060ff2000
  . GEN3  : 0xffffffa060ff2000 - 0xffffffa52d9bf000
  . DATA  : 0xffffffa52d9bf000 - 0xffffffab93c26000
  Metadata: 0xffffff8a31616000 - 0xffffff8a51616000
  Bitmaps : 0xffffff8a51616000 - 0xffffff8a53416000
  Extra   : 0 - 0

last started kext at 17045942211: @filesystems.autofs	3.0 (addr 0xffffff7f973ff000, size 32768)
loaded kexts:
com.khronokernel.CryptexFixup	1.0.1
as.acidanthera.BlueToolFixup	2.6.9
as.acidanthera.BrcmFirmwareStore	2.6.9
as.vit9696.!AALC	1.9.0
as.vit9696.WhateverGreen	1.6.7
as.vit9696.Lilu	1.6.8
@filesystems.autofs	3.0
>AGPM	135
>X86PlatformShim	1.0.0
>!APlatformEnabler	2.7.0d0
>!AGraphicsDevicePolicy	8.1.9
>!AUpstreamUserClient	3.6.11
>!AMCCSControl	1.17
@AGDCPluginDisplayMetrics	8.1.9
>pmtelemetry	1
@filesystems.nfs	1
|IOUserEthernet	1.0.1
>usb.!UUserHCI	1
>!AUIO	1
>!ASunrise	1
>!AHV	1
>!A!ISlowAdaptiveClocking	4.0.0
>!A!IMCEReporter	115
>!A16X50ACPI	3.2
>ACPI_SMC_PlatformPlugin	1.0.0
>!AFIVRDriver	4.1.0
>!AFileSystemDriver	3.0.1
@filesystems.tmpfs	1
@filesystems.lifs	1
@filesystems.apfs	2236.141.1
@filesystems.hfs.kext	650.140.2
@BootCache	40
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
>!AAHCIPort	380
>!AACPIButtons	6.1
@private.KextAudit	1.0
>!ARTC	2.0.1
>!ASMBIOS	2.1
>!AAPIC	1.7
@!ASystemPolicy	2.0.0
@nke.applicationfirewall	405
|IOKitRegistryCompatibility	1
|EndpointSecurity	1
@Dont_Steal_Mac_OS_X	7.0.0
>!ADiskImages2	276.120.7
@kec.!AEncryptedArchive	1
>!AGraphicsControl	8.1.9
>!ASMBus!C	1.0.18d1
|IOSMBus!F	1.1
>!ASMBusPCI	1.0.14d1
>!UAudio	640.12
|IOAudio!F	540.3
@vecLib.kext	1.2.0
>!AAudioClockLibs	340.8
@kext.triggers	1.0
>IOHIDPowerSource	1
>!ACallbackPowerSource	1
|IOAVB!F	1220.1
>!ARSMChannel	1
|IORSM!F	1
>!AIPAppender	1.0
@!AGPUWrangler	8.1.9
@!AGraphicsDeviceControl	8.1.9
|IOSlowAdaptiveClocking!F	1.0.0
@plugin.IOgPTPPlugin	1240.15
>!A16X50Serial	3.2
>IOPlatformPluginLegacy	1.0.0
>X86PlatformPlugin	1.0.0
>IOPlatformPlugin!F	6.0.0d8
>driverkit.serial	6.0.0
|IONDRVSupport	598
|IOGraphics!F	598
>usb.IOUSBHostHIDDevice	1.2
>usb.cdc	5.0.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
>!AXsanScheme	3
>!ABSDKextStarter	3
|IOSurface	352.50.1
@filesystems.hfs.encodings.kext	1
>!ASyntheticGame!C	11.6.1
>usb.!UHostPacketFilter	1.0
|IOUSB!F	900.4.2
|IOAHCI!F	304
>!AVirtIO	221.100.2
|IOSerial!F	11
>usb.!UXHCIPCI	1.2
>usb.!UXHCI	1.2
|IOHID!F	2.0.0
>!AEFINVRAM	2.1
>!AEFIRuntime	2.1
|IOTimeSync!F	1240.15
|IOSkywalk!F	1.0
>mDNSOffloadUserClient	1.0.1b8
|IONetworking!F	3.4
>DiskImages	493.0.0
|IO!B!F	9.0.0
|IOReport!F	47
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
>!ASSE	1.0
>!ALockdownMode	1
>!AKeyStore	2
>!UTDM	556
|IOUSBMass!SDriver	245
|IOUSBHost!F	1.2
>!UHostMergeProperties	1.2
>usb.!UCommon	1.0
>!ABusPower!C	1.0
|IOSCSIBlockCommandsDevice	495
|IOSCSIArchitectureModel!F	495
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
|IO!S!F	2.1
>!ACyrus	1
>!AMobileFileIntegrity	1.0.5
$!AImage4	6.3.0
@kext.CoreTrust	1
>!ACredentialManager	1.0
|CoreAnalytics!F	1
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
>!AACPIPlatform	6.1
>!ASMC	3.1.9
|IOPCI!F	2.9
|IOACPI!F	1.4
>watchdog	1
@kec.pthread	1
@kec.XrtHostedXnu	1
@kec.Libm	1
@kec.Compression	1.0
@kec.corecrypto	14.0

Screenshots (optional)

No response

[kroese]

I have no clue why this happens. Only solution I can offer right now is to use an older version, like Ventura.

[TridentInsurgence]

I am also having this issue.

[urfavteddybear]

I have no clue why this happens. Only solution I can offer right now is to use an older version, like Ventura.

Also happened on ventura

[kroese]

Its not so useful to just hear "happens to me too", it would be better to include docker logfile like @jbramburek did.

That way we can try to track down under which circumstances this happens. For example, do you all have an "AMD Ryzen 5 4600G" processor like @jbramburek or do you all have a weird clocksource (@jbramburek has kvm-clock which is very strange, like he uses nested virtualization?), etc. Only if we discover what you guys all share in common, we can figure out what is causing this.

[urfavteddybear]

1. Docker logs almost look the same
2. I am using nested virtualization with AMD EPYC 7B13 and AMD Ryzen 9 5950X. Both have the same issues and identical logs.

[kroese]

@urfavteddybear So you get the same message:

 unexpected clocksource: kvm-clock

?

[urfavteddybear]

Exactly. It happens on Sequoia, Sonoma, Ventura, and Monterey. I haven't tried Big Sur yet, though it seems fine when running on 1 CPU.

[kroese]

I created a new version now (v1.14) that will automaticly detect nested virtualization and restricts the container to a single core when the default of 2 was selected. To prevent this problem for new users.

But ofcourse this doesnt solve the root problem, that macOS crashes when using nested virtualization on AMD in combination with multiple cores.

However Im afraid I will not be able to solve it. Either it is a problem in macOS, or in OpenCore or in QEMU/KVM.

Is there any good reason why you run this container with nested virtualization? I never expected anyone to do so. Because if you already have a way to run VM's there seems no need for a container that can run a VM?

[jbramburek]

I am running this in a VM on my Proxmox server. I haven't really explored other options to get this os so far.

[kroese]

@jbramburek I would advice to install macOS directly in Proxmox then, instead of using this container. This project is more for people that do not have Proxmox but just Docker.

[urfavteddybear]

Somehow I can make it work on AMD EPYC 7443P but not with Ryzen series.