Don't Panic! Upgrading FreeBSD to 11.2 on GanetiYour Bug Report Will Be Graded

Mon 12 November 2018 | -- (permalink)

Hit a wall while upgrading a pair of FreeBSD 10.4 VMs to 11.2. All went well until upgrade tried to boot from the new kernel, at which point I hit:

FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): text 80x25
CPU: QEMU Virtual CPU version 2.1.2 (2666.81-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x663  Family=0x6  Model=0x6  Stepping=3
  Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 4294967296 (4096 MB)
avail memory = 4088406016 (3899 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <BOCHS  BXPCAPIC>
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kernel trap 9 with interrupts disabled


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff810e9ae6
stack pointer       = 0x28:0xffffffff82272c20
frame pointer       = 0x28:0xffffffff82272c80
code segment        = base 0x0, limit 0xfffff, type 0x1b
                    = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = resume, IOPL = 0
current process     = 0 (swapper)
trap number         = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80b3d587 at kdb_backtrace+0x67
#1 0xffffffff80af6b27 at vpanic+0x177
#2 0xffffffff80af69a3 at panic+0x43
#3 0xffffffff80f77fdf at trap_fatal+0x35f
#4 0xffffffff80f7759e at trap+0x5e
#5 0xffffffff80f57fbc at calltrap+0x8
#6 0xffffffff810ec5f3 at apic_setup_io+0x53
#7 0xffffffff80a92898 at mi_startup+0x118
#8 0xffffffff8031002c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort

These are pretty boring VMs: GENERIC kernel, UFS2 filesystems, one CPU per VM, no weird kernel modules. The only things slightly odd about them were that they're running on Ganeti, and that, as a direct consequence, they use virtio disk and network devices.

FreeBSD mailing lists to the rescue. A kind soul suggested setting hw.x2apic_enable=0 before booting. Which just worked. Thanks, Konstantin!

Recipes for the cut-and-paste oriented:

From the loader prompt if you just want to test this:

set hw.x2apic_enable=0
boot

In /boot/loader.conf to make this permanent:

hw.x2apic_enable=0