NetBSD Not Quite Installed On Hetzner EX101
NetBSD has a serial, non-video installer which seems to work with qemu inside tmux inside Hetzner's rescue system:
root@rescue ~ # cat start-install
qemu-system-x86_64 \
-enable-kvm \
-nographic \
-cpu host \
-boot once=d \
-cdrom ./boot-com.iso \
-m 4096 \
-drive file=/dev/nvme0n1,format=raw,media=disk,if=virtio \
-drive file=/dev/nvme1n1,format=raw,media=disk,if=virtio
root@rescue ~ #
One downloads from the above linked page the SHA512 file and the boot-com.iso file, checks the SHA512, and runs something like the above qemu-command. A prerequisite might be having used Hetzner installimage to put a non-RAID install on the server so that the rescue system's qemu will see /dev/nvme* devices instead of /dev/md*.
After running through the install sequence and rebooting inside qemu inside tmux, the system seemed to come up from the NVMe drive.
After shutdown -h now inside NetBSD and then Ctrl-b, x, y to get out of tmux and then rebooting to get out of the Hetzner Rescue System, the server seemed to come up! It responded to IPv4 ping, but ssh [email protected] gave me ssh: connect to host fsn.metalvps.com port 22: Network is unreachable.
Re-enabling the Hetzner rescue system and restarting qemu, NetBSD booted again from the NVMe disk (same qemu command as above, but with the following two lines removed).
-boot once=d \
-cdrom ./boot-com.iso \
Eventually I may figure out why the live, booted-from-metal server could respond to ping but ssh wouldn't work. It's been more than 20 years since I last used NetBSD, so I've forgotten everything. 
Why didn't I use Hetzner's free KVM console option? Well, if qemu works, I don't have to bother Hetzner's support guys to have them attach and then remove the KVM.
Thanks to the depenguin.me guys for helpful hints.
Here's ps aux output from the rebooted server running inside qemu. Notice that there are fewer processes than contemporary Linux.
fsn# date -u; ps aux
Tue May 16 23:57:19 UTC 2023
USER PID %CPU %MEM VSZ RSS TTY STAT STARTED TIME COMMAND
root 0 0.0 0.8 0 33688 ? DKl 11:30PM 0:00.32 [system]
root 1 0.0 0.0 11944 1660 ? Is 11:30PM 0:00.00 init
postfix 328 0.0 0.1 21456 4900 ? I 11:30PM 0:00.01 pickup -l -t unix
postfix 329 0.0 0.1 21504 4968 ? I 11:30PM 0:00.00 qmgr -l -t unix -
root 331 0.0 0.0 13360 1488 ? Is 11:30PM 0:00.00 /usr/sbin/inetd -
root 339 0.0 0.0 12212 1740 ? Is 11:30PM 0:00.00 /usr/sbin/cron
root 351 0.0 0.1 18408 2676 ? S 11:30PM 0:00.03 ntpd: asynchronou
root 829 0.0 0.1 18036 2476 ? Ss 11:30PM 0:00.03 /usr/sbin/syslogd
root 1313 0.0 0.0 11768 1416 ? Is 11:30PM 0:00.00 /usr/sbin/powerd
root 1442 0.0 0.4 18272 17824 ? Ss 11:30PM 0:00.14 /usr/sbin/ntpd -p
root 1532 0.0 0.1 23044 3200 ? Is 11:30PM 0:00.00 sshd: /usr/sbin/s
root 1821 0.0 0.1 21528 2876 ? Is 11:30PM 0:00.01 /usr/libexec/post
root 306 0.0 0.0 12076 1720 tty00 O+ 11:57PM 0:00.00 ps -aux
root 335 0.0 0.1 12584 2228 tty00 S 11:31PM 0:00.07 -sh
root 347 0.0 0.3 24872 10492 tty00 Is 11:30PM 0:00.09 login
root 348 0.0 0.0 11860 1684 ttyE1 Is+ 11:30PM 0:00.00 /usr/libexec/gett
root 2013 0.0 0.0 11856 1684 ttyE2 Is+ 11:30PM 0:00.00 /usr/libexec/gett
root 1118 0.0 0.0 11856 1680 ttyE3 Is+ 11:30PM 0:00.00 /usr/libexec/gett
fsn#
Here's dmesg output from the rebooted server running inside qemu.
fsn# dmesg | cat -n
1 [ 1.000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
2 [ 1.000000] 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
3 [ 1.000000] 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023
4 [ 1.000000] The NetBSD Foundation, Inc. All rights reserved.
5 [ 1.000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
6 [ 1.000000] The Regents of the University of California. All rights reserved.
7
8 [ 1.000000] NetBSD 10.99.4 (GENERIC) #0: Sat May 13 11:19:19 UTC 2023
9 [ 1.000000] [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
10 [ 1.000000] total memory = 4095 MB
11 [ 1.000000] avail memory = 3935 MB
12 [ 1.000000] timecounter: Timecounters tick every 10.000 msec
13 [ 1.000000] Kernelized RAIDframe activated
14 [ 1.000000] timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
15 [ 1.000004] mainbus0 (root)
16 [ 1.000004] ACPI: RSDP 0x00000000000F5AF0 000014 (v00 BOCHS )
17 [ 1.000004] ACPI: RSDT 0x00000000BFFE1550 000034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
18 [ 1.000004] ACPI: FACP 0x00000000BFFE1404 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
19 [ 1.000004] ACPI: DSDT 0x00000000BFFE0040 0013C4 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
20 [ 1.000004] ACPI: FACS 0x00000000BFFE0000 000040
21 [ 1.000004] ACPI: APIC 0x00000000BFFE1478 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
22 [ 1.000004] ACPI: HPET 0x00000000BFFE14F0 000038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)
23 [ 1.000004] ACPI: WAET 0x00000000BFFE1528 000028 (v01 BOCHS BXPCWAET 00000001 BXPC 00000001)
24 [ 1.000004] ACPI: 1 ACPI AML tables successfully acquired and loaded
25 [ 1.000004] ioapic0 at mainbus0 apid 0: pa 0xfec00000, version 0x11, 24 pins
26 [ 1.000004] cpu0 at mainbus0 apid 0
27 [ 1.000004] cpu0: Use lfence to serialize rdtsc
28 [ 1.000004] cpu0: 13th Gen Intel(R) Core(TM) i9-13900, id 0xb0671
29 [ 1.000004] cpu0: node 0, package 0, core 0, smt 0
30 [ 1.000004] acpi0 at mainbus0: Intel ACPICA 20221020
31 [ 1.000004] acpi0: X/RSDT: OemId <BOCHS ,BXPCRSDT,00000001>, AslId <BXPC,00000001>
31 [ 1.000004] acpi0: X/RSDT: OemId <BOCHS ,BXPCRSDT,00000001>, AslId <BXPC,00000001> [76/526]
32 [ 1.000004] LNKS: ACPI: Found matching pin for 0.1.INTA at func 3: 9
33 [ 1.000004] LNKC: ACPI: Found matching pin for 0.3.INTA at func 0: 11
34 [ 1.000004] LNKD: ACPI: Found matching pin for 0.4.INTA at func 0: 11
35 [ 1.000004] LNKA: ACPI: Found matching pin for 0.5.INTA at func 0: 10
36 [ 1.000004] acpi0: SCI interrupting at int 9
37 [ 1.000004] acpi0: fixed power button present
38 [ 1.000004] timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
39 [ 1.046041] hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
40 [ 1.046041] timecounter: Timecounter "hpet0" frequency 100000000 Hz quality 2000
41 [ 1.046619] pckbc1 at acpi0 (KBD, PNP0303) (kbd port): io 0x60,0x64 irq 1
42 [ 1.046619] pckbc2 at acpi0 (MOU, PNP0F13) (aux port): irq 12
43 [ 1.046619] fdc0 at acpi0 (FDC0, PNP0700): io 0x3f2-0x3f5,0x3f7 irq 6 drq 2
44 [ 1.046619] lpt0 at acpi0 (LPT1, PNP0400-1): io 0x378-0x37f irq 7
45 [ 1.046619] com0 at acpi0 (COM1, PNP0501-1): io 0x3f8-0x3ff irq 4
46 [ 1.046619] com0: ns16550a, 16-byte FIFO
47 [ 1.046619] com0: console
48 [ 1.046619] qemufwcfg0 at acpi0 (FWCF, QEMU0002): io 0x510-0x51b
49 [ 1.046619] qemufwcfg0: <QEMU>
50 [ 1.046619] ACPI: Enabled 2 GPEs in block 00 to 0F
51 [ 1.046619] pckbd0 at pckbc1 (kbd slot)
52 [ 1.046619] pckbc1: using irq 1 for kbd slot
53 [ 1.046619] wskbd0 at pckbd0 mux 1
54 [ 1.046619] pms0 at pckbc1 (aux slot)
55 [ 1.046619] pckbc1: using irq 12 for aux slot
56 [ 1.046619] wsmouse0 at pms0 mux 0
57 [ 1.046619] pci0 at mainbus0 bus 0: configuration mode 1
58 [ 1.046619] pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
59 [ 1.046619] pchb0 at pci0 dev 0 function 0: Intel 82441FX (PMC) PCI and Memory Controller (rev. 0x02)
60 [ 1.046619] pcib0 at pci0 dev 1 function 0: Intel 82371SB (PIIX3) PCI-ISA Bridge (rev. 0x00)
61 [ 1.046619] piixide0 at pci0 dev 1 function 1: Intel 82371SB IDE Interface (PIIX3) (rev. 0x00)
62 [ 1.046619] piixide0: bus-master DMA support present
63 [ 1.046619] piixide0: primary channel wired to compatibility mode
64 [ 1.046619] piixide0: primary channel interrupting at ioapic0 pin 14
65 [ 1.046619] atabus0 at piixide0 channel 0
66 [ 1.046619] piixide0: secondary channel wired to compatibility mode
67 [ 1.046619] piixide0: secondary channel interrupting at ioapic0 pin 15
68 [ 1.046619] atabus1 at piixide0 channel 1
69 [ 1.046619] piixpm0 at pci0 dev 1 function 3: Intel 82371AB (PIIX4) Power Management Controller (rev. 0x03) [38/526]
70 [ 1.046619] timecounter: Timecounter "piixpm0" frequency 3579545 Hz quality 1000
71 [ 1.046619] piixpm0: 24-bit timer
72 [ 1.046619] piixpm0: interrupting at ioapic0 pin 9
73 [ 1.046619] iic0 at piixpm0 port 0: I2C bus
74 [ 1.046619] vga0 at pci0 dev 2 function 0: vendor 1234 product 1111 (rev. 0x02)
75 [ 1.046619] wsdisplay0 at vga0 kbdmux 1
76 [ 1.046619] wsmux1: connecting to wsdisplay0
77 [ 1.046619] wskbd0: connecting to wsdisplay0
78 [ 1.046619] drm at vga0 not configured
79 [ 1.046619] wm0 at pci0 dev 3 function 0, 64-bit DMA: Intel i82540EM 1000BASE-T Ethernet (rev. 0x03)
80 [ 1.046619] wm0: interrupting at ioapic0 pin 11
81 [ 1.046619] wm0: 32-bit 33MHz PCI bus
82 [ 1.046619] wm0: 64 words (6 address bits) MicroWire EEPROM
83 [ 1.046619] wm0: Ethernet address 52:54:00:12:34:56
84 [ 1.046619] wm0: 0x200002<LOCK_EECD,WOL>
85 [ 1.046619] makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 0
86 [ 1.046619] makphy0: Failed to access EADR. Are you an emulator?
87 [ 1.046619] makphy0: Failed to read EXTSR. Are you an emulator?. Regard as 1000BASE-T.
88 [ 1.046619] makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
89 [ 1.046619] virtio0 at pci0 dev 4 function 0
90 [ 1.046619] virtio0: block device (id 2, rev. 0x00)
91 [ 1.046619] ld0 at virtio0: features: 0x10000a54<INDIRECT_DESC,CONFIG_WCE,FLUSH,BLK_SIZE,GEOMETRY,SEG_MAX>
92 [ 1.046619] virtio0: allocated 1060864 byte for virtqueue 0 for I/O request, size 256
93 [ 1.046619] virtio0: using 1048576 byte (65536 entries) indirect descriptors
94 [ 1.046619] virtio0: config interrupting at msix0 vec 0
95 [ 1.046619] virtio0: queues interrupting at msix0 vec 1
96 [ 1.046619] ld0: 1788 GB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 3750748848 sectors
97 [ 1.046619] virtio1 at pci0 dev 5 function 0
98 [ 1.046619] virtio1: block device (id 2, rev. 0x00)
99 [ 1.046619] ld1 at virtio1: features: 0x10000a54<INDIRECT_DESC,CONFIG_WCE,FLUSH,BLK_SIZE,GEOMETRY,SEG_MAX>
100 [ 1.046619] virtio1: allocated 1060864 byte for virtqueue 0 for I/O request, size 256
101 [ 1.046619] virtio1: using 1048576 byte (65536 entries) indirect descriptors
102 [ 1.046619] virtio1: config interrupting at msix1 vec 0
103 [ 1.046619] virtio1: queues interrupting at msix1 vec 1
104 [ 1.046619] ld1: 1788 GB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 3750748848 sectors
105 [ 1.046619] isa0 at pcib0
106 [ 1.046619] attimer0 at isa0 port 0x40-0x43
107 [ 1.046619] pcppi0 at isa0 port 0x61
108 [ 1.046619] spkr0 at pcppi0: PC Speaker
109 [ 1.046619] wsbell at spkr0 not configured
110 [ 1.046619] midi0 at pcppi0: PC speaker
111 [ 1.046619] sysbeep0 at pcppi0
112 [ 1.046619] attimer0: attached to pcppi0
113 [ 1.046619] acpicpu0 at cpu0: ACPI CPU
114 [ 1.046619] acpicpu0: C1: HLT, lat 0 us, pow 0 mW
115 [ 1.046619] vmt0 at cpu0
116 [ 1.046619] vmt0: UUID: 00000000-0000-0000-0000-000000000000
117 [ 1.046619] vmware: open failed, eax=0xffffffff, ecx=0x1e, edx=0x5658
118 [ 1.046619] vmt0: autoconfiguration error: failed to open backdoor RPC channel (TCLO protocol)
119 [ 1.046619] timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
120 [ 1.046619] fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
121 [ 1.046619] ld1: GPT GUID: 7ac59b06-d775-499c-b672-f71133e62986
122 [ 1.046619] ld0: GPT GUID: 868d363e-b930-4142-944f-e70905997bdb
123 [ 1.046619] IPsec: Initialized Security Association Processing.
124 [ 1.046619] dk0 at ld1: "be45d1a2-c15f-4fa9-a36b-db22ab2e15e0", 524288 blocks at 4096, type: msdos
125 [ 1.046619] dk1 at ld1: "c6fc5a64-7669-464c-8578-77c52a1ebe6c", 1048576 blocks at 528384, type: <unknown>
126 [ 1.046619] dk2 at ld1: "24fde94d-ca86-46bd-8358-93c1ac2f8b9e", 3749171855 blocks at 1576960, type: <unknown>
127 [ 1.046619] dk3 at ld0: "02a153e5-14e1-45f0-8e65-d55f5f7da62c", 3742359552 blocks at 2048, type: ffs
128 [ 1.046619] dk4 at ld0: "981d2b55-1018-4fc9-9f3c-5847d6751eb7", 8387215 blocks at 3742361600, type: swap
129 [ 5.005793] atapibus0 at atabus1: 2 targets
130 [ 5.015571] cd0 at atapibus0 drive 0: <QEMU DVD-ROM, QM00003, 2.5+> cdrom removable
131 [ 5.015571] cd0: 32-bit data port
132 [ 5.015571] cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
133 [ 5.015571] cd0(piixide0:1:0): using PIO mode 4, DMA mode 2 (using DMA)
134 [ 5.015571] swwdog0: software watchdog initialized
135 [ 5.025557] WARNING: 1 error while detecting hardware; check system log.
136 [ 5.025557] boot device: ld0
137 [ 5.025557] root on dk3 dumps on dk4
138 [ 5.025557] root file system type: ffs
139 [ 5.025557] kern.module.path=/stand/amd64/10.99.4/modules
140 [ 10.485560] wsdisplay0: screen 1 added (80x25, vt100 emulation)
141 [ 10.485560] wsdisplay0: screen 2 added (80x25, vt100 emulation)
142 [ 10.485560] wsdisplay0: screen 3 added (80x25, vt100 emulation)
143 [ 10.485560] wsdisplay0: screen 4 added (80x25, vt100 emulation)
fsn#
Tagged:
Comments
Just curious, what is the use case for netbsd these days?
The all seeing eye sees everything...
pfsense is based on NetBSD so a lot of people come across it through that.
zfs is probably the main reason I can think of to actively chose it over Linux.
I think nowadays NetBSD can also run Linux binaries, so it might work out pretty well. For me, the big turn off was not having good wireguard support - I know it's possible to run usermode, but for something like that I'd want it in the kernel.
NetBSD 10 has in-kernel wireguard support.
That sounds weird...
I wonder if the real hardware just uses a different device name for the network interface. How do the boot messages from the real hardware boot look like (hopefully those would show up in
/var/log/messages)?A bit of an update. . . . Been through the install a couple more times. The installer seems to work fine inside qemu. During the installation (and also by rebooting still inside qemu from the NVMe drive after the installation has finished), I have been trying to adjust the network configuration so that IPv4 and IPv6 work when the installed system is booted raw, "in real life."
I woke up this morning to find emails from Hetzner saying there was a router issue affecting the server.
I did look all through /var/log/messages. There were multiple boots recorded. Both the boots inside qemu and also the direct from hardware boots seemed included. I didn't notice anything special. It would have been easy for me to miss something.
As of right now, I could retry NetBSD, reinstall Debian, or play with something else. It's been so, so much fun to play with NetBSD again! I've been skimming manpages, the NetBSD wiki, etc. I see that, with stuff like Wireguard in the kernel and fast + slow processor support, NetBSD still is as cool as ever!
MetalVPS
I guess the thing to look out for is where it detects the network interface
If the direct hardware boot shows a different interface name, you'll likely have to create a corresponding
/etc/ifconfig.if0configuration file for that interface (changeif0to the interface name).Been down the fun rabbit hole a few more times, with Alpine and FreeBSD in addition to NetBSD. I've also enjoyed equally grand unsuccess with depenguin.me.
On the initial group of NetBSD installs I thought I did see previous boot messages from the direct hardware boots showing up in NetBSD's /var/log/messages when NetBSD subsequently was rebooted into qemu running inside the rescue system. Unfortunately, I didn't save these logs.
On the later group of NetBSD installs and on the Alpine installs, no direct from hardware boot messages seem to appear on subsequent reboots back into qemu.
Following the early group of qemu installs, I did believe that the server initially seemed to respond to ping when booted directly. Later, maybe checking more carefully, there seemed to be no response to ping and no ssh on either IPv4 or IPv6 after hardware boot.
I imagined that a problem could be that the installs done from qemu somehow were remembering the qemu MAC address. If so, I imagined that Hetzner's MAC address filtering might have blocked network connectivity? I tried a few installs with the MAC address manually configured to match the actual hardware address. I also tried manually configuring the IPv4 and IPv6 networking. Lots of joy, but still no workee.
Besides the possible change of ethernet interface name, and the somehow perhaps possible persistence of the qemu MAC or IP addresses into the hardware install, I have a new idea about what the problem might be. Looking at the qemu command shown in the OP, I'm wondering if there is a boot environment mismatch between my simple qemu command and the real boot firmware/hardware. Maybe I need to add something to the qemu command to make the qemu based install emulate the real hardware?
Of course I probably could get the installs to work by using Hetzner's free KVM Console instead of qemu.
Happily, the one thing that does seem to work is rebooting the installed NVMe disk back into qemu even after having tried a full hardware boot. It does seem like the installs indeed might be being written to the NVMe hard disks and that hand made changes persist.
I guess I'm gonna try looking at the early stage booting some more. Unless somebody has a better idea? As always, any hints would be much appreciated! Thanks so much!
MetalVPS
Okay, so it looks like I have to install the Debian ovmf package along with inside the rescue system and boot the installer inside qemu with Tianocore. That might make the installers select and install for UEFI instead of BIOS? See https://github.com/tianocore/tianocore.github.io/wiki/OVMF-FAQ#what-is-open-virtual-machine-firmware-ovmf See also How To Boot UEFI On QEMU
Guessing that the nice, new hardware on the i9-13900 might not be happy with BIOS. Does that sound right? Maybe I will try the ovmf and see if the installs work better. . . . 🤔
MetalVPS
From
dmidecodeoutput:MetalVPS
Honestly for the CPU it doesn't really matter if you install EFI or BIOS, as it is handled by motherboard, not CPU. If your CPU supports UEFI, it will support BIOS and will play nicely with BIOS as well.
Unless you need secure boot or other features supported by EFI boot, BIOS will work just fine, specially for a server, which does not reboot often, and thus mostly runs the OS. Once the OS takes over from the boot manager, UEFI or BIOS makes almost no differences.
For network boots, EFI is generally not supported and only BIOS works.
You can read this for more info on differences between UEFI and BIOS managers: https://www.freecodecamp.org/news/uefi-vs-bios/
Final note from intel website. Link: https://www.intel.com/content/www/us/en/products/docs/processors/embedded/13th-gen-core-product-brief.html
Text 1:
Text 2:
So your CPU clearly supports and plays nicely with legacy bios boot managers.
Somik.org - Server admins cheat codes
Hi @somik! Thanks for your helpful comments!
Every morning brings an opportunity for me to learn a little more! Best wishes! Tom
MetalVPS