I recently obtained a KVM virtual server on SolusIO platform, and I want to install Ubuntu 20.04 Server from the official ISO image.
This is not as easy as I hoped, but I figured it out.

Note: if you are in a hurry, skip the "Background" and start from "Part 1" section.

Background: SolusIO cannot Mount ISO Image

SolusIO is a virtual infrastructure management solution published by Plesk International Gmbh, the same company behind the popular SolusVM software.
They describe SolusIO to be the successor of SolusVM, with more focus on the self-service approach for end users.

SolusIO inherits the same clean user interface from SolusVM, and is easy to use.
However, as a power user, I notice several features are missing in SolusIO.
One of these features is the ability to install the operating system from an ISO image.

Presently, SolusIO only supports installing operating systems from the provided templates.
While a template installation is convenient, I prefer to install from official ISO images so that I can have better control over what goes into my system, and avoid vulnerabilities such as the debianuser backdoor.

An easy way to load a ISO installer is through netboot.xyz, a hosted service that can boot a server over the Internet and download the installer of many Linux and BSD distributions.
Even if the VPS hosting platform does not support mounting ISO images, it is possible to install iPXE bootloader per Tip: Install OS via netboot.xyz without ISO support, and then start netboot.xyz from there.

However, this method did not work for me.
The iPXE package, as distributed in most operating systems, only supports IPv4.
To make it work on my IPv6-only server, I would have to recompile iPXE with IPv6 enabled, which could be a complicated task.
I need another way.

Part 1: Boot from ISO on Hard Disk

I thought: can I download the ISO image and load it from hard disk, and start the installer, just like netboot.xyz does, but without going through the network?
I searched for "how to boot ISO from hard disk", and found a helpful guide from Ubuntu documentation site: Grub2/ISOBoot.
Piecing together the information, I found the exact steps to start the Ubuntu installer.

1. Install Debian 10 from SolusIO template.

   As much as I love Ubuntu, it does not work for some reason.

2. SSH into the server as root, and download official Ubuntu ISO image.

   wget https://releases.ubuntu.com/focal/ubuntu-20.04.2-live-server-amd64.iso
   

3. Modify /etc/grub.d/40_custom file, and append the following text:

   menuentry "Ubuntu 20.04 ISO" {
    set isofile="/root/ubuntu-20.04.2-live-server-amd64.iso"
    rmmod tpm
    loopback loop (hd0,1)$isofile
    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile noprompt noeject toram
    initrd (loop)/casper/initrd
   }
   

   In this snippet:

• The path after isofile= is where we downloaded the ISO image.
• (hd0,1) is /dev/sda1 written in a syntax recognized by the GRUB bootloader.
  If the ISO image file is on a different disk partition, you may need different numbers.
  Search for "drive number" on Grub2/ISOBoot page for more information.
• The toram option copies the ISO image from the hard drive to the memory, which is necessary because the installer needs to reformat and overwrite the hard drive.
  This implies that this method would only work on a machine with large enough memory to fit the entire ISO image, and would not work on 1GB or smaller systems.

4. Run update-grub.

   This command generates /boot/grub/grub.cfg that is directly used by the GRUB bootloader.
   It would include the menuentry entered in the last step.

5. Open VNC window in SolusIO.

6. Type reboot in SSH session.

7. A blue screen titled "GNU GRUB" should appear in VNC, but only for 3 seconds.
   Quickly press the ⬇️ key, select "Ubuntu 20.04 ISO" menu, and press ENTER.

   

8. You might see a message:

   error: no such module.

   Press any key to continue..._

   This is harmless, and you can press ENTER to bypass it.

9. A minute later, the Ubuntu installer should start, and display the "Please choose you preferred language" screen.

Part 2: Install Ubuntu from In-Memory ISO

Now that we have the installer started, but we need to take care of one more detail before proceeding.

1. On the very first "Please choose you preferred language" screen, press F2 key, which opens a shell.

2. Type the following commands:

   losetup -d /dev/loop0
   umount /isodevice
   

   These commands umount the ISO image, so that the same hard drive can be used as an installation target.

3. Type exit to return to the installer.

4. The installer would attempt to detect the network, and it will probably fail because the KVM server is IPv6-only and the network does not support IPv6 autoconfiguration.

   Here I select "Continue without network", because the downloaded ISO image contains all the essential packages.

   

5. The rest of installation process is same as any other Ubuntu Server installation.

6. The installed system will be without network configuration.
   It's necessary to login via VNC, add IP address and routes with ip addr add and ip route add commands, before I can SSH into the server and upload my expertly written Netplan configuration.

Common Errors

Temporary failure in name resolution

When I'm downloading the official Ubuntu ISO image, I encountered this error:

# wget https://releases.ubuntu.com/focal/ubuntu-20.04.2-live-server-amd64.iso
--2021-04-28 01:48:26--  https://releases.ubuntu.com/focal/ubuntu-20.04.2-live-server-amd64.iso
Resolving releases.ubuntu.com (releases.ubuntu.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘releases.ubuntu.com’

It turns out that the Debian 10 template from SolusIO has configured an IPv4 address as the DNS server, which does not work in an IPv6-only network:

# cat /etc/resolv.conf
nameserver 10.0.2.3

To resolve the error, configure an IPv6 DNS server such as Cloudflare DNS, before downloading the ISO image:

echo 'nameserver 2606:4700:4700::1111' >/etc/resolv.conf

/etc/grub.d/40_custom: menuentry: not found

When I run update-grub command, I encountered this error:

# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.19.0-16-amd64
Found initrd image: /boot/initrd.img-4.19.0-16-amd64
/etc/grub.d/40_custom: 2: /etc/grub.d/40_custom: menuentry: not found
done

This is because I mistakenly deleted the original content of /etc/grub.d/40_custom file, which starts with:

#!/bin/sh
exec tail -n +3 $0

The update-grub command expects every file in /etc/grub.d directory to be a bash script.
What the exec tail line does is that, it prints the content of this file starting from the third line, which happens to be our menuentry.
If this line is deleted, bash would interpret menuentry as a command name, which does not exist, thus causing the error.

To resolve the error, put these two lines back to the top of /etc/grub.d/40_custom.

Block probing did not discover any disks

The Ubuntu installer fails with this message:

Guided storage configuration

Block probing did not discover any disks. Unfortunately this means that
installation will not be possible.

This is because I did not run the commands to umount the ISO, so that the installer is unable to write to the hard drive that contains the ISO image.
Apparently this is a decade old bug.

To resolve the error, restart the server, enter the installer again, and remember to run the umount commands in a shell on the first screen.

Final Words

This article describes how to install Ubuntu 20.04 Server from the official ISO image on an IPv6-only KVM server managed by SolusIO platform.
The steps are verified on Hosterlabs IPv6 only servers - IPv6 Plus plan.

The KVM server hosting my website went offline last month.
Thinking the server might have crashed, I went to Virtualizor, the VPS control panel, to reboot the VPS.
It did not solve the problem, so I proceeded with my disaster recovery plan.

The hosting provider, Spartan Host, explained that it was a router bug.
They fixed the router after 4 hours, but my server did not come online.

Symptom

To investigate what went wrong with my VPS, I came back to Virtualizor to enable VNC access.
Having VNC access is like attaching a monitor and a keyboard to the server.
It would allow me to see any error messages printed on the screen and login to check whether there are configuration errors.

I didn't see any error through VNC connection.
Thinking it might be a routing problem, I logged in with username and password, and ran a traceroute.
To my surprise, the traceroute was able to reach Internet destination.
Moreover, I can SSH into this server again.

Seeing the problem went away, I disabled VNC access in Virtualizor.
Then, I pressed the reboot button in Virtualizor, so that the hypervisor would apply the VNC settings; rebooting via SSH would not be effective.

Then, I started a ping to the server from my desktop, and eagerly waited.
One minute, two minutes, …, the server did not come online.
What went wrong again?

I repeated the process, re-enabled VNC access, and saw nothing wrong.
I disabled VNC again, and the VPS lost connectivity again.
Clearly, there's a correlation between the VNC toggle and network connectivity.

Diagnostics Ⅰ

Virtualizor is known to push feature updates in patch releases that sometimes breaks things, so I asked whether there was a Virtualizor update recently, but the answer was no.
I couldn't figure out the problem, so I opened a support ticket with Spartan Host, and sent over the Netplan configuration file in this Ubuntu 20.04 server.

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
      addresses:
        - 2001:db8:2604:9cc0::1/64
        - 2001:db8:2604:9cc0::80/64:
            lifetime: 0
      routes:
        - to: ::/0
          via: 2001:db8:2604::1
          on-link: true

In this configuration:

• IPv4 address is acquired from DHCP, which is provided by the hypervisor.
• IPv6 is statically configured.
• NIC name ens3 is hard-coded, because it never changes.

I've been using similar Netplan configuration files in several other KVM servers, and never had a problem.

Ryan McCully, the managing director at Spartan Host, performed some tests on my KVM.
He found that, if VNC is disabled, the network interface on the VPS seems to be "completely dead", as there's no ARP or any other traffic seen at the hypervisor side.
He was also puzzled why VNC would affect network interface, but offered an explanation of why my VPS was working in the past few months:

• As mentioned above, setting changes in Virtualizor are applied to the hypervisor only after the reboot button in Virtualizor is pressed.
• Most likely, when I finished the initial setup, I disabled VNC but didn't reboot in Virtualizor.
• In this case, VNC tabs are hidden in Virtualizor, but VNC is still enabled on the hypervisor.

Diagnostics Ⅱ

Ryan spent a few more hours of extensive testing, and was able to reproduce this issue.
It turned out that disabling VNC in Virtualizor changes the network interface name.

The KVM hypervisor realizes VNC through an emulated VGA monitor.
Enabling VNC attaches a VGA controller to the virtual server, while disabling VNC detaches it.
To see this effect, we can schedule to run lspci command upon reboot in crontab, and look at the output file when we regain access.

$ : VNC enabled
$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:04.0 Multimedia audio controller: Intel Corporation 82801AA AC'97 Audio Controller (rev 01)
00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon

$ : VNC disabled
$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:03.0 Multimedia audio controller: Intel Corporation 82801AA AC'97 Audio Controller (rev 01)
00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon

More importantly, addition or removal of the VGA controller changes the PCI address of the Ethernet controller.
This, in turn, changes the network interface name, because Ubuntu adopts Consistent Network Device Naming, in which the name of a network interface is derived from its PCI address.

Therefore, "consistent" network device naming scheme is consistent only if the PCI address doesn't change.
However, PCI addresses aren't always stable.
I've seen PCI address changing on a dedicated server when I configure PCI bifurcation in BIOS settings.
Now I've seen PCI address changing on a KVM virtual server.

Treatment

My Netplan configuration assumes the Ethernet adapter on the KVM server is always "ens3".
Disabling VNC changes the network interface name to "ens2", and Netplan would not bring up an interface that it doesn't know about.
This caused the VPS to lose connectivity.

To solve this problem, the Netplan configuration should identify the network interface by its MAC address.
MAC address can be considered a stable identifier of the network interface, because it's one of the outputs from Virtualizor's Create VPS API.

Therefore, I changed Netplan configuration to this:

network:
  version: 2
  ethernets:
    uplink:
      match:
        macaddress: 6a:ed:d6:3b:49:f4
      set-name: uplink
      dhcp4: true
      addresses:
        - 2001:db8:2604:9cc0::1/64
        - 2001:db8:2604:9cc0::80/64:
            lifetime: 0
      routes:
        - to: ::/0
          via: 2001:db8:2604::1
          on-link: true

Unless the VPS is deleted and re-created with a different MAC address, this should continue to work.
I also renamed the network interface to "uplink", so that I don't need to check whether it's "ens3" or "ens2" when I type commands.

Elsewhere

I checked the VNC situation on several other KVM servers that I have.

WebHorizon and Evolution Host both modified Virtualizor such that there isn't an option to disable VNC.
This prevents the issue, but increases the risk of my VPS being compromised via VNC.

WebHosting24 kept the VNC option intact in Virtualizor.
Disabling VNC would lead to changed network interface names, but my new Netplan configuration works.

SolusVM, the VPS control panel used at VirMach and Nexril, keeps the VGA controller attached at all times.
Disabling VNC blocks the VNC port so that nobody can connect to it, but does not affect the KVM server itself.
I think this is a better approach.

Acknowledgement

Kudos to Ryan McCully at Spartan Host for helping me hunt down this issue.
I wouldn't have anticipated the root cause without his help.

I bought a few Virtual Private Servers (VPS) on Black Friday, and have been busy setting them up.
Nowadays, most VPS comes with an IPv6 subnet that contains millions of possible addresses.
Initially, only one IPv6 address is assigned to the server, but the user can assign additional addresses as desired.
Given that I plan to run multiple services within a server, I added a few more IPv6 addresses so that each service can have a unique IPv6 address.

One of my servers is using KVM virtualization technology, in which I installed Ubuntu 20.04 operating system manually from an ISO image.
Unlike a template-based installation, an ISO-installed Ubuntu 20.04 system manages its networks using Netplan, a backend-agnostic network configuration utility that generates network configuration from YAML files.
Most VPS control panels, including SolusVM and Virtualizer, are unable to generate the YAML files needed by Netplan.
IPv4 works out of box via DHCP, but IPv6 has to be configured manually.
To assign two IPv6 addresses to my server, I need to write the following in /etc/netplan/01-netcfg.yaml:

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
      addresses:
        - 2001:db8:30fa:5877::1/64
        - 2001:db8:30fa:5877::beef/64
      routes:
        - to: ::/0
          via: 2001:db8:30fa::1
          on-link: true
      nameservers:
        addresses:
        - 2001:4860:4860::8888
        - 2606:4700:4700::1111

I intend to host my secret beef recipes on its unique IPv6 address 2001:db8:30fa:5877::beef, and use the other address 2001:db8:30fa:5877::1 for outbound traffic such as pings and traceroutes.
However, I noticed that the wrong address is being selected for outgoing packets:

$ ping 2001:db8:57eb:8479::2

$ sudo tcpdump -n icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:44:48.704099 IP6 2001:db8:30fa:5877::beef > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 1, length 64
00:44:48.704188 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::beef: ICMP6, echo reply, seq 1, length 64
00:44:49.704011 IP6 2001:db8:30fa:5877::beef > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 2, length 64
00:44:49.704099 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::beef: ICMP6, echo reply, seq 2, length 64

I started searching for a solution, and learned that:

• Default Address Selection for Internet Protocol version 6 (IPv6) is a very complicated topic.

• An application can explicitly specify a source address.
  For example, I can invoke ping -I 2001:db8:30fa:5877::1 2001:db8:57eb:8479::2 to use the desired source address.

• Each local IPv6 address can be either "preferred" or "deprecated".
  If the application does not specify a source address, the system would prefer to use a "preferred" address instead of a "deprecated" address.

Currently, both addresses are "preferred" on my server:

$ ip addr show dev ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 92:b6:ab:eb:04:00 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.30/24 brd 192.0.2.255 scope global dynamic ens3
       valid_lft 21599977sec preferred_lft 21599977sec
    inet6 2001:db8:30fa:5877::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2001:db8:30fa:5877::beef/64 scope global
       valid_lft forever preferred_lft forever

This means, both addresses are equally possible of being used as the default source address.
If I can make 2001:db8:30fa:5877::1 "preferred" and all other addresses "deprecated", I would achieve my goal of making 2001:db8:30fa:5877::1 the default source address for outbound traffic.

How can I set an IPv6 address as "deprecated"?
After some digging, I found that it is controlled by the preferred_lft (preferred lifetime) attribute.
This attribute indicates the remaining time an IP address is to remain "preferred".
Unless it is set to "forever", preferred_lft counts down every second, and the IP address becomes "deprecated" when it reaches zero.
If the IP address was added with preferred_lft set to zero, it would be "deprecated" since the beginning.

Netplan gained support for preferred_lft setting very recently since version 0.100-0ubuntu4.
The syntax to specify preferred_lft of an IPv6 address is:

addresses:
  - 2001:db8:30fa:5877::1/64
  - 2001:db8:30fa:5877::beef/64:
      lifetime: 0

Notice that there is a colon (:) after the IPv6 address 2001:db8:30fa:5877::beef/64, because it is syntactically a map key instead of a string value.
The equivalent structure in JSON looks like:

{
  "addresses": [
    "2001:db8:30fa:5877::1/64",
    {
      "2001:db8:30fa:5877::beef/64": {
        "lifetime": 0
      }
    }
  ]
}

After applying this change, the IPv6 address 2001:db8:30fa:5877::beef is correctly marked as "deprecated" and no longer selected as the default source address.
Now I can securely host my secret beef recipes on 2001:db8:30fa:5877::beef without worrying about others discovering this "deprecated" IPv6 address through my outbound network traffic.

$ sudo netplan apply

$ ip addr show dev ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 92:b6:ab:eb:04:00 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.30/24 brd 192.0.2.255 scope global dynamic ens3
       valid_lft 21598263sec preferred_lft 21598263sec
    inet6 2001:db8:30fa:5877::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2001:db8:30fa:5877::beef/64 scope global deprecated
       valid_lft forever preferred_lft 0sec

$ ping 2001:db8:57eb:8479::2

$ sudo tcpdump -n icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:44:48.704099 IP6 2001:db8:30fa:5877::1 > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 1, length 64
00:44:48.704188 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::1: ICMP6, echo reply, seq 1, length 64
00:44:49.704011 IP6 2001:db8:30fa:5877::1 > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 2, length 64
00:44:49.704099 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::1: ICMP6, echo reply, seq 2, length 64

This article explained how to change default IPv6 source address selection by marking an IPv6 address "deprecated" via Netplan.
The described technique works in KVM and Ubuntu 20.04, and has been tested in a VPS provided by Spartan Host.
If you are using a OpenVZ 7 container, check out How to Select Default IPv6 Source Address for Outbound Traffic in OpenVZ 7.

Wireguard is now in bionic-updates.
PPA+dkms no longer required for bionic users.

Apparently, there was an official announce by Jason Donenfeld (author) on the mailing list on the next day : Aug 3, 2020.

You can safely remove the PPA config from /etc/apt/sources.d/ if you were previously on it on the supported OSes.
https://lists.zx2c4.com/pipermail/wireguard/2020-August/005737.html

Hi folks,

At long last, Ubuntu now supports WireGuard on releases 20.04, 19.10,
18.04, and 16.04, which means we've got all currently supported LTS
releases covered. For that reason, we're in the process of sunsetting
the PPA that previously provided packages to some users. This email
details possible changes users might consider.

The right way to install WireGuard on Ubuntu now consists of a single
command:

$ sudo apt install wireguard

This "wireguard" package will automatically pull in either one or two
packages with it:

1) wireguard-tools: this will always be pulled in and provides wg(8)
and wg-quick(8).
2) wireguard-dkms: this will only be pulled in if your kernel doesn't
already come with WireGuard.

As suggested by (2), most Ubuntu kernels now come with WireGuard out of
the box, even older releases, to which WireGuard has been backported.
This is great news and will result in much better reliability during
upgrades, as well as smoother compatibility with SecureBoot.

--snipped--

As a very good general recommendation for people new to wireguard ,
USE https://github.com/Nyr/wireguard-install/ (read the README first)

I have reviewed nyr's wireguard-install script and I am satisfied that it is well done.
I compared it with a manually configured wireguard tunnel, and all the defaults seem sensible.
Anything that saves you time should be welcomed.
Shoutout to @nyr, a long time LET/S MVP.

For anyone running Debian or Ubuntu in OpenVZ 7 containers, what has your experience been upgrading between OS versions, à la dist-upgrade/do-release-upgrade? Is it safe to assume it works these days, or would you rather reinstall a provided template?

@AnthonySmith At InceptionHosting in particular, is it safe to upgrade my servers without reinstalling?

Thanks.