The KVM server hosting my website went offline last month.
Thinking the server might have crashed, I went to Virtualizor, the VPS control panel, to reboot the VPS.
It did not solve the problem, so I proceeded with my disaster recovery plan.

The hosting provider, Spartan Host, explained that it was a router bug.
They fixed the router after 4 hours, but my server did not come online.

Symptom

To investigate what went wrong with my VPS, I came back to Virtualizor to enable VNC access.
Having VNC access is like attaching a monitor and a keyboard to the server.
It would allow me to see any error messages printed on the screen and login to check whether there are configuration errors.

I didn't see any error through VNC connection.
Thinking it might be a routing problem, I logged in with username and password, and ran a traceroute.
To my surprise, the traceroute was able to reach Internet destination.
Moreover, I can SSH into this server again.

Seeing the problem went away, I disabled VNC access in Virtualizor.
Then, I pressed the reboot button in Virtualizor, so that the hypervisor would apply the VNC settings; rebooting via SSH would not be effective.

Then, I started a ping to the server from my desktop, and eagerly waited.
One minute, two minutes, …, the server did not come online.
What went wrong again?

I repeated the process, re-enabled VNC access, and saw nothing wrong.
I disabled VNC again, and the VPS lost connectivity again.
Clearly, there's a correlation between the VNC toggle and network connectivity.

Diagnostics Ⅰ

Virtualizor is known to push feature updates in patch releases that sometimes breaks things, so I asked whether there was a Virtualizor update recently, but the answer was no.
I couldn't figure out the problem, so I opened a support ticket with Spartan Host, and sent over the Netplan configuration file in this Ubuntu 20.04 server.

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
      addresses:
        - 2001:db8:2604:9cc0::1/64
        - 2001:db8:2604:9cc0::80/64:
            lifetime: 0
      routes:
        - to: ::/0
          via: 2001:db8:2604::1
          on-link: true

In this configuration:

• IPv4 address is acquired from DHCP, which is provided by the hypervisor.
• IPv6 is statically configured.
• NIC name ens3 is hard-coded, because it never changes.

I've been using similar Netplan configuration files in several other KVM servers, and never had a problem.

Ryan McCully, the managing director at Spartan Host, performed some tests on my KVM.
He found that, if VNC is disabled, the network interface on the VPS seems to be "completely dead", as there's no ARP or any other traffic seen at the hypervisor side.
He was also puzzled why VNC would affect network interface, but offered an explanation of why my VPS was working in the past few months:

• As mentioned above, setting changes in Virtualizor are applied to the hypervisor only after the reboot button in Virtualizor is pressed.
• Most likely, when I finished the initial setup, I disabled VNC but didn't reboot in Virtualizor.
• In this case, VNC tabs are hidden in Virtualizor, but VNC is still enabled on the hypervisor.

Diagnostics Ⅱ

Ryan spent a few more hours of extensive testing, and was able to reproduce this issue.
It turned out that disabling VNC in Virtualizor changes the network interface name.

The KVM hypervisor realizes VNC through an emulated VGA monitor.
Enabling VNC attaches a VGA controller to the virtual server, while disabling VNC detaches it.
To see this effect, we can schedule to run lspci command upon reboot in crontab, and look at the output file when we regain access.

$ : VNC enabled
$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:04.0 Multimedia audio controller: Intel Corporation 82801AA AC'97 Audio Controller (rev 01)
00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon

$ : VNC disabled
$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:03.0 Multimedia audio controller: Intel Corporation 82801AA AC'97 Audio Controller (rev 01)
00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon

More importantly, addition or removal of the VGA controller changes the PCI address of the Ethernet controller.
This, in turn, changes the network interface name, because Ubuntu adopts Consistent Network Device Naming, in which the name of a network interface is derived from its PCI address.

Therefore, "consistent" network device naming scheme is consistent only if the PCI address doesn't change.
However, PCI addresses aren't always stable.
I've seen PCI address changing on a dedicated server when I configure PCI bifurcation in BIOS settings.
Now I've seen PCI address changing on a KVM virtual server.

Treatment

My Netplan configuration assumes the Ethernet adapter on the KVM server is always "ens3".
Disabling VNC changes the network interface name to "ens2", and Netplan would not bring up an interface that it doesn't know about.
This caused the VPS to lose connectivity.

To solve this problem, the Netplan configuration should identify the network interface by its MAC address.
MAC address can be considered a stable identifier of the network interface, because it's one of the outputs from Virtualizor's Create VPS API.

Therefore, I changed Netplan configuration to this:

network:
  version: 2
  ethernets:
    uplink:
      match:
        macaddress: 6a:ed:d6:3b:49:f4
      set-name: uplink
      dhcp4: true
      addresses:
        - 2001:db8:2604:9cc0::1/64
        - 2001:db8:2604:9cc0::80/64:
            lifetime: 0
      routes:
        - to: ::/0
          via: 2001:db8:2604::1
          on-link: true

Unless the VPS is deleted and re-created with a different MAC address, this should continue to work.
I also renamed the network interface to "uplink", so that I don't need to check whether it's "ens3" or "ens2" when I type commands.

Elsewhere

I checked the VNC situation on several other KVM servers that I have.

WebHorizon and Evolution Host both modified Virtualizor such that there isn't an option to disable VNC.
This prevents the issue, but increases the risk of my VPS being compromised via VNC.

WebHosting24 kept the VNC option intact in Virtualizor.
Disabling VNC would lead to changed network interface names, but my new Netplan configuration works.

SolusVM, the VPS control panel used at VirMach and Nexril, keeps the VGA controller attached at all times.
Disabling VNC blocks the VNC port so that nobody can connect to it, but does not affect the KVM server itself.
I think this is a better approach.

Acknowledgement

Kudos to Ryan McCully at Spartan Host for helping me hunt down this issue.
I wouldn't have anticipated the root cause without his help.

I bought a few Virtual Private Servers (VPS) on Black Friday, and have been busy setting them up.
Nowadays, most VPS comes with an IPv6 subnet that contains millions of possible addresses.
Initially, only one IPv6 address is assigned to the server, but the user can assign additional addresses as desired.
Given that I plan to run multiple services within a server, I added a few more IPv6 addresses so that each service can have a unique IPv6 address.

One of my servers is using KVM virtualization technology, in which I installed Ubuntu 20.04 operating system manually from an ISO image.
Unlike a template-based installation, an ISO-installed Ubuntu 20.04 system manages its networks using Netplan, a backend-agnostic network configuration utility that generates network configuration from YAML files.
Most VPS control panels, including SolusVM and Virtualizer, are unable to generate the YAML files needed by Netplan.
IPv4 works out of box via DHCP, but IPv6 has to be configured manually.
To assign two IPv6 addresses to my server, I need to write the following in /etc/netplan/01-netcfg.yaml:

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
      addresses:
        - 2001:db8:30fa:5877::1/64
        - 2001:db8:30fa:5877::beef/64
      routes:
        - to: ::/0
          via: 2001:db8:30fa::1
          on-link: true
      nameservers:
        addresses:
        - 2001:4860:4860::8888
        - 2606:4700:4700::1111

I intend to host my secret beef recipes on its unique IPv6 address 2001:db8:30fa:5877::beef, and use the other address 2001:db8:30fa:5877::1 for outbound traffic such as pings and traceroutes.
However, I noticed that the wrong address is being selected for outgoing packets:

$ ping 2001:db8:57eb:8479::2

$ sudo tcpdump -n icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:44:48.704099 IP6 2001:db8:30fa:5877::beef > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 1, length 64
00:44:48.704188 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::beef: ICMP6, echo reply, seq 1, length 64
00:44:49.704011 IP6 2001:db8:30fa:5877::beef > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 2, length 64
00:44:49.704099 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::beef: ICMP6, echo reply, seq 2, length 64

I started searching for a solution, and learned that:

• Default Address Selection for Internet Protocol version 6 (IPv6) is a very complicated topic.

• An application can explicitly specify a source address.
  For example, I can invoke ping -I 2001:db8:30fa:5877::1 2001:db8:57eb:8479::2 to use the desired source address.

• Each local IPv6 address can be either "preferred" or "deprecated".
  If the application does not specify a source address, the system would prefer to use a "preferred" address instead of a "deprecated" address.

Currently, both addresses are "preferred" on my server:

$ ip addr show dev ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 92:b6:ab:eb:04:00 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.30/24 brd 192.0.2.255 scope global dynamic ens3
       valid_lft 21599977sec preferred_lft 21599977sec
    inet6 2001:db8:30fa:5877::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2001:db8:30fa:5877::beef/64 scope global
       valid_lft forever preferred_lft forever

This means, both addresses are equally possible of being used as the default source address.
If I can make 2001:db8:30fa:5877::1 "preferred" and all other addresses "deprecated", I would achieve my goal of making 2001:db8:30fa:5877::1 the default source address for outbound traffic.

How can I set an IPv6 address as "deprecated"?
After some digging, I found that it is controlled by the preferred_lft (preferred lifetime) attribute.
This attribute indicates the remaining time an IP address is to remain "preferred".
Unless it is set to "forever", preferred_lft counts down every second, and the IP address becomes "deprecated" when it reaches zero.
If the IP address was added with preferred_lft set to zero, it would be "deprecated" since the beginning.

Netplan gained support for preferred_lft setting very recently since version 0.100-0ubuntu4.
The syntax to specify preferred_lft of an IPv6 address is:

addresses:
  - 2001:db8:30fa:5877::1/64
  - 2001:db8:30fa:5877::beef/64:
      lifetime: 0

Notice that there is a colon (:) after the IPv6 address 2001:db8:30fa:5877::beef/64, because it is syntactically a map key instead of a string value.
The equivalent structure in JSON looks like:

{
  "addresses": [
    "2001:db8:30fa:5877::1/64",
    {
      "2001:db8:30fa:5877::beef/64": {
        "lifetime": 0
      }
    }
  ]
}

After applying this change, the IPv6 address 2001:db8:30fa:5877::beef is correctly marked as "deprecated" and no longer selected as the default source address.
Now I can securely host my secret beef recipes on 2001:db8:30fa:5877::beef without worrying about others discovering this "deprecated" IPv6 address through my outbound network traffic.

$ sudo netplan apply

$ ip addr show dev ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 92:b6:ab:eb:04:00 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.30/24 brd 192.0.2.255 scope global dynamic ens3
       valid_lft 21598263sec preferred_lft 21598263sec
    inet6 2001:db8:30fa:5877::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2001:db8:30fa:5877::beef/64 scope global deprecated
       valid_lft forever preferred_lft 0sec

$ ping 2001:db8:57eb:8479::2

$ sudo tcpdump -n icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:44:48.704099 IP6 2001:db8:30fa:5877::1 > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 1, length 64
00:44:48.704188 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::1: ICMP6, echo reply, seq 1, length 64
00:44:49.704011 IP6 2001:db8:30fa:5877::1 > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 2, length 64
00:44:49.704099 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::1: ICMP6, echo reply, seq 2, length 64

This article explained how to change default IPv6 source address selection by marking an IPv6 address "deprecated" via Netplan.
The described technique works in KVM and Ubuntu 20.04, and has been tested in a VPS provided by Spartan Host.
If you are using a OpenVZ 7 container, check out How to Select Default IPv6 Source Address for Outbound Traffic in OpenVZ 7.