Installing OpenWRT on top of LXC in Proxmox

terrorgen

I recently realized that I don't get the full speed promised by my ISP, and the reason behind it is that I run my OpenWRT router in KVM. I do not have a Ryzen to run my Proxmox and instead of shelling some cash to get one, I decided to try to run OpenWRT on LXC instead.

OpenWRT on LXC is not supported either by OpenWRT nor Proxmox, but there are multiple successful cases, plus an image is available on vanilla LXC, so I decided to give it a try.

I am documenting here what works for me as they differ from what was being documented. Maybe this would encourage providers here to offer an OpenWRT template on their Proxmox host

If you decide to follow what I wrote here, remember YMMV.

The combination that works: Proxmox VE 6.3-3, privileged container, OpenWRT Snapshot (20201216).

# pveversion
pve-manager/6.3-3/eee5f901 (running kernel: 5.4.78-2-pve)

I couldn't make it work with unprivileged containers nor the latest stable.

First download the rootfs.tar.gz from https://downloads.openwrt.org/snapshots/targets/x86/64/.

Next, get opct.sh from https://github.com/DazzyWalkman/oplxc4pve. This will make container creation and update much, much easier.

Open EDITOR (vi, nano, emacs,...) and change parameters to fit your Proxmox install. You'll also define how much memory, swap, cpu cores you'll allocate to this container here. Remember to change unprivileged to '0' or it may not work.

Pay attention to the bind mount settings. It came biting me when I restarted my Proxmox machine and found that the OpenWRT container refused to start, since I do not have the host directory set up. I am not sure if commenting it out would work, but it is worth a try.

Once you're done editing, give opct.sh executable privileges and copy to $PATH.

Next, run opct.sh new <VMID> <LOCATION OF OPENWRT ROOTFS>

An OpenWRT container will appear on Proxmox's CP. All you need to do now is rename the container (in DNS), add network interfaces, and start the container to configure OpenWRT to your needs.

miegl

@terrorgen said: I recently realized that I don't get the full speed I have promised by my ISP, and the reason behind it is that I run my OpenWRT router in KVM.

That's weird, do you know why that was?

Thanks for the guide, however I fail to see the point of running OpenWRT on beefy x86_64 hardware. The whole point of OpenWRT is to make a distribution viable for 4MB 8MB TP-Link routers. Why would you run that on a beefy piece of hardware, when you can use a Linux distribution with way larger selection of software instead?

terrorgen

Probably KVM overheads. Even with virtio I am unable to push line speed.

I run a few iperf tests to reach that conclusion.
Here is what I observed (IIRC):
Host to LXC: 20+ Gbps
LXC to LXC: 20+ Gbps
LXC to KVM: 40+ Mbps (yes M, not G)
KVM to KVM: 10-20 Mbps

@miegl said:

Why would you run that on a beefy piece of hardware

Because OpenWRT's origin on small ARM routers makes it an extremely efficient linux distro for networking. I had PfSense and OPNSense as my main routers in the past. While they are feature rich, they take minutes to boot. OpenWRT on x64, on the other hand, takes less than a minute to boot on KVM, even less on LXC.

OpenWRT also comes with all the routing related kernel parameters turned on. These are usually turned off for general purpose distros.

The number of packages you can install is limited by the amount of flash available on the ARM router. This limit is non existent on x64. There are thousands of packages available officially on OpenWRT's repo (see https://downloads.openwrt.org/snapshots/packages/x86_64/) and more if you count 3rd party repositories.

I won't run everything I need run on the OpenWRT. I have other LXC/KVMs for that. For those I need run on OpenWRT, I haven't found any that wouldn't.

havoc

Everything about that sounds odd.

The proxmox bridge runs at 10gbps afaik & KVM overheads should be minimal. I'm guessing there is some element of hardware acceleration that is being passed through to LXC but not KVM?

lebuser

@miegl said:
Why would you run that on a beefy piece of hardware, when you can use a Linux distribution with way larger selection of software instead?

Which Linux distribution do you recommend for routers? Do they have DHCPv6-PD clients, and servers and are able to receive IPv6 prefixes from the upstream routers, which they split into sub-prefixes that are assigned to interfaces, or delegated to other routers using its DHCPv6 server?

terrorgen

@havoc said:
Everything about that sounds odd.

The proxmox bridge runs at 10gbps afaik & KVM overheads should be minimal. I'm guessing there is some element of hardware acceleration that is being passed through to LXC but not KVM?

I am not sure...

miegl

@terrorgen said: Because OpenWRT's origin on small ARM routers makes it an extremely efficient linux distro for networking. I had PfSense and OPNSense as my main routers in the past. While they are feature rich, they take minutes to boot. OpenWRT on x64, on the other hand, takes less than a minute to boot on KVM, even less on LXC.

I agree with that - OpenWRT works wonders for what it's supposed to do. The thing is, once you go from a simple setup to a more advanced one, you will be limited by the networking scripts, and depending on your requirements, you might be after a more complex network management solution, like systemd-networkd.

@terrorgen said: The number of packages you can install is limited by the amount of flash available on the ARM router. This limit is non existent on x64. There are thousands of packages available officially on OpenWRT's repo (see https://downloads.openwrt.org/snapshots/packages/x86_64/) and more if you count 3rd party repositories.

OpenWRT has a great selection of network-related packages, however, they are sometimes lacking in features that are normally found on standard Linux distributions.

I would say my biggest problem with OpenWRT is updating - feature-rich package managers that are found on modern desktop/server distributions have no problems with updating even to a new release. But with OpenWRT, your best bet is to compile an image of its new version and reflash/reinstall. Even updating single packages with opkg can be problematic and is generally not recommended.

@lebuser said: Which Linux distribution do you recommend for routers?

Depends what you need. If you don't need other functionality that is not baked into OpenWRT, there is probably no reason to use something else. OpenWRT is great for the basic routing it provides.
I, for example, use OpenWRT on my Wi-Fi APs and simple routers. But, for my main router, where I have a ton of tunnels, ip rules, ip addresses, network services and bunch of other crap, OpenWRT would be a nightmare.
I think you can use just about any general-purpose/server distributions. My heart in this case leans close to Debian - due to the stability and huge software repos it provides.

@lebuser said: Do they have DHCPv6-PD clients, and servers and are able to receive IPv6 prefixes from the upstream routers, which they split into sub-prefixes that are assigned to interfaces, or delegated to other routers using its DHCPv6 server?

systemd-networkd, which is by default in every major modern distribution, meets all your requirements.

terrorgen

@miegl said: OpenWRT has a great selection of network-related packages, however, they are sometimes lacking in features that are normally found on standard Linux distributions.

Why do you need non networking related packages on OpenWRT? It's not like I am going to develop the Next Big Thing(TM) there!

@miegl said: I would say my biggest problem with OpenWRT is updating

For upgrading OpenWRT on x86, all you need to do is download the kernel, place it on /boot/vmlinuz, then manually download and install the kernel.ipk which is just a flag to tell the package manager that you have the right kernel, reboot, update repos, then upgrade all packages. No recompiling, no reflashing.

@miegl said: for my main router, where I have a ton of tunnels, ip rules, ip addresses, network services and bunch of other crap, OpenWRT would be a nightmare.

Luckily I don't have anything that complex, so OpenWRT works for me. I'll probably switch to Alpine Linux if I ever graduated from OpenWRT. I currently run a few Debian VPSs from the great providers here.

lebuser

@miegl said:

systemd-networkd, which is by default in every major modern distribution, meets all your requirements.

There is no DHCPv6 server in systemd anyway. A DHCPv6 server with prefix delegation was one of the requirements.

Open issue on systemd: network: implement DHCPv6 Server #16039