What applications can't be run on LXC VPS? What are its limitations?

Ympker

This may be a very basic question, but I recently added a LXC VPS to my vps collection and while most of the VPS I have (or had in the past) are KVM VPS (back in the days lots of ovz, too), I was wondering what the limitations, in terms of use-cases, of a LXC VPS really are?

@Not_Oles has been providing the community with lots of nice&free LXC (and KVM?) vps, but so far I haven't found any clear limitations regarding LXC.

My take-away so far is:

• Virtualization/Proxmox is not possible, because LXC is a container
• Kernel tweaks are not possible, because you don't have your "own" kernel in LXC (I wouldn't know why I would need this, so I assume it doesn't really limit me in my usage)
• Host/Provider can easily enter LXC shell, with KVM there are ways to make it "harder"
• LXC is usually more oversold, but this doesn't really limit my use-case (performance can probably be even better if not heavily oversold due to no virtualization overhead?)
• Pterodactyl doesn't want you to use LXC
• Docker can be enabled for LXC containers (https://bobcares.com/blog/docker-inside-lxc/ ; and other tutorials)

So far, I didn't run into any limitations or issues. Granted, my use-case is mostly just OpenVPN/Wireguard, some webhosting panel (Keyhelp etc) and/or a gameserver.

Has there ever been an application or use-case you couldn't host on a LXC VPS?

ehab

k8s without lxc privileges and tweaks won't work.

did you try nfs?

Ympker

@ehab said:
k8s without lxc privileges and tweaks won't work.

did you try nfs?

Never dealt with Kubernates.
Didn't try NFS. Don't see myself needing Kubernates any time soon, but this, of course, can be relevant for others.
Thanks for bringing it up, mate!

havoc

Stuff that requires kernel modules. So like TUN/TAP for VPNs. Gitlab also needs some tweaks to work in LXC

Docker can be enabled for LXC containers (https://bobcares.com/blog/docker-inside-lxc/ ; and other tutorials)

Take care with that on shared servers - those instructions significantly weaken the barrier vs the host.

Ympker

@havoc said:
Stuff that requires kernel modules. So like TUN/TAP for VPNs. Gitlab also needs some tweaks to work in LXC

Docker can be enabled for LXC containers (https://bobcares.com/blog/docker-inside-lxc/ ; and other tutorials)

Take care with that on shared servers - those instructions significantly weaken the barrier vs the host.

I'm running ovpn and Wireguard server on a lxc vps with tun/tap enabled without an issue. Maybe there's a workaround for it, then?

jtk

@Ympker said:
I was wondering what the limitations, in terms of use-cases, of a LXC VPS really are?

I wouldn't say it is a limitation, but there may be some minor differences in the base OS that some apps may not be expecting. For example, you might only have /dev/tty, /dev/tty1, and /dev/tty2 devices in LXC and no more ttys than that. Say something like syslog-ng defaults to sending console messages to /dev/tty10. You will eventually realize the file /dev/tty10, not the device, growing to fill whatever the size of the partition /dev allows. Real example.

skorous

@Ympker said:

@havoc said:
Stuff that requires kernel modules. So like TUN/TAP for VPNs. Gitlab also needs some tweaks to work in LXC

Docker can be enabled for LXC containers (https://bobcares.com/blog/docker-inside-lxc/ ; and other tutorials)

Take care with that on shared servers - those instructions significantly weaken the barrier vs the host.

I'm running ovpn and Wireguard server on a lxc vps with tun/tap enabled without an issue. Maybe there's a workaround for it, then?

I believe if the LXC host is set up it can load the module and your LXC uses it.

Ympker

@skorous said:

@Ympker said:

@havoc said:
Stuff that requires kernel modules. So like TUN/TAP for VPNs. Gitlab also needs some tweaks to work in LXC

Docker can be enabled for LXC containers (https://bobcares.com/blog/docker-inside-lxc/ ; and other tutorials)

Take care with that on shared servers - those instructions significantly weaken the barrier vs the host.

I'm running ovpn and Wireguard server on a lxc vps with tun/tap enabled without an issue. Maybe there's a workaround for it, then?

I believe if the LXC host is set up it can load the module and your LXC uses it.

Yeah, I had to create some script, following the provider's knowledge base to make use of tun/tap:

!/bin/bash

mkdir /dev/net
mknod /dev/net/tun c 10 200
chmod 0666 /dev/net/tun

And add it as @reboot in crontab.

dwight

@havoc said:
Stuff that requires kernel modules. So like TUN/TAP for VPNs. Gitlab also needs some tweaks to work in LXC

Basically all containers have this problem; unless they don't share kernel

Neoon

You cannot generalize this, it depends on many factors, such as kernel version and what the provider uses or allows.
Same goes for OpenVZ, so for me its always a hit and miss, sometimes this does work, sometimes this does not.

They can allow docker or nested LXC or they can disable it.
I have seen Kernels from 3.x with LXC and up.

terrorgen

Also for LXC you are limited by the distro images your provider provides.

Usually not a problem but if you need a non common distro, you are out of luck.

Ympker

@terrorgen said:
Also for LXC you are limited by the distro images your provider provides.

Usually not a problem but if you need a non common distro, you are out of luck.

True, but usually also no problem for me (most hosts would have Debian/ubuntu) Still, fair point.

Ympker

@dwight said:

@havoc said:
Stuff that requires kernel modules. So like TUN/TAP for VPNs. Gitlab also needs some tweaks to work in LXC

Basically all containers have this problem; unless they don't share kernel

On my lxc I was able to manually enable tun/tap on my lxc (see my post above). In my case, provider also allowed Docker on lxc.

I realise this depends on the provider's settings, though. Maybe creating lxc on my dedi and trying to find out in how far I can limit it will be interesting to see

Not_Oles

Hi @Ympker!

I am not aware of any good source for a list of applications that cannot be run on LXC. What I do is Google the name of the application together with LXC. If there are issues, often I can see various forum or Github comments / issues.

A more general way to consider the question of what cannot be run on LXC is to look at the list of system calls which are prohibited in the LXC setup you are running. Obviously, anything that needs one of the prohibited system calls isn't going to work. Together with the system calls, there are devices and cgroups and namespaces to be considered.

I have yet to get my head above water in the LXC pond. There are multiple configuration files in different places, all of which need to be considered. There also is systemd, which sometimes can be used to start LXC. There is LXD, which has a confusingly named command called lxc.

The first place I'd send you is dshcherb's tutorial, Running QEMU/KVM Virtual Machines in Unprivileged LXD Containers I tried this tutorial fairly recently, and I failed, probably in part because I was trying not to use systemd to start lxc. If you look at Getting Started on linuxcontainers.org you will see that they used systemd and Ubuntu in those directions.

You also might want to look at Liz Rice's LXC tutorials and her videos on Youtube. She has simplified versions of LXC-like programs. The older ones are in C and the newer ones are in Go. Last but not least, there always is the LXC sources themselves.

With help from @yoursunny, I think we did get lxc running on the Alpine server I previously had. I've been wanting to go through that thread again, carefully, and make notes on the LXC configuration used there.

Haha, you will grok LXC before I do!

Ympker

@Not_Oles said:
Hi @Ympker!

I am not aware of any good source for a list of applications that cannot be run on LXC. What I do is Google the name of the application together with LXC. If there are issues, often I can see various forum or Github comments / issues.

A more general way to consider the question of what cannot be run on LXC is to look at the list of system calls which are prohibited in the LXC setup you are running. Obviously, anything that needs one of the prohibited system calls isn't going to work. Together with the system calls, there are devices and cgroups and namespaces to be considered.

I have yet to get my head above water in the LXC pond. There are multiple configuration files in different places, all of which need to be considered. There also is systemd, which sometimes can be used to start LXC. There is LXD, which has a confusingly named command called lxc.

The first place I'd send you is dshcherb's tutorial, Running QEMU/KVM Virtual Machines in Unprivileged LXD Containers I tried this tutorial fairly recently, and I failed, probably in part because I was trying not to use systemd to start lxc. If you look at Getting Started on linuxcontainers.org you will see that they used systemd and Ubuntu in those directions.

You also might want to look at Liz Rice's LXC tutorials and her videos on Youtube. She has simplified versions of LXC-like programs. The older ones are in C and the newer ones are in Go. Last but not least, there always is the LXC sources themselves.

With help from @yoursunny, I think we did get lxc running on the Alpine server I previously had. I've been wanting to go through that thread again, carefully, and make notes on the LXC configuration used there.

Haha, you will grok LXC before I do!

Hey, friend. You mentioned some interesting points. Will look into this more and maybe create some setup, where I can deploy lxc to learn more about the restrictions

Encoders

AFAIK the only can't do is running different OS. as you need to follow the host (if it's linux then you must use linux too), which post above already mention, you also have more restriction in distro selection.

LXC is tedious, sure it can run n or n+1 but you'll need to tweaks here and there, and if there's a provider using LXC they will almost certainly oversold (so you'll get crap performance for teh glorious $7). It's hard to justify using LXC when KVM/OpenVZ is easily available. I mean, can't imagine gentlemen and gentlewomen on LES spending hours just to tweak stuff, they'll lose millions on their business

Ympker

@Encoders said:
AFAIK the only can't do is running different OS. as you need to follow the host (if it's linux then you must use linux too), which post above already mention, you also have more restriction in distro selection.

LXC is tedious, sure it can run n or n+1 but you'll need to tweaks here and there, and if there's a provider using LXC they will almost certainly oversold (so you'll get crap performance for teh glorious $7). It's hard to justify using LXC when KVM/OpenVZ is easily available. I mean, can't imagine gentlemen and gentlewomen on LES spending hours just to tweak stuff, they'll lose millions on their business

Haha fair point Although, on the other green forum I also read about KVM using ballooning to over-provision RAM.
I'm not that familar with ballooning, and LinuxAteMyRam dictates that not all ram needs to be free at all times, just saying I assume there would be ways to oversell KVM, too?

Regarding justification for LXC: some providers simply charge less for LXC (because they can oversell more ofc).

Encoders

@Ympker said:

@Encoders said:
AFAIK the only can't do is running different OS. as you need to follow the host (if it's linux then you must use linux too), which post above already mention, you also have more restriction in distro selection.

LXC is tedious, sure it can run n or n+1 but you'll need to tweaks here and there, and if there's a provider using LXC they will almost certainly oversold (so you'll get crap performance for teh glorious $7). It's hard to justify using LXC when KVM/OpenVZ is easily available. I mean, can't imagine gentlemen and gentlewomen on LES spending hours just to tweak stuff, they'll lose millions on their business

Haha fair point Although, on the other green forum I also read about KVM using ballooning to over-provision RAM.
I'm not that familar with ballooning, and LinuxAteMyRam dictates that not all ram needs to be free at all times, just saying I assume there would be ways to oversell KVM, too?

Regarding justification for LXC: some providers simply charge less for LXC (because they can oversell more ofc).

overselling RAM is always possible but I highly doubt it since the market widely sees "KVM" is the better virtualization type. If a provider want to boast/marketing-speak about KVM against other virt type, they'll talk about performance directly, hence they will likely to avoid overselling their RAM

skorous

@Ympker said:

@Encoders said:
AFAIK the only can't do is running different OS. as you need to follow the host (if it's linux then you must use linux too), which post above already mention, you also have more restriction in distro selection.

LXC is tedious, sure it can run n or n+1 but you'll need to tweaks here and there, and if there's a provider using LXC they will almost certainly oversold (so you'll get crap performance for teh glorious $7). It's hard to justify using LXC when KVM/OpenVZ is easily available. I mean, can't imagine gentlemen and gentlewomen on LES spending hours just to tweak stuff, they'll lose millions on their business

Haha fair point Although, on the other green forum I also read about KVM using ballooning to over-provision RAM.
I'm not that familar with ballooning, and LinuxAteMyRam dictates that not all ram needs to be free at all times, just saying I assume there would be ways to oversell KVM, too?

Regarding justification for LXC: some providers simply charge less for LXC (because they can oversell more ofc).

Ballooning just has an agent in the instance that when the host is short on free memory "allocates" memory ( like a balloon to fill space inside a box ) to guarantee it isn't actually written to.

Ympker

@skorous said:

@Ympker said:

@Encoders said:
AFAIK the only can't do is running different OS. as you need to follow the host (if it's linux then you must use linux too), which post above already mention, you also have more restriction in distro selection.

LXC is tedious, sure it can run n or n+1 but you'll need to tweaks here and there, and if there's a provider using LXC they will almost certainly oversold (so you'll get crap performance for teh glorious $7). It's hard to justify using LXC when KVM/OpenVZ is easily available. I mean, can't imagine gentlemen and gentlewomen on LES spending hours just to tweak stuff, they'll lose millions on their business

Haha fair point Although, on the other green forum I also read about KVM using ballooning to over-provision RAM.
I'm not that familar with ballooning, and LinuxAteMyRam dictates that not all ram needs to be free at all times, just saying I assume there would be ways to oversell KVM, too?

Regarding justification for LXC: some providers simply charge less for LXC (because they can oversell more ofc).

Ballooning just has an agent in the instance that when the host is short on free memory "allocates" memory ( like a balloon to fill space inside a box ) to guarantee it isn't actually written to.

Ah okay, fair enough. Thanks!

havoc

@Ympker said:

On my lxc I was able to manually enable tun/tap on my lxc (see my post above). In my case, provider also allowed Docker on lxc.

I realise this depends on the provider's settings, though. Maybe creating lxc on my dedi and trying to find out in how far I can limit it will be interesting to see

Pretty sure that script only works cause the provider enabled it their side too. See here:

https://pve.proxmox.com/wiki/OpenVPN_in_LXC

tbh I usually give up pretty fast with this stuff. It's on a home server so I just enable whatever is needed or switch to VM.

Gitlab is the only one I put time in cause it's a resource hog so having it in a thinner isolation LXC was of interest

Ympker

@havoc said:

@Ympker said:

On my lxc I was able to manually enable tun/tap on my lxc (see my post above). In my case, provider also allowed Docker on lxc.

I realise this depends on the provider's settings, though. Maybe creating lxc on my dedi and trying to find out in how far I can limit it will be interesting to see

Pretty sure that script only works cause the provider enabled it their side too. See here:

https://pve.proxmox.com/wiki/OpenVPN_in_LXC

tbh I usually give up pretty fast with this stuff. It's on a home server so I just enable whatever is needed or switch to VM.

Gitlab is the only one I put time in cause it's a resource hog so having it in a thinner isolation LXC was of interest

Probably the same way Docker is enabled for LXC with my provider, because I remember Docker usually wouldn't be a LXC thing, right?
Anyway, guess I am happy they enabled it then
I'd probably be the reverse example. On my dedi, I setup Proxmox with LXC, because the templates were super easy to dl&run and I didn't run into any issues (yet).

rubenmdh

Any *BSD system. I'd say that its limitations are similar to OpenVZ, but with a current kernel. But I use it daily and it is pretty awesome, but functionality depends on LXC implementation.

havoc

https://thelastguardian.me/posts/2020-01-10-kubernetes-in-lxc-on-proxmox/

Ympker

@havoc said:
https://thelastguardian.me/posts/2020-01-10-kubernetes-in-lxc-on-proxmox/

lol, the site's name "Thelastguardian"

jtk

Another potential issue with any container-based virtualization, including LXC is that you are at the mercy of the host's clock. ntpd, chrony, OpenNTPD, etc. won't be able to help you, because the container shares a system hardware clock that it cannot adjust. If the host clock is not keeping an accurate notion of time, this could be a minor annoyance or a huge problem depending on your use-case. I'm not aware of any providers who are known to have failed in this basic function, but I wouldn't be surprised if they are out there.

terrorgen

@jtk said:
Another potential issue with any container-based virtualization, including LXC is that you are at the mercy of the host's clock. ntpd, chrony, OpenNTPD, etc. won't be able to help you, because the container shares a system hardware clock that it cannot adjust. If the host clock is not keeping an accurate notion of time, this could be a minor annoyance or a huge problem depending on your use-case. I'm not aware of any providers who are known to have failed in this basic function, but I wouldn't be surprised if they are out there.

I had a KVM provider who messed up their time and nothing can be done on their vps.

jtk

@terrorgen said:
I had a KVM provider who messed up their time and nothing can be done on their vps.

You should be able to run an ntp daemon on your KVM to mitigate that no?

terrorgen

@jtk said:

@terrorgen said:
I had a KVM provider who messed up their time and nothing can be done on their vps.

You should be able to run an ntp daemon on your KVM to mitigate that no?

I tried but for some reasons they blocked ntp on their firewall 🤷🏻‍♂️

jtk

@terrorgen said:
I tried but for some reasons they blocked ntp on their firewall 🤷🏻‍♂️

Can you name the provider? I'd guess they were blocking more, perhaps all of UDP too?

terrorgen

@jtk said:

@terrorgen said:
I tried but for some reasons they blocked ntp on their firewall 🤷🏻‍♂️

Can you name the provider? I'd guess they were blocking more, perhaps all of UDP too?

Probably