I'm not sure this is right, because we don't even have installed either the LXD package or the lxc command (which is part of LXD and not part of LXC). We do have the lxc and lxcfs packages installed, but they do not include the lxc command.
fmt:~# apk info lxd
lxd-5.0.2-r3 description:
A container hypervisor and a new user experience for LXC - 'LTS' release channel
lxd-5.0.2-r3 webpage:
https://linuxcontainers.org/lxd/
lxd-5.0.2-r3 installed size:
43 MiB
fmt:~# apk -e info lxd
fmt:~# # No output here means it's not installed.
I tried commenting out bash and adding a line about systemd to my /home/notoles Debian LXC config.
~/.local/share/lxc/debian $ cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist debian --release sid --arch amd64
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64
# Container specific configuration
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 1000000000 65536
lxc.idmap = g 0 1000000000 65536
lxc.rootfs.path = dir:/home/notoles/.local/share/lxc/debian/rootfs
lxc.uts.name = debian
# Network configuration
lxc.net.0.ipv4.address = 192.168.188.11/24
lxc.net.0.ipv4.gateway = 192.168.188.1
lxc.net.0.ipv6.address = 2602:fba1:999:1c00:11::/64
lxc.net.0.ipv6.gateway = 2602:fba1:999:1c00::
# lxc.init.cmd = /bin/bash
systemd_container=yes
~/.local/share/lxc/debian $
The result was these errors:
fmt:~$ lxc-start -F -n debian
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
fmt:~$
If there is no way to get Alpine's LXC without LXD to run an unprivileged container for a systemd OS, then, yes, maybe we should install LXD.
I will check to see if starting the unprivileged container with bash and then stopping/disabling systemd-networkd is enough to get an unprivileged Debian container to start in the usual way, without setting PID 1 as bash.
@yoursunny seems pretty sure that systemd won't work inside the LXC containers:
@yoursunny said: systemd in the container will not work, so the entry process is changed to bash.
@terrorgen Did you try changing the entry process to bash for your nix container?
Do you guys understand how/why making unprivileged containers with LXD as explained in https://wiki.alpinelinux.org/wiki/LXD could enable systemd to work inside those containers when getting systemd working might not be possible inside unprivileged containers made with lxc-create? What's the difference between LXD's lxc command and LXC's lxc-create command that allows systemd to work with unprivileged LXD containers but not with unprivileged LXC containers?
@Not_Oles said: I'm not sure this is right, because we don't even have installed either the LXD package or the lxc command (which is part of LXD and not part of LXC). We do have the lxc and lxcfs packages installed, but they do not include the lxc command.
But /etc/conf.d/lxc does exist, so it may not have to do with LXD.
@Not_Oles said: @terrorgen Did you try changing the entry process to bash for your nix container?
That won't work. NixOS is heavily dependant on systemd to get things working. Also, LXC "boots" the container by starting /sbin/init inside the container. In a NixOS container this is actually a generated bash script, which among other things contains the system configuration that will be activated at "boot". My init script is throwing errors because it cannot find a proper sys mount.
@Not_Oles said: What's the difference between LXD's lxc command and LXC's lxc-create
My understanding is, LXD's lxc command is a higher level management tool that abstracts away the lower level configuration of lxc-*. It can also manage VMs. I don't claim to be an expert for either, though.
@Not_Oles said: By the way, where / how did you get your nix LXC image?
Found a guide that downloads the rootfs tarball from NixOS's own repository. NixOS developers decided against publishing them to linuxcontainers.org because it seems counterintuitive.
I personally have a NixOS LXC container running in my homelab Proxmox server, so I know it can be done.
So it just clicked that most of my problems (besides cgroup delegation) is permissions:
❯ lxc-start nixos -F
lxc-start: nixos: ../src/lxc/utils.c: safe_mount: 1220 Resource busy - Failed to mount "sys" onto "/usr/lib/lxc/rootfs/dev/.lxc/sys"
<<< NixOS Stage 2 >>>
install: cannot change permissions of '/tmp': Operation not permitted
running activation script...
ln: failed to create symbolic link '/bin/.sh.tmp': Permission denied
mv: cannot stat '/bin/.sh.tmp': No such file or directory
Activation script snippet 'binsh' failed (1)
install: cannot change permissions of '/root': Operation not permitted
mkdir /var/lib: Permission denied at /nix/store/snb4523ghvw9917q15j401fz26d5plh3-update-users-groups.pl line 17.
Activation script snippet 'users' failed (13)
setting up /etc...
Died at /nix/store/rg5rf512szdxmnj9qal3wfdnpfsx38qi-setup-etc.pl line 27.
Activation script snippet 'etc' failed (13)
/nix/store/8ndxpvlgfjbbas506vqrad69rzjzxwsp-nixos-system-nixos-23.05pre452927.6ccc4a59c3f/activate: line 129: /etc/shadow: No such file or directory
Activation script snippet 'hashes' failed (1)
ln: failed to create symbolic link '/sbin/init': Permission denied
Activation script snippet 'installInitScript' failed (1)
install: cannot create directory '/nix/var': Permission denied
install: cannot create directory '/nix/var': Permission denied
/nix/store/8ndxpvlgfjbbas506vqrad69rzjzxwsp-nixos-system-nixos-23.05pre452927.6ccc4a59c3f/activate: line 167: /root/.nix-channels: Permission denied
Activation script snippet 'nix' failed (1)
mkdir: cannot create directory '/usr/bin': Permission denied
ln: failed to create symbolic link '/usr/bin/.env.tmp': No such file or directory
mv: cannot stat '/usr/bin/.env.tmp': No such file or directory
Activation script snippet 'usrbinenv' failed (1)
mkdir: cannot create directory '/var/tmp': Permission denied
mkdir: cannot create directory '/var/empty': Permission denied
find: '/var/empty': No such file or directory
chmod: cannot access '/var/empty': No such file or directory
chown: invalid user: 'root:root'
Activation script snippet 'var' failed (1)
chown: invalid user: 'root:root'
chown: invalid user: 'root:messagebus'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
chown: invalid user: 'root:root'
Activation script snippet 'wrappers' failed (1)
mkdir: cannot create directory '/nix/var': Permission denied
ln: failed to create symbolic link '/nix/var/nix/gcroots/current-system': No such file or directory
cp: cannot create regular file '/etc/nixos/configuration.nix': Permission denied
terminate called after throwing an instance of 'nix::Error'
what(): error: cannot determine user's home directory
/nix/store/l411104qj58cq7f1gg2wiryi0lzly5jk-local-cmds: line 17: 132 Aborted /nix/store/nnznavnhyli08264apz6lanbjza48si1-nix-2.13.2/bin/nix-store --load-db < /nix-path-registration
terminate called after throwing an instance of 'nix::Error'
what(): error: cannot determine user's home directory
/nix/store/l411104qj58cq7f1gg2wiryi0lzly5jk-local-cmds: line 20: 133 Aborted /nix/store/nnznavnhyli08264apz6lanbjza48si1-nix-2.13.2/bin/nix-env -p /nix/var/nix/profiles/system --set /run/current-system
unpacking the NixOS/Nixpkgs sources...
mkdir: cannot create directory '/nix/var': Permission denied
terminate called after throwing an instance of 'nix::Error'
what(): error: cannot determine user's home directory
/nix/store/l411104qj58cq7f1gg2wiryi0lzly5jk-local-cmds: line 32: 135 Aborted /nix/store/nnznavnhyli08264apz6lanbjza48si1-nix-2.13.2/bin/nix-env -p /nix/var/nix/profiles/per-user/root/channels -i /nix/store/70hcm36cm9v6wwvl224w2zvxvshrh1ff-nixos-23.05pre452927.6ccc4a59c3f --quiet --option build-use-substitutes false
mkdir: cannot create directory '/root/.nix-defexpr': Permission denied
ln: failed to create symbolic link '/root/.nix-defexpr/channels': No such file or directory
mkdir: cannot create directory '/var/lib': Permission denied
touch: cannot touch '/var/lib/nixos/did-channel-init': No such file or directory
/sbin/init: line 130: /etc/machine-id: Permission denied
starting systemd...
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
Comparing notes between my homelab NixOS container vs MetalVPS's...
rootfs permission in my homelab is set to uid:100000 and gid:100000, which maps to root:root in the container.
Whereas my rootfs' permission in MetalVPS is set to my uid:gid.
so I did a tweak in my config file:
lxc.idmap = u 0 1015 1
lxc.idmap = g 0 1015 1
lxc.idmap = u 1 1015000000 65535
lxc.idmap = g 1 1015000000 65535
so the container root is myself in MetalVPS.
solved most of the permission issues above.
now this is how it looks like:
❯ lxc-start nixos -F
lxc-start: nixos: ../src/lxc/utils.c: safe_mount: 1220 Resource busy - Failed to mount "sys" onto "/usr/lib/lxc/rootfs/dev/.lxc/sys"
<<< NixOS Stage 2 >>>
running activation script...
setting up /etc...
starting systemd...
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
@terrorgen said: So it just clicked that most of my problems (besides cgroup delegation) is permissions
I did a tweak in my config file:
lxc.idmap = u 0 1015 1
lxc.idmap = g 0 1015 1
lxc.idmap = u 1 1015000000 65535
lxc.idmap = g 1 1015000000 65535
so the container root is myself in MetalVPS.
Congrats on figuring out the permissions issues!
@terrorgen said: Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
Looks just like the errors I am seeing when starting a Debian container:
@Not_Oles said:
fmt:~$ lxc-start -F -n debian
Failed to find module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
fmt:~$
@terrorgen I will look at this some more, including your suggested changes to the node configuration and raising the number of containers permission limit.
Now that you fixed your user permissions, might starting with bash as PID 1 work? And be helpful?
rc-update add dbus
Reboot and lxd should be working.
fmt:~# date
Thu Feb 23 00:35:42 UTC 2023
fmt:~# cat /etc/conf.d/lxc
# Configuration for /etc/init.d/lxc[.*]
# Enable cgroup for systemd-based containers.
#systemd_container=no
systemd_container=yes
# autostart groups (comma separated)
#lxc_group="onboot"
# Directory for containers' logs (used for symlinked runscripts lxc.*).
#logdir="/var/log/lxc"
fmt:~#
fmt:~# rc-update add lxc
* service lxc added to runlevel default
fmt:~# rc-update add lxd
* rc-update: service `lxd' does not exist
fmt:~# rc-update add lxcfs
* service lxcfs added to runlevel default
fmt:~# rc-update add dbus
* rc-update: service `dbus' does not exist
fmt:~#
Do we need to apk add lxd and apk add dbus plus enable both before rebooting, or is it worth while to try just adding lxc and lxcfs to runlevel default?
Basically you’d need root to crate you a /sys/fs/cgroup/user.doskanoness cgroup or something similar, then chown it over to you and move your shell’s PID into it.
At that point, lxc-start should be able to detect that and since you now own that cgroup, will be able to create its own entries in there for the container.
I'd want to read about making cgroups, but it looks like we could do it. Do you want to go ahead with cgroups, or stay awhile longer on the present path? Should we (1) go ahead and reboot, (2) add lxd and dbus and then reboot, (3) revert the changes I just made to /etc/conf.d/lxc and the rc scripts, (4) work on the cgroups, or (5) some combination?
Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
Solution proposed as:
OK - I figured it out after stepping away from it and trying fresh this morning. I had an issue in my /etc/pam.d/system-login file. I fixed the cgfs line to read like this:
Now that I blew my budget on the i9-12900K EX100 I got from Hetzner I regretfully have had to ask @Cloudie to cancel this server. The next payment is due March 3. I don't know exactly when the cancellation will occur.
@yoursunny@terrorgen@subenhon You guys have been using the fmt server lately. I don't know whether @Cloudie would allow it, and I don't know what the price might be, but perhaps one or the group of you might want to take over the server? You could try contacting @Cloudie, perhaps via Route48's Discord.
Guys who have been using this server are warmly invited to request accounts in the above linked EX100 thread. I realize the EX100 is in Finland instead of California, and that makes a big latency difference. Another big difference is that the Helsinki server runs Debian sid and not Alpine.
I hope the EX100 will stay around for awhile! I'm not planning to cancel it. I'm looking forward to seeing you guys in Helsinki!
@zudaz said:
If possible, I would like to get one.
pm key
Congrats on your first post
"A single swap file or partition may be up to 128 MB in size. [...] [I]f you need 256 MB of swap, you can create two 128-MB swap partitions." (M. Welsh & L. Kaufman, Running Linux, 2e, 1996, p. 49)
Hi @zudaz! Sorry, but this server has been returned to @Cloudie. Welcome to LES! If you want to post a little about who and where you are and about what you are doing, I am sure everybody would be interested. It's a nice group here at LES! Best wishes! Tom
Comments
Edited. . . .
MetalVPS
@terrorgen Please note 12, 13, and 14.
MetalVPS
I'm not sure this is right, because we don't even have installed either the LXD package or the
lxccommand (which is part of LXD and not part of LXC). We do have the lxc and lxcfs packages installed, but they do not include thelxccommand.I tried commenting out bash and adding a line about systemd to my /home/notoles Debian LXC config.
The result was these errors:
If there is no way to get Alpine's LXC without LXD to run an unprivileged container for a systemd OS, then, yes, maybe we should install LXD.
I looked at https://wiki.alpinelinux.org/wiki/LXC. This page suggests stopping and disabling systemd-networkd inside a privileged container:
I will check to see if starting the unprivileged container with bash and then stopping/disabling systemd-networkd is enough to get an unprivileged Debian container to start in the usual way, without setting PID 1 as bash.
By the way, where / how did you get your nix LXC image? Nix doesn't seem to be on the linuxcontainers.org image server.
MetalVPS
@yoursunny seems pretty sure that systemd won't work inside the LXC containers:
@terrorgen Did you try changing the entry process to bash for your nix container?
Do you guys understand how/why making unprivileged containers with LXD as explained in https://wiki.alpinelinux.org/wiki/LXD could enable systemd to work inside those containers when getting systemd working might not be possible inside unprivileged containers made with
lxc-create? What's the difference between LXD'slxccommand and LXC'slxc-createcommand that allows systemd to work with unprivileged LXD containers but not with unprivileged LXC containers?MetalVPS
But
/etc/conf.d/lxcdoes exist, so it may not have to do with LXD.That won't work. NixOS is heavily dependant on systemd to get things working. Also, LXC "boots" the container by starting
/sbin/initinside the container. In a NixOS container this is actually a generated bash script, which among other things contains the system configuration that will be activated at "boot". My init script is throwing errors because it cannot find a propersysmount.My understanding is, LXD's
lxccommand is a higher level management tool that abstracts away the lower level configuration oflxc-*. It can also manage VMs. I don't claim to be an expert for either, though.Found a guide that downloads the rootfs tarball from NixOS's own repository. NixOS developers decided against publishing them to linuxcontainers.org because it seems counterintuitive.
I personally have a NixOS LXC container running in my homelab Proxmox server, so I know it can be done.
The all seeing eye sees everything...
So it just clicked that most of my problems (besides cgroup delegation) is permissions:
Comparing notes between my homelab NixOS container vs MetalVPS's...
rootfs permission in my homelab is set to uid:100000 and gid:100000, which maps to root:root in the container.
Whereas my rootfs' permission in MetalVPS is set to my uid:gid.
so I did a tweak in my
configfile:so the container
rootis myself in MetalVPS.solved most of the permission issues above.
now this is how it looks like:
The all seeing eye sees everything...
Congrats on figuring out the permissions issues!
Looks just like the errors I am seeing when starting a Debian container:
@terrorgen I will look at this some more, including your suggested changes to the node configuration and raising the number of containers permission limit.
Now that you fixed your user permissions, might starting with bash as PID 1 work? And be helpful?
MetalVPS
it may boot successfully but because systemd is heavily relied upon by NixOS, it won't be any useful.
I am sorry if I am not being helpful.
The all seeing eye sees everything...
You are always very helpful!
I meant: "Now that you fixed your user permissions, might starting with bash as PID 1 work? And be helpful to you?"
MetalVPS
seems like there is another solution:
https://discuss.linuxcontainers.org/t/failed-to-mount-cgroup-at-sys-fs-cgroup-systemd-operation-not-permitted/13646
The all seeing eye sees everything...
Do we need to
apk add lxdandapk add dbusplus enable both before rebooting, or is it worth while to try just adding lxc and lxcfs to runlevel default?MetalVPS
Okay, at the link you posted @stgraber says:
I'd want to read about making cgroups, but it looks like we could do it. Do you want to go ahead with cgroups, or stay awhile longer on the present path? Should we (1) go ahead and reboot, (2) add lxd and dbus and then reboot, (3) revert the changes I just made to /etc/conf.d/lxc and the rc scripts, (4) work on the cgroups, or (5) some combination?
MetalVPS
Cgroup comparison between fmt (Alpine) and ex100 (Debian sid):
MetalVPS
Let's go ahead and reboot and see if it works! If it doesn't after the reboot, we'll try something else
The all seeing eye sees everything...
Hi @terrorgen!
Reboot:
Following the reboot:
MetalVPS
We got progress!
same error message here.
The all seeing eye sees everything...
Welp, for whatever it's worth:
Unpriviliged container wont start - Failed to allocate manager object
Same errors in Arch Linux:
Solution proposed as:
Here is our /etc/pam.d/system-login:
We don't have a pam_cgfs line in our /etc/pam.d/system-login.
I found a Debian libpam-cgfs package and a Github repo for pam_cgfs.c which seems to be part of LXC. Apparently, Alpine might not have the separate package.
Maybe I could try adding the suggested pam_cgfs line, but I don't understand it enough yet. Ideas?
MetalVPS
Worth a shot.
The all seeing eye sees everything...
I tried adding the suggested line to /etc/pam.d/system-login and then rebooting.
That addition produced these errors, which seem to be the same:
So I reverted the change and rebooted. Restarted the networking and ndpresponder.
MetalVPS
I logged in and nobody was around. So. . . .
MetalVPS
Now that I blew my budget on the i9-12900K EX100 I got from Hetzner I regretfully have had to ask @Cloudie to cancel this server. The next payment is due March 3. I don't know exactly when the cancellation will occur.
@yoursunny @terrorgen @subenhon You guys have been using the fmt server lately. I don't know whether @Cloudie would allow it, and I don't know what the price might be, but perhaps one or the group of you might want to take over the server? You could try contacting @Cloudie, perhaps via Route48's Discord.
Guys who have been using this server are warmly invited to request accounts in the above linked EX100 thread. I realize the EX100 is in Finland instead of California, and that makes a big latency difference. Another big difference is that the Helsinki server runs Debian sid and not Alpine.
I hope the EX100 will stay around for awhile! I'm not planning to cancel it. I'm looking forward to seeing you guys in Helsinki!
MetalVPS
😬
At least the other server has systemd.
Webhosting24 aff best VPS; ServerFactory aff best VDS; Cloudie best ASN; Huel aff best brotein.
https://busybox.net/kill_it_with_fire.txt
MetalVPS
Can you add me too
Teehee!
@Nubuki said:
Let's please get this server cleaned up and back to @Cloudie. When I have a chance, I will try adding you on the EX100 if that's okay with you.
MetalVPS
If possible, I would like to get one.
pm key
Congrats on your first post
"A single swap file or partition may be up to 128 MB in size. [...] [I]f you need 256 MB of swap, you can create two 128-MB swap partitions." (M. Welsh & L. Kaufman, Running Linux, 2e, 1996, p. 49)
Hi @zudaz! Sorry, but this server has been returned to @Cloudie. Welcome to LES! If you want to post a little about who and where you are and about what you are doing, I am sure everybody would be interested. It's a nice group here at LES! Best wishes! Tom
MetalVPS