Kubernetes or else on low end hardware?

Robotex

Hello,
I'd like to try setting up a HA cluster for learning purposes, however I'm not sure about the choice of software to use as base. I tried already once, a year ago, to create a Kubernetes cluster using Rancher on top of Debian on a VPS with 2 vCPU and 1GB RAM but the provider shut it off after a few hours for IOPS overload (probably memory swapping?), despite it was only an etcd agent node together with an another VPS as control plane without any service running nor created. For that reason, I had given up never booting it up again and left those VPS idle for the remaining year, believing that Kubernetes had a too high minimum hardware requirements to run.
Now that I got my hands on a VPS with double the resources and a dedicated from kimsufi, I'd like to try again and hopefully run something on it. My idea is to run a static website, a mesh vpn to secure the access, a load balancer and maybe something more serious.
So here are my questions:
1. I read on Google that people are able to create a cluster using only raspberry pis, how is that feasible? Because I own a rPI 3 and AFAIK it has only 1GB of RAM like my previous VPS, so if my previous VPS didn't have enough memory for Kubernetes, how could a few rPI even run anything on it?
2. Is there some way to reduce the resources needed for Kubernetes alone? Because so far, it seems to me that only people with big hardware and €€€ can learn this stuff, since from what I read it requires atleast 3 nodes to set up something that isn't just a toy far from reality.
3. What would be the best method to deploy Kubernetes on a VPS or bare metal server? I know that I can use Ansible to configure and install the required packages, however I heard also about distros like CoreOS, RancherOS, k3OS that are made specifically for running containers, though almost no provider has them available as template. Somehow I was able to mount succesfully a Live ISO of CoreOS (writing a iPXE script, setting boot media, adding ssh key etc) but a lot of stuff, especially configurations, still seems to me like reading arab. My ideal final situation would be that I would just need to press some buttons to install/deploy/upgrade.
4. Does Kubernetes allow to deploy a container only to a specific node? Let's say that I'd like to run a Minecraft server on Kimsufi so that it doesn't overload the lower specs VPS, is that possible?
5. I'm currently giving Portainer a try which uses Docker Swarm and seems to be less eager in RAM usage, so what does Kubernetes give that I cannot do with Portainer?
Thanks in advance to anyone who will be willing to answer my questions

andrej

1) You need 1GB ram or more for workers and 1.5GB ram or more for master. What you will run on it would depend on hardware. What are you trying to do ?
2) Resources are already very minimal. RAM is the biggest problem and if you follow recommended hardware guide, there is no way to reduce it.
3) You need 2 nodes, master and node. All software is available thru packages so that should not be a problem. How to deploy an app in your deployment is something different. Read this for a start https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro/
4) As long as you stick to minimum requirements you will be fine. If you have server with 5 gb ram and you use 3, you will be able to run master on that specific server. I am not sure if that answers your question.
5) They are similar and different. Portainer is a docker orchestration tool. Kubernetes is container management tool. For example you can manage JVM container form Kubernetes. It all depends what do you want to do.

gks

Find a VPS provider who can support dedicated CPUs (vCore), get a system with RAM size 10 GB at least, [PHPFriendz had offer] or other providers like Hetzner.
Don't pick CoreOS, RancherOS, k3OS etc stuffs early stage, that may distract learning Kubernetes.

Docker -> Docker Compose -> Kubernetes on standlone --> Kubernetes on Distributed is good way. Use Simple Micro Services on Kubernetes instead of complex ones. For Example, I recommend Prometheus, Grafana, Node Exporters, InfluxDB, PostgreSQL in different containers on Kubernetes, then expand to Node.js. Running Grafana with HA setup using PostgreSQL as backend will help you scale and check distributed cluster.

The second stage of learning is using Service Discovery by utilizing etcd, or consul will help to discover and auto balance HA applications configurations.

seanho

https://k3s.io/

Kubernetes for the edge. I'm working on an ansible role to install it on top of debian; this is my plan to run it on my fleet of VPSes. Sticking with KVM >1GB for now. K3s uses sqlite instead of etcd, so no HA server yet (they're working on it, using dqlite).

tgl

you can use docker swarm for ha if you have stateless services

the only downside of docker swarm is handling volumes

chimichurri

I agree with seanho - k3s might be a good idea if you want something lightweight.

4. Does Kubernetes allow to deploy a container only to a specific node? Let's say that I'd like to run a Minecraft server on Kimsufi so that it doesn't overload the lower specs VPS, is that possible?
Yes!

seanho

Probably best way to do that is via CPU request

chrisfy

k3s is a great place to start as it is easy to install, full k8s has a lot of moving parts to figure out the first time, so unless you were interested in the sysadminy part of it k3s would be worth it as a starting point

InceptionHosting

Using kubernetes for your cats static blog!

taken from nixcraft: https://www.facebook.com/nixcraft/?tn-str=k*F

chrisfy

1. For learning you don't need a HA cluster, you can do it with whatever you have available. From running the hobby-kube terraform my memory was the master node used 1gb and the workers 512mb, k3s from memory is half that at 512 master /256 worker. Again, don't fret over the HA element if you just want to learn.
2. k3s on a single node
3. hobby-kube terraform script for full kubernetes or k3s - both are very easy
4. Yes possible
5. Kubernetes has more features and flexibility (therefore more complexity), unsure on specifics

stevefan1999

1. It is disastrous. Most images on Docker Hub aren't compatible to aarch64 yet, let alone arm64
2. That's how computer consensus works: you will need to have odd numbers of nodes, and we all try to keep up on sharing the same state. Once we had some kind of disconnections and discontinuity of state (also known as network partition, e.g. misconfigured server, network downtime), whoever had more than 50% of control of the state (logical majority) wins and the lost nodes will then start to follow up to replicate the rest of the majority (also, 1 node is useless, 2 nodes will degrade to 1 node because 50/50, and thus minimum 3 nodes, in other word even number of nodes n will degenerate to the case of nearest odd number n-1), then being "functional part of the society" again (regain vote/quorum). If no group had more than 50% then the entire consensus/state machine will crash and will go read only, to wait for you manually tamper/repair (this is the infamous split-brain situation; you are recommended to recover from a known valid checkpoint, or pretend nothing happened and try to merge the difference). This does not guarantee valid data but is the most democratic ways of sorting things out in a HA system. And democracy is expensive and unnecessarily complicated.
3. k3s is the best all rounder as it already set up some useful application such as traefik and flannel for you. You basically need to open tcp 6443 and tcp 10250 to get communication working. I use WireGuard on Flannel so I open udp 51820 as well
4. You can do it in three ways: Node Selector, Node Affinity and Taints/Tolerance model. Node Selector will pick only one specific node, Node Affinity will match the label and also gives you more freedom (using logical conditions, so picking a node with a label of High Performance AND NOT Busy, is possible, you can also pick preference to allow some general purpose node to host game server if they are idling too). Node taint/tolerance is an extremist approach that no pod with a tolerance to a certain taint can schedule/execute on that particular node with it. To think about how to use the taints, you can say NoExecute is basically like a DMZ where only soldiers can enter. To think about NoSchedule, think about that a country had martial law in effect and thus closed its border, and only existing citizen can enter. For PreferNoSchedule, you can think a country allowed some foreigners that passed COVID-19 self-isolation check can enter
5. Portainer hasn't updated for a long time and it is limited to Docker and Docker Swarm only. Docker Swarm is the dead competitor to Kubernetes. Portainer thus is also quite dead.

ehab

@stevefan1999 said:

4- i liked how you connected scheduling with real life experince.

5- Docker is not dead in fact some are going back from k8s to swarm.

stevefan1999

@ehab said:

@stevefan1999 said:

4- i liked how you connected scheduling with real life experince.

5- Docker is not dead in fact some are going back from k8s to swarm.

I never said Docker is dead. But Docker Swarm, which is a component of Docker, had blowflies already.

seanho

I agree that k8s is the way forward instead of swarm, but there are still a lot of swarm installations out there.

Just a small correction: no pod without a matching tolerance is scheduled on nodes with a given taint. The terminology is super confusing.

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

akhfa

1. I think about 2 GB of ram for master, and yes 1 GB RAM for nodes (I use some node with this spec to only receive traffic)
2. Try to use k3s. It can be set as HA with external mysql or postgres as etcd replacement with less resources than fully featured kubernetes
3. best method, imo, just use rancher, if you have generous resources. You can install it on top of docker, no need to use rancherOS if you are not familiar.
4. yes
5. Many, I think. Never use docker swarm before. Just use kubernetes for the container orchestrasion. I think even Nomad is still better than swarm in term of long term community support.

akhfa

seems like necro posting, from when k3s doesn't support HA, until now it do support

seanho

K3s 1.19 has now reverted to using etcd just like upstream kubernetes. For a long while they didn't have HA, then they tried dqlite, then just gave in to using etcd. You can still use external DB if you like.

ehab

@seanho said:
K3s 1.19 has now reverted to using etcd just like upstream kubernetes. For a long while they didn't have HA, then they tried dqlite, then just gave in to using etcd. You can still use external DB if you like.

did you install it? what is the new size?

seanho

I'm running k3s 1.19 on my home lab, yes. I haven't gotten to the point of building a cluster of VPSes yet (but I plan to). On home lab I have 128GB of RAM per node and plenty of disk, and so haven't paid much attention to install size.

ehab

when you have 128GB of RAM per node and plenty of disk ..... you have all the space in the world unless your building for space

AaronSS

Anyone used this before? https://microk8s.io/high-availability

havoc

@aaronstuder said:
Anyone used this before? https://microk8s.io/high-availability

No but gonna check it out soon. Bought a 3rd raspberry for the purpose.