Very high CPU usage in random moments, VPS goes down, can't find why
I have a small Hetzner VPS all with Docker containers running. My normal usage is fine. 1.7 GB Ram of 2.0 GB are in use (with buffers), uptime reports sane values all 3 around 0.60-0.90 (2 cores). Works pretty well.
In what I believe are random moments of the day, I receive a Hetrixtools notification that my stuff is down, and it is. Can't even login. In case it happens when I am logged in, I can't even give a command.
When it's over, uptime reports insane values, like 49.6 38.5 22.9 or something.
The issue: htop does not report anything out of the ordinary, docker stats the same.
All I see is typical processes which are using 2-10% of the CPU and small spikes of the usual suspects (a mysql process using 60% for a second or so).
Hetrixtools CPU monitor is even flatter. This is the last 24 hours according to it. 
It does not make sense. I can't see what it is when it happens, and when I can see, nothing looks weird.
Anything else I could look at? It's even happening as I am writing this post now.
Edit: this looks ridicolous
To add to this, I am using Ubuntu 20.04, ufw enabled.
Comments
Your iowait seems pretty high. Make sure you have enough memory and not swapping.
Nexus Bytes Ryzen Powered NVMe VPS | NYC|Miami|LA|London|Netherlands| Singapore|Tokyo
Storage VPS | LiteSpeed Powered Web Hosting + SSH access | Switcher Special |
@seriesn thanks for trying! 0 swap in use
I am 85% sure it's one of the containers. Will update this thread in case I find the issue.
Seems like no swap available ( free = 0 in the screenshot ).
Maybe having some swap might be helpful?
https://chrisdown.name/2018/01/02/in-defence-of-swap.html
https://news.ycombinator.com/item?id=23455051
Good luck! Best wishes from Mexico!
MetalVPS
@Not_Oles said:
Oh jeez, you (and @seriesn by proxy) are actually right, thanks! But this is Hetzner default Ubuntu 20.04 installation.
Was swap removed from Ubuntu installation at some point, and I missed it? Is it Hetzner image that does not set one?
I'll add one for sure!
Looks like an issue running out of memory given how little is free, add swap as others suggested, it might well help.
Clouveo - SSD/NVMe Cloud VPS & Hosting
Performance & AMD EPYC Cloud | DDoS Protected | Snapshots | Auto Backups
Most kvm templates these days doesn’t include swap by default. But yeah, usually OOM would be the culprit.
I would also try to trace whatever is using up your memory and work my way back from there. Who knows, maybe you need to upgrade your plan?
Nexus Bytes Ryzen Powered NVMe VPS | NYC|Miami|LA|London|Netherlands| Singapore|Tokyo
Storage VPS | LiteSpeed Powered Web Hosting + SSH access | Switcher Special |
Sorry, I don't know the current default install status of swap in Ubuntu generally or on VPSes at Hetzner. However, I recently installed Hetzner's default Ubuntu 20.04 image on a bare metal server there. top(1) on that machine shows:
MetalVPS
Agreed!
@seriesn is excellent for proxy!
But @seriesn also is far better than excellent for VPN
MetalVPS
Ubuntu & docker - arrgh!
KISS. Just saying.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I know, I know. But free livepatch!
Alright, I finished my investigation. It was the missing SWAP / insufficient RAM. What sparkled it was however a container, as I suspected: CodiMD (a.k.a. HackMD), something is happening that makes its MySQL usage in RAM grow while idling. This caused the out of RAM issue and the missing swap was, of course, what caused the VPS to hang.
Thanks everyone for your help!
Second thing: I span up a fresh new Hetzner VPS with Ubuntu 20.04. I confirm that swap is not configured. I have never seen this before!
SSD/NVMe? Likely to stop folks thrashing them, when vm.swapinness is set at silly 60 default - set to 0 or 1.
Anyway: Ubuntu for desktop; debian/CentOS for server.
(I have seen other templates set with no swap but never use provider's templates when commissioning a server. Grab the network settings (where required) and use a minimal/network ISO.)
Personally, i usually set 1GB swap for 2GB RAM -if it needs more than that then something's up: insufficient resources/poorly tuned or runaway processes (hello, clamd!).
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)