@Not_Oles said:
Hello! The server has been rebooted into the Hetzner rescue system. Nobody can login now. Apologies! More details soon.
What happened? Did it kernel panick?
Guys!
Hetzner's careful monitoring caught an outbound port scan coming from our i9-13900 server. Since we already were planning to reinstall the server, it seemed like the abuse issue could be most definitively and immediately resolved by taking the server offline. So I activated the rescue system.
To make sure that everyone understands, it was my decision to take the server offline. Not Hetzner's. Hetzner did not interfere with our server in any way. All Hetzner did was to send me their helpful notifiction. I am really grateful to Hetzner for doing a great job, not only with monitoring, but with everything else which could be expected from a top provider! Thank you Hetzner!
Please note that only the i9-13900 is offline. The i9-9900K still is up and functioning normally, as of a moment ago when I checked.
Best wishes and kindest regards,
Tom
Thanks for all that you do for the Les community @Not_Oles !
Um I've got a question , google didn't really help much so here I am:
What's port scanning and why is it considered bad/abuse by hosting providers?
@Not_Oles said:
Hello! The server has been rebooted into the Hetzner rescue system. Nobody can login now. Apologies! More details soon.
What happened? Did it kernel panick?
Guys!
Hetzner's careful monitoring caught an outbound port scan coming from our i9-13900 server. Since we already were planning to reinstall the server, it seemed like the abuse issue could be most definitively and immediately resolved by taking the server offline. So I activated the rescue system.
To make sure that everyone understands, it was my decision to take the server offline. Not Hetzner's. Hetzner did not interfere with our server in any way. All Hetzner did was to send me their helpful notifiction. I am really grateful to Hetzner for doing a great job, not only with monitoring, but with everything else which could be expected from a top provider! Thank you Hetzner!
Please note that only the i9-13900 is offline. The i9-9900K still is up and functioning normally, as of a moment ago when I checked.
Best wishes and kindest regards,
Tom
Thanks for all that you do for the Les community @Not_Oles !
Thank you @Nubuki! Your kind words are much appreciated!
Um I've got a question , google didn't really help much so here I am:
What's port scanning and why is it considered bad/abuse by hosting providers?
@SheGivMeItAgain said:
Alright! I'll wait then. Also apologies for rushing you, I was in the middle doing something and it stopped and I am not sure if I lost data or anything.
Anything on MetalVPS should have a backup. EXPECT data loss. Read the first post.
Please make your own redundant, offsite backups! It's easy to download or sync or clone your backup to a safe place. Please also make sure that you actually can restore from your backups! Please think of your MetalVPS account as ephemeral! It might blow up! We or you might reinstall the Node! 🤦♂️
@Not_Oles said:
Hetzner's careful monitoring caught an outbound port scan coming from our i9-13900 server. Since we already were planning to reinstall the server, it seemed like the abuse issue could be most definitively and immediately resolved by taking the server offline. So I activated the rescue system.
This is why we can't have nice things... Abusing free services and causing problem for others...
@Nubuki said:
Um I've got a question , google didn't really help much so here I am:
What's port scanning and why is it considered bad/abuse by hosting providers?
Port scanning is used as a way to see which ports are active on the servers around the world. Then you can mount a target attack to the server with open ports. It's like making a list of targets for hacking. So port scanning is considered abuse, cause you are helping the hackers update the list of targets.
@SheGivMeItAgain said:
Alright! I'll wait then. Also apologies for rushing you, I was in the middle doing something and it stopped and I am not sure if I lost data or anything.
Anything on MetalVPS should have a backup. EXPECT data loss. Read the first post.
Please make your own redundant, offsite backups! It's easy to download or sync or clone your backup to a safe place. Please also make sure that you actually can restore from your backups! Please think of your MetalVPS account as ephemeral! It might blow up! We or you might reinstall the Node! 🤦♂️
@Not_Oles said:
I really appreciate you! Thanks so much again for the monitoring help!
Hey, if you are paying for the server, letting the members here use it for free and going through the trouble to setting things up and reinstalling OS, me just monitoring is not a big deal.
@Not_Oles said:
I'm looking forward to reinstalling the server and restarting it. A big question is how should the server be configured in the reinstall?
I want apache user directories OR nginx installed with user access to the /etc/nginx/sites-available and /etc/nginx/sites-enabled directories.
I never used the VMs but did notice no one used LXC, so maybe can skip it? Can skip the KVM too if you want
If you do setup KVM, set up limits on RAM and enable quotas on linux user directories pls: https://linuxhint.com/disk_quota_ubuntu/
Maybe docker? Not sure anyone here is familier with it, so if not, no need to go through the extra trouble of adding user to both KVM and docker usergroups...
Also I would recommend proxmox, but then it wont be a metal VPS anymore, would it?
@Not_Oles said:
Another big question is how should requests to use the server be filtered?
Set it by the number of posts and age of account? So minimum XX posts over X months?
Hello, I was really shocked when I saw the news about the server stopping because of the ports. Yes, this is very common in servers. The best solution is not to allow the ports to be opened and to make a one virtual Windows or Ubuntu device and put all the users inside
How should the upcoming, refreshed i9-13900 server be set up?
is it too crazy to imagine that some guys might want to pay something toward the € 89 monthly cost of renting the server from Hetzner?
Thanks for your ideas!
Friendly greetings!
Tom
maybe lets try to not give everyone access to the server, i barely have used it except for the times i set up vms, and the suggestion i had of maybe having admins or people monitor the server and terminate any process that might be a resource hog or port scanner
How should the upcoming, refreshed i9-13900 server be set up?
is it too crazy to imagine that some guys might want to pay something toward the € 89 monthly cost of renting the server from Hetzner?
Thanks for your ideas!
Friendly greetings!
Tom
maybe lets try to not give everyone access to the server, i barely have used it except for the times i set up vms, and the suggestion i had of maybe having admins or people monitor the server and terminate any process that might be a resource hog or port scanner
donations could help out and all
Or could go with a refundable security deposit. It'll limit abusers to some extent as they mainly focus on free content.
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
@terrorgen said:
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
I saw 0 users of LXC. Only those qemu windows users.
@Not_Oles said:
Hello! The server has been rebooted into the Hetzner rescue system. Nobody can login now. Apologies! More details soon.
What happened? Did it kernel panick?
Hetzner's careful monitoring caught an outbound port scan coming from our i9-13900 server. Since we already were planning to reinstall the server, it seemed like the abuse issue could be most definitively and immediately resolved by taking the server offline. So I activated the rescue system.
To make sure that everyone understand, it was my decision to take the server offline. Not Hetzner's. Hetzner did not interfere with our server in any way. All Hetzner did was to send me their helpful notifiction. I am really grateful to Hetzner for doing a great job, not only with monitoring, but with everything else which could be expected from a top provider! Thank you Hetzner!
Please note that only the i9-13900 is offline. The i9-9900K still is up and functioning normally, as of last time I checked.
I take your request as a sign of continued confidence in MetalVPS.
I wish I could immediately add you to the i9-9900K. However, it will be a little while before new account creation is resumed.
Best wishes and kindest regards,
Tom
Alright! I'll wait then. Also apologies for rushing you, I was in the middle doing something and it stopped and I am not sure if I lost data or anything.
Hope all is okay. I wish we had real time communication with everyone on the server.
@terrorgen said:
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
I saw 0 users of LXC. Only those qemu windows users.
Meaning MetalVPS is not attracting people who would like to use LXC. I wonder why /s.
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
@terrorgen said:
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
I saw 0 users of LXC. Only those qemu windows users.
Meaning MetalVPS is not attracting people who would like to use LXC. I wonder why /s.
My guess is that not too many people want to use LXC. The low level of interest in LXC at MetalVPS might be as much or more about LXC as about MetalVPS. 🐱
@terrorgen said:
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
I saw 0 users of LXC. Only those qemu windows users.
Meaning MetalVPS is not attracting people who would like to use LXC. I wonder why /s.
My guess is that not too many people want to use LXC. The low level of interest in LXC at MetalVPS might be as much or more about LXC as about MetalVPS. 🐱
Respectfully disagree. Remember a few of us were trying hard to get LXC working on fmt.
@Not_Oles said:
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
It's good to hear that Hetzner is accepting your statement, that means MetalVPS will probably be back soon.
@terrorgen said:
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
I saw 0 users of LXC. Only those qemu windows users.
Meaning MetalVPS is not attracting people who would like to use LXC. I wonder why /s.
My guess is that not too many people want to use LXC. The low level of interest in LXC at MetalVPS might be as much or more about LXC as about MetalVPS. 🐱
Respectfully disagree. Remember a few of us were trying hard to get LXC working on fmt.
Hmm. Yep! I remember. So . . . what you are saying is that MetalVPS is unattractive to LXC fans because there were no limits on the "qemu windows users?" I think it's important that MetalVPS be attractive to someone like you. How can we make MetalVPS attractive to you?
@Not_Oles said:
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
Does this mean it will back on today? Or...? I don't understand how hetzner support team works. Sorry!
@Not_Oles said:
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
It's good to hear that Hetzner is accepting your statement, that means MetalVPS will probably be back soon.
Thanks! Yes.
@Not_Oles said: A big question is how should the server be configured in the reinstall?
Got what @somik said above about adding a web server. Also what @terrorgen said about LXC. Anything else?
Another big question is how should requests to use the server be filtered?
No filtering?
Identity, location verification?
LES account maturity, posting activity?
Other website activity?
Payment verification (charge a fee, then refund, like Oracle Free Tier)?
The TL;DR version: let's get MetalVPS back to fmt and days before.
I missed the fmt days where people who use the resources are actively contributing back to the community by sharing what they are doing on the machine.
The current incarnation seems to attract people who are hiding behind AI generated language and secretive on what they want to do, and only when poked multiple times, provided a vague "I want to learn running VMs and LXC", where in fact, they are running Windoze and nested VMs (why?) and playing games on it.
And the excuses about "my computer is too slow to run such kind of things" I call BS. Unless you have a chromebook (no offence to you, Tom), computers made in the last 15 years is capable of running VMs. My homelab server was on an Intel Core 2 Q6600 at one point.
So, if you make the claim that your computer is unable to run VMs, be prepared to prove it.
Not saying we shouldn't let people run Windoze on a VM on MetalVPS. I think we should be vetting people, and place some limits, especially when you are offering your money and time.
You, Tom, is exceptional and generous enough to provide this to the community. I think I just don't want to see you and the resources you offered getting abused. Yet we already have one here. Someone running a port scanner.
@Not_Oles said:
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
Does this mean it will back on today? Or...? I don't understand how hetzner support team works. Sorry!
Welp, all is okay with Hetzner. Thank you, Hetzner!
At present, we don't know which user sent off the evil packets. So I don''t want to just turn the server back on and permit whoever it was to continue. Probably there is a way we could monitor for and detect a repeat. We might even be able to find out who did it. But the monitoring and the archaeology might be a little above my pay grade at the moment.
Maybe the server will be back up today. If not today, then 🔜
You said above that you had no ideas for how the refreshed server should be configured. And there is the additional question of how accounts should be filtered. We can't just add all the old accounts to the refreshed server, because, then, we'd be letting the evil guy back in. I don't think he is you, but I suppose it's not impossible. Maybe you could think carefully about the configuration and the filtration issues and contribute your ideas. Thanks very much! Best wishes!
@terrorgen said:
The TL;DR version: let's get MetalVPS back to fmt and days before.
I missed the fmt days where people who use the resources are actively contributing back to the community by sharing what they are doing on the machine.
The current incarnation seems to attract people who are hiding behind AI generated language and secretive on what they want to do, and only when poked multiple times, provided a vague "I want to learn running VMs and LXC", where in fact, they are running Windoze and nested VMs (why?) and playing games on it.
And the excuses about "my computer is too slow to run such kind of things" I call BS. Unless you have a chromebook (no offence to you, Tom), computers made in the last 15 years is capable of running VMs. My homelab server was on an Intel Core 2 Q6600 at one point.
So, if you make the claim that your computer is unable to run VMs, be prepared to prove it.
Not saying we shouldn't let people run Windoze on a VM on MetalVPS. I think we should be vetting people, and place some limits, especially when you are offering your money and time.
You, Tom, is exceptional and generous enough to provide this to the community. I think I just don't want to see you and the resources you offered getting abused. Yet we already have one here. Someone running a port scanner.
Thank you for reading my rant.
Yeah I feel like this should be shared. I don't have a PC myself, as I physically can't afford one for reasons I do not feel comfortable sharing. However, people like Tom give me the opportunity to learn VMs, like nested virtualization, KVM sharing, etc. I came here to make a VM that I can actually use Visual Studio is as all I have is a Xbox and remote in PCs that I can use. It also gives me chance to learn more about coding which I am passionate about.
People not sharing what they are doing can be quite harmful to the server as for all we know, they could be making a VM to ddos someone! Or maybe even making a botnet. But who knows, people should speak out about what their intentions are!
With this Tom can be at rest without constantly worrying about what's going on in the server. If anything, people are more worried as their data could be gone quicker than I count if the wrong person gets hold of the server! Especially proot can be a MAJOR problem as it gives them fake root users by messing with Linux files that are accessable to the norm guest user. It's amazing what people have achieved these days especially with Linux and viruses. But this shouldn't have to worry anyone here! We all should be friends and do our own stuff.
@Not_Oles said:
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
Does this mean it will back on today? Or...? I don't understand how hetzner support team works. Sorry!
Welp, all is okay with Hetzner. Thank you, Hetzner!
At present, we don't know which user sent off the evil packets. So I don''t want to just turn the server back on and permit whoever it was to continue. Probably there is a way we could monitor for and detect a repeat. We might even be able to find out who did it. But the monitoring and the archaeology might be a little above my pay grade at the moment.
Maybe the server will be back up today. If not today, then 🔜
You said above that you had no ideas for how the refreshed server should be configured. And there is the additional question of how accounts should be filtered. We can't just add all the old accounts to the refreshed server, because, then, we'd be letting the evil guy back in. I don't think he is you, but I suppose it's not impossible. Maybe you could think carefully about the configuration and the filtration issues and contribute your ideas. Thanks very much! Best wishes!
I have thought about it, and thanks to @terrorgen, I have thought of a idea.
Here is what I roughly thought about the filtering issue:
Limiting how many users should even be allowed on the server
Asking them to prove "Their PCs CANT run VMs what so ever".
Make them to thoroughly explain what their uses are for the server.
Maybe limit network usage from users to prevent DDoS attacks/port scans originating from the server itself.
Monitor the users home dir by checking for any malicious files, and if possibly running a small VM to check .qcow2/.img files.
Edit 1:
I have also thought about the configuration of the server:
Limit network outbound and possibly incoming to prevent port scans, DDoS attacks, etc.
Possibly try and find a open source AI to prevent malicious apps, like miners, etc. (Like NVIDIA GeForce Now's UADML which prevents exploits, etc)
Hopefully this helps! If I think of anything else I'll let you know ASAP!
@Not_Oles said: @SheGivMeItAgain Blake! Thanks for your careful comments on the important issues! I will take time to give them a good read. Thanks again! Tom
Your welcome! Just let me know if you do end up putting them in place!
Comments
Thanks for all that you do for the Les community @Not_Oles !
Um I've got a question , google didn't really help much so here I am:
What's port scanning and why is it considered bad/abuse by hosting providers?
Teehee!
Thank you @Nubuki! Your kind words are much appreciated!
Maybe take a look at https://en.wikipedia.org/wiki/Port_scanner
Best wishes!
Tom
MetalVPS
@Not_Oles Can I still get unsuspended account in MetalVPS after reinstallation?
Anything on MetalVPS should have a backup. EXPECT data loss. Read the first post.
Somik.org - Server admins cheat codes
This is why we can't have nice things... Abusing free services and causing problem for others...
Port scanning is used as a way to see which ports are active on the servers around the world. Then you can mount a target attack to the server with open ports. It's like making a list of targets for hacking. So port scanning is considered abuse, cause you are helping the hackers update the list of targets.
Somik.org - Server admins cheat codes
Hi @somik !
I really appreciate you! Thanks so much again for the monitoring help!
I'm looking forward to reinstalling the server and restarting it. A big question is how should the server be configured in the reinstall?
Another big question is how should requests to use the server be filtered?
I appreciate all you guys who have joined as neighbors. Especially those who have commented helpfully here in this thread!
Best wishes!
Tom
MetalVPS
Hey, if you are paying for the server, letting the members here use it for free and going through the trouble to setting things up and reinstalling OS, me just monitoring is not a big deal.
I want apache user directories OR nginx installed with user access to the
/etc/nginx/sites-availableand/etc/nginx/sites-enableddirectories.I never used the VMs but did notice no one used LXC, so maybe can skip it? Can skip the KVM too if you want
If you do setup KVM, set up limits on RAM and enable quotas on linux user directories pls: https://linuxhint.com/disk_quota_ubuntu/
Maybe docker? Not sure anyone here is familier with it, so if not, no need to go through the extra trouble of adding user to both KVM and docker usergroups...
Also I would recommend proxmox, but then it wont be a metal VPS anymore, would it?
Set it by the number of posts and age of account? So minimum XX posts over X months?
Somik.org - Server admins cheat codes
Hello, I was really shocked when I saw the news about the server stopping because of the ports. Yes, this is very common in servers. The best solution is not to allow the ports to be opened and to make a one virtual Windows or Ubuntu device and put all the users inside
http://dough.h4ck.me/ - we gonna get out of mcdonalds with this one!!!
Or could go with a refundable security deposit. It'll limit abusers to some extent as they mainly focus on free content.
Somik.org - Server admins cheat codes
I think LXC still has its place in MetalVPS, just need to make it attractive enough for the right people.
To make it worthwhile for Tom and others who have a serious use of the machine, limits need to be set.
The all seeing eye sees everything...
I saw 0 users of LXC. Only those qemu windows users.
Somik.org - Server admins cheat codes
A sad day. I was going to suggest
ytalk, but it looks like it was pulled from debian in 2019: https://packages.qa.debian.org/y/ytalk.htmlMeaning MetalVPS is not attracting people who would like to use LXC. I wonder why /s.
The all seeing eye sees everything...
Good morning! I just woke up from a good night's sleep to find an email from Hetzner saying that the statement I sent them regarding the port scan issue was accepted and the matter was closed. Yaaay! 🎉
MetalVPS
My guess is that not too many people want to use LXC. The low level of interest in LXC at MetalVPS might be as much or more about LXC as about MetalVPS. 🐱
MetalVPS
Later today or soon I will go through yesterday's posts in this thread. There are a few to which I haven't responded yet. Thanks! 🙏
MetalVPS
Respectfully disagree. Remember a few of us were trying hard to get LXC working on fmt.
The all seeing eye sees everything...
It's good to hear that Hetzner is accepting your statement, that means MetalVPS will probably be back soon.
Hmm. Yep! I remember. So . . . what you are saying is that MetalVPS is unattractive to LXC fans because there were no limits on the "qemu windows users?" I think it's important that MetalVPS be attractive to someone like you. How can we make MetalVPS attractive to you?
MetalVPS
Does this mean it will back on today? Or...? I don't understand how hetzner support team works. Sorry!
Thanks! Yes.
Got what @somik said above about adding a web server. Also what @terrorgen said about LXC. Anything else?
MetalVPS
The TL;DR version: let's get MetalVPS back to fmt and days before.
I missed the fmt days where people who use the resources are actively contributing back to the community by sharing what they are doing on the machine.
The current incarnation seems to attract people who are hiding behind AI generated language and secretive on what they want to do, and only when poked multiple times, provided a vague "I want to learn running VMs and LXC", where in fact, they are running Windoze and nested VMs (why?) and playing games on it.
And the excuses about "my computer is too slow to run such kind of things" I call BS. Unless you have a chromebook (no offence to you, Tom), computers made in the last 15 years is capable of running VMs. My homelab server was on an Intel Core 2 Q6600 at one point.
So, if you make the claim that your computer is unable to run VMs, be prepared to prove it.
Not saying we shouldn't let people run Windoze on a VM on MetalVPS. I think we should be vetting people, and place some limits, especially when you are offering your money and time.
You, Tom, is exceptional and generous enough to provide this to the community. I think I just don't want to see you and the resources you offered getting abused. Yet we already have one here. Someone running a port scanner.
Thank you for reading my rant.
The all seeing eye sees everything...
Welp, all is okay with Hetzner. Thank you, Hetzner!
At present, we don't know which user sent off the evil packets. So I don''t want to just turn the server back on and permit whoever it was to continue. Probably there is a way we could monitor for and detect a repeat. We might even be able to find out who did it. But the monitoring and the archaeology might be a little above my pay grade at the moment.
Maybe the server will be back up today. If not today, then 🔜
You said above that you had no ideas for how the refreshed server should be configured. And there is the additional question of how accounts should be filtered. We can't just add all the old accounts to the refreshed server, because, then, we'd be letting the evil guy back in. I don't think he is you, but I suppose it's not impossible.
Maybe you could think carefully about the configuration and the filtration issues and contribute your ideas. Thanks very much! Best wishes! 
MetalVPS
Yeah I feel like this should be shared. I don't have a PC myself, as I physically can't afford one for reasons I do not feel comfortable sharing. However, people like Tom give me the opportunity to learn VMs, like nested virtualization, KVM sharing, etc. I came here to make a VM that I can actually use Visual Studio is as all I have is a Xbox and remote in PCs that I can use. It also gives me chance to learn more about coding which I am passionate about.
People not sharing what they are doing can be quite harmful to the server as for all we know, they could be making a VM to ddos someone! Or maybe even making a botnet. But who knows, people should speak out about what their intentions are!
With this Tom can be at rest without constantly worrying about what's going on in the server. If anything, people are more worried as their data could be gone quicker than I count if the wrong person gets hold of the server! Especially proot can be a MAJOR problem as it gives them fake root users by messing with Linux files that are accessable to the norm guest user. It's amazing what people have achieved these days especially with Linux and viruses. But this shouldn't have to worry anyone here! We all should be friends and do our own stuff.
I have thought about it, and thanks to @terrorgen, I have thought of a idea.
Here is what I roughly thought about the filtering issue:
Edit 1:
I have also thought about the configuration of the server:
Hopefully this helps! If I think of anything else I'll let you know ASAP!
Thanks,
Blake
@SheGivMeItAgain Blake! Thanks for your careful comments on the important issues! I will take time to give them a good read. Thanks again! Tom
MetalVPS
Your welcome! Just let me know if you do end up putting them in place!
Guys!
Sadly, I have to report that the i9-9900K server has been shut down due to information which came to my attention.
Best wishes!
Tom
MetalVPS
My messages just getting ignored...