It seems the server might have booted about four hours ago. What happened? When? Why?
root@Proxmox-VE ~ # date
Fri 20 Jan 2023 05:39:09 AM UTC
root@Proxmox-VE ~ #
From /var/log/messages:
Jan 19 16:13:03 Proxmox-VE kernel: [552651.050197] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/ZAAOPBF6QOV76SOU4CURM2ZQUD' does not support file handles, falling
back to xino=off.
Jan 19 16:13:03 Proxmox-VE kernel: [552651.075135] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/OO54Z7VIVSLWLVUU2TJPBYRHHU' does not support file handles, falling
back to xino=off.
Jan 19 16:13:03 Proxmox-VE kernel: [552651.101942] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/OO54Z7VIVSLWLVUU2TJPBYRHHU' does not support file handles, falling
back to xino=off.
Jan 19 16:13:04 Proxmox-VE kernel: [552651.978316] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/ZAAOPBF6QOV76SOU4CURM2ZQUD' does not support file handles, falling
back to xino=off.
Jan 19 16:13:04 Proxmox-VE kernel: [552652.177562] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/ZAAOPBF6QOV76SOU4CURM2ZQUD' does not support file handles, falling
back to xino=off.
Jan 19 16:13:04 Proxmox-VE kernel: [552652.202955] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/KG6XV5ST5DRG7QBYN5VCVUJLD2' does not support file handles, falling
back to xino=off.
Jan 19 16:13:05 Proxmox-VE kernel: [552652.222600] overlayfs: fs on '/root/Download/docke
r-data-root/overlay2/l/KG6XV5ST5DRG7QBYN5VCVUJLD2' does not support file handles, falling
back to xino=off.
Jan 19 16:13:05 Proxmox-VE kernel: [552653.091045] overlayfs: fs on '/root/Download/docker-data-root/overlay2/l/ZAAOPBF6QOV76SOU4CURM2ZQUD' does not support file handles, falling back to xino=off.
Jan 20 01:37:25 Proxmox-VE kernel: [ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-16
Jan 20 01:37:25 Proxmox-VE kernel: [ 0.000000] Linux version 5.15.83-1-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) ()
Ideas, hints, and suggestions are very welcome!
The server seems to be working okay again.
@Not_Oles
The Directory "/root/Download/docker-data-root/" is familiar to me .
I created a folder ”docker-data-root“ for storing my docker's data in my LXC container,and ran several docker build job before the server down.
Maybe some error cause the host server to be down,but I dont know why.
Thanks for your report! It's above my current pay grade to explain those kernel messages, but I will guess that it's unlikely that your use of Docker inside your container broke the server.
Please feel free to continue with whatever you were doing. If something happens again, we might get the opportunity to learn.
Maybe someone here can help by explaining those kernel messages and how to find out more on what happened to the server.
@sanvit said:
Just out of curiosity, how much did you pay for it?
it says in post, € 81.70/m
But € 81.70 is expensive to me. And nobody seems interested in the Hard Disk.
@Not_Oles said: I imagined the guys here being excited about the HDD. All the time people seem to be looking for backup space. But there has been hardly anything said about the 16 TB of HDD on this free server.
The 16 TB HDD plus the increase from 64 to 128 GB RAM seems to raise the cost of the server by about $30/month.
Well, today I found an auction listing at € 119.70 / month for a similar server with only 10 TB HDD. Looks like this server might have been a pretty good deal after all. I wonder how long it will take the server I found today to drop into Low End territory. Or if someone will buy it.
This thread's server still is set to cancel on -- do I remember correctly? -- February 7. But I thought the price of the new listing found today was enough higher (and for less HDD capacity) that some of you guys might be interested to hear about it.
I hope everyone finds the servers they want! Friendly greetings!
@Not_Oles said:
So, this server costs € 81.70 / month for i9-9900K, 128 GB RAM, 2 x 1.92G NVMe, and 16 TB HDD.
But € 81.70 is expensive to me. And nobody seems interested in the Hard Disk.
The 16 TB HDD plus the increase from 64 to 128 GB RAM seems to raise the cost of the server by about $30/month.
Well, today I found an auction listing at € 119.70 / month for a similar server with only 10 TB HDD. Looks like this server might have been a pretty good deal after all. I wonder how long it will take the server I found today to drop into Low End territory. Or if someone will buy it.
I said before, but when I was buying mine, the prices on the auction seemed very low, probably because everyone was busy looking at Black Friday deals not the regular server auctions. But then, I got 128GB for about $1 more than 64GB. That was the only 128GB without a huge HDD, as most of the other SSD-only ones were mostly 64GB. The rest of the 128GB ones were quite a bit more as they all had the same huge disk as yours. I can't remember how much, but it might well have been around your €81.70 price.
As to why people aren't that interested in disk, it's probably because large disk either attracts people who are torrenting or people who want reliable long-term storage. The way you've been advertising your machines in general makes it seem good for experiments, but not particularly what you want if you want long term security for your data.
I also remember you thinking about asking if they'd remove the HDD, but I don't think that's how Hetzner works. These server auctions are not configurable beyond IPv4 or no IPv4. I think it's probably easier for them and you to just find another auction when it drops down low, and then just cancel this one. But as you have your second Heztner box now, maybe you don't need a third!
As to how the auctions work, it seems to be that when the timer drops down to 0:00:00, then the price drops by €1 and resets to something (maybe 24 hours?). And whenever someone buys that configuration, the price for all of them jumps back to the maximum. I think they rely on keeping the process a bit obscure, so it's hard to find out recent low points for the prices, so that people will buy earlier at the higher prices.
Thanks very much for your helpful and interesting comments!
@ralf said: whenever someone buys that configuration, the price for all of them jumps back to the maximum
All similar type of item prices return to that type of item's maximum price when someone buys the lowest priced instance of that item!? I didn't know that! I had imagined, apparently wrongly, that the typical selling price of an item type might be a little below the lowest priced instance of that item.
@ralf said: I think they rely on keeping the process a bit obscure
One time I asked in the chat window what the next price would be on a server that was about to decrease. The chat agent said he didn't know. I looked around and did not find an official Hetzner page documenting the auction features. At that time I did not find any good unofficial documentation either.
I'm still unsure what, if anything, I am going to do about retracting the pending cancellation. . . .
Does anybody know of a similar analysis of the Hetzner Auction?
@ralf said: The way you've been advertising your machines in general makes it seem good for experiments, but not particularly what you want if you want long term security for your data.
Trying to figure out whether to keep this server, I decided to take a look at the SMART values. It looks like the NVMe disks are less than half gone (41 and 42%). Does anyone see an indication of a disk problem?
This server presently has two NVMe disks on RAID 1. Having another, similar server set up on RAID 0 seems way faster just doing ordinary operations like updating.
This server's present setup has the hard drive set up by Proxmox with LVM. One thought I imagined was to reinstall this server with RAID 0 and then keep a backup on the hard disk. The RAID 0 plus HDD backup might be a great possibility because there would be both noticeably faster NVMe disks with double the useable space plus also a reliable HDD backup. Does Proxmox offer some specific way to set up a synced backup of RAID 0 NVMes to the HDD?
I'd be grateful for any helpful comments or suggestions. Thanks!
For /dev/nvme0n1
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 36 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 41%
Data Units Read: 47,787,336 [24.4 TB]
Data Units Written: 327,775,652 [167 TB]
Host Read Commands: 2,246,930,273
Host Write Commands: 7,802,107,885
Controller Busy Time: 31,188
Power Cycles: 34
Power On Hours: 18,561
Unsafe Shutdowns: 12
Media and Data Integrity Errors: 0
Error Information Log Entries: 175
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 36 Celsius
Temperature Sensor 2: 46 Celsius
For /dev/nvme1n1
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 43 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 42%
Data Units Read: 36,636,042 [18.7 TB]
Data Units Written: 321,507,611 [164 TB]
Host Read Commands: 1,934,799,054
Host Write Commands: 6,648,407,405
Controller Busy Time: 31,451
Power Cycles: 34
Power On Hours: 17,702
Unsafe Shutdowns: 12
Media and Data Integrity Errors: 0
Error Information Log Entries: 173
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 43 Celsius
Temperature Sensor 2: 56 Celsius
@Not_Oles said:
Trying to figure out whether to keep this server, I decided to take a look at the SMART values. It looks like the NVMe disks are less than half gone (41 and 42%). Does anyone see an indication of a disk problem?
So, if writing approx 165TB has used up 42% of the write threshold, you should be good for at least another 200TB of writes. These disks have taken about 2 years to get this much write activity, so I guess it all depends if you're likely to keep the server for more than 2 years. Anything under that, and the odds are good that it won't fail before you get rid of it.
Also, I suspect these figures are quite cautious. Not every drive will fail at the 100% threshold, many will go on way past that (and of course, it could fail before that) as the counter is just a guess-timate indication of how many blocks have been written vs how many are expected to fail during the drive's lifetime while still leaving enough good blocks to maintain its official capacity.
@ralf Thanks for your helpful comment on the SMART values!
Does anyone want to chime in on the idea of switching the NVMes to RAID 0 plus using the HDD for a synced backup of the NVMe drives? And how best to set that up?
Comments
@Not_Oles
The Directory "/root/Download/docker-data-root/" is familiar to me .
I created a folder ”docker-data-root“ for storing my docker's data in my LXC container,and ran several docker build job before the server down.
Maybe some error cause the host server to be down,but I dont know why.
Hi @subenhon!
Thanks for your report!
It's above my current pay grade to explain those kernel messages, but I will guess that it's unlikely that your use of Docker inside your container broke the server.
Please feel free to continue with whatever you were doing. If something happens again, we might get the opportunity to learn.
Maybe someone here can help by explaining those kernel messages and how to find out more on what happened to the server.
Thanks again @subenhon!
Tom
MetalVPS
Here are some possibly helpful links:
File Handle
File Descriptor
Handle (computing)
MetalVPS
Hello!
So, this server costs € 81.70 / month for i9-9900K, 128 GB RAM, 2 x 1.92G NVMe, and 16 TB HDD.
But € 81.70 is expensive to me. And nobody seems interested in the Hard Disk.
Well, today I found an auction listing at € 119.70 / month for a similar server with only 10 TB HDD. Looks like this server might have been a pretty good deal after all. I wonder how long it will take the server I found today to drop into Low End territory. Or if someone will buy it.
This thread's server still is set to cancel on -- do I remember correctly? -- February 7. But I thought the price of the new listing found today was enough higher (and for less HDD capacity) that some of you guys might be interested to hear about it.
I hope everyone finds the servers they want! Friendly greetings!
MetalVPS
I said before, but when I was buying mine, the prices on the auction seemed very low, probably because everyone was busy looking at Black Friday deals not the regular server auctions. But then, I got 128GB for about $1 more than 64GB. That was the only 128GB without a huge HDD, as most of the other SSD-only ones were mostly 64GB. The rest of the 128GB ones were quite a bit more as they all had the same huge disk as yours. I can't remember how much, but it might well have been around your €81.70 price.
As to why people aren't that interested in disk, it's probably because large disk either attracts people who are torrenting or people who want reliable long-term storage. The way you've been advertising your machines in general makes it seem good for experiments, but not particularly what you want if you want long term security for your data.
I also remember you thinking about asking if they'd remove the HDD, but I don't think that's how Hetzner works. These server auctions are not configurable beyond IPv4 or no IPv4. I think it's probably easier for them and you to just find another auction when it drops down low, and then just cancel this one. But as you have your second Heztner box now, maybe you don't need a third!
As to how the auctions work, it seems to be that when the timer drops down to 0:00:00, then the price drops by €1 and resets to something (maybe 24 hours?). And whenever someone buys that configuration, the price for all of them jumps back to the maximum. I think they rely on keeping the process a bit obscure, so it's hard to find out recent low points for the prices, so that people will buy earlier at the higher prices.
Hi @ralf
Thanks very much for your helpful and interesting comments!
All similar type of item prices return to that type of item's maximum price when someone buys the lowest priced instance of that item!? I didn't know that! I had imagined, apparently wrongly, that the typical selling price of an item type might be a little below the lowest priced instance of that item.
One time I asked in the chat window what the next price would be on a server that was about to decrease. The chat agent said he didn't know. I looked around and did not find an official Hetzner page documenting the auction features. At that time I did not find any good unofficial documentation either.
I'm still unsure what, if anything, I am going to do about retracting the pending cancellation. . . .
For HN there is on Github A List of Hacker News's Undocumented Features and Behaviors; see also A List of Hacker News's Undocumented Features and Behaviors (HN discussion of the Github page).
Does anybody know of a similar analysis of the Hetzner Auction?
Agreed. Well said!
MetalVPS
Trying to figure out whether to keep this server, I decided to take a look at the SMART values. It looks like the NVMe disks are less than half gone (41 and 42%). Does anyone see an indication of a disk problem?
This server presently has two NVMe disks on RAID 1. Having another, similar server set up on RAID 0 seems way faster just doing ordinary operations like updating.
This server's present setup has the hard drive set up by Proxmox with LVM. One thought I imagined was to reinstall this server with RAID 0 and then keep a backup on the hard disk. The RAID 0 plus HDD backup might be a great possibility because there would be both noticeably faster NVMe disks with double the useable space plus also a reliable HDD backup. Does Proxmox offer some specific way to set up a synced backup of RAID 0 NVMes to the HDD?
I'd be grateful for any helpful comments or suggestions. Thanks!
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 36 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 41%
Data Units Read: 47,787,336 [24.4 TB]
Data Units Written: 327,775,652 [167 TB]
Host Read Commands: 2,246,930,273
Host Write Commands: 7,802,107,885
Controller Busy Time: 31,188
Power Cycles: 34
Power On Hours: 18,561
Unsafe Shutdowns: 12
Media and Data Integrity Errors: 0
Error Information Log Entries: 175
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 36 Celsius
Temperature Sensor 2: 46 Celsius
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 43 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 42%
Data Units Read: 36,636,042 [18.7 TB]
Data Units Written: 321,507,611 [164 TB]
Host Read Commands: 1,934,799,054
Host Write Commands: 6,648,407,405
Controller Busy Time: 31,451
Power Cycles: 34
Power On Hours: 17,702
Unsafe Shutdowns: 12
Media and Data Integrity Errors: 0
Error Information Log Entries: 173
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 43 Celsius
Temperature Sensor 2: 56 Celsius
For /dev/sda
MetalVPS
So, if writing approx 165TB has used up 42% of the write threshold, you should be good for at least another 200TB of writes. These disks have taken about 2 years to get this much write activity, so I guess it all depends if you're likely to keep the server for more than 2 years. Anything under that, and the odds are good that it won't fail before you get rid of it.
Also, I suspect these figures are quite cautious. Not every drive will fail at the 100% threshold, many will go on way past that (and of course, it could fail before that) as the counter is just a guess-timate indication of how many blocks have been written vs how many are expected to fail during the drive's lifetime while still leaving enough good blocks to maintain its official capacity.
@ralf Thanks for your helpful comment on the SMART values!
Does anyone want to chime in on the idea of switching the NVMes to RAID 0 plus using the HDD for a synced backup of the NVMe drives? And how best to set that up?
MetalVPS
No idea about Proxmox, but I guess the idea would be to create an LVM snapshot and then copy from that snapshot to the backup disk.
Just a friendly reminder! The server's still set to cancel on February 7. @cmeerw @iaecm @subenhon @toumi111
MetalVPS
Tomorrow!
MetalVPS