Connection being bottle necked by lack of resources.

tonytony
edited May 2020 in Technical

Since a few days I experienced instability on my VPS, it may be since the beginning but I really saw it when my VPN started to work. I want to share my measurement techniques to know if you think they are reliable enough and if some of you experienced the same kind of problems. The support answered to me that Hetzner may be under a DDoS atack.

On the VPS I run :

iperf -s -p 19003

On my Linux box (wired) :

iperf -c vps -p 19003 -i 0.5 -t 300 >> bw.log

I then rework the log file with cut, sed and a little bit of manual editing to put the data in Excell and generates the graphs attached.

I also made the measurements using ipv6 and the results are better, not perfect but at least it does not fall to zero for long periods.

Comments

  • InceptionHostingInceptionHosting Hosting ProviderOG

    how much Ram do you have?

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • tonytony
    edited May 2020

    @AnthonySmith said:
    how much Ram do you have?

    256MB
    Edit: after some more tests it looks like it is more a CPU throttling or something like that which causes the network freeze. I continue the investigation.

  • InceptionHostingInceptionHosting Hosting ProviderOG
    edited May 2020

    @tony said: 256MB
    Edit: after some more tests it looks like it is more a CPU throttling or something like that which causes the network freeze. I continue the investigation.

    You have a VPS with 1998 RAM in a super high contention environment while testing it with 2020 tools, I would not put much store by your results tbh.

    If it works for what you need great, if it does not, pay more, you are at the super ultra mega budget end of the spectrum.

    You don't enter a skateboard in to a NASCAR race then complain about the 0-60 acceleration :)

    Thanked by (1)g4m3r

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • tonytony

    I understand that this is low end but I also like to understand what happens. Also I will be the only user of this service and I can deal with less performances but the periodic freezes are definitely problematic.

    I can see server side the OpenVPN process rising to 100% CPU and then everything freezes. I am currently testing different Cipher to see if I can improve the performances, I may also try to throttle bandwidth or limit simultaneous connexions.

  • InceptionHostingInceptionHosting Hosting ProviderOG

    I have gone ahead and updated the title to reflect reality rather than give bad PR then.

    Thanked by (1)Hetzner_OL

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • schnafischnafi OG

    I had the same problem on another machine with 256mb ram I am using. Even ssh was not usable and the system hung every couple of seconds. I fixed it by adding 512MB swap. Don't know why it happened, but it fixed the server freezes permanently.

  • InceptionHostingInceptionHosting Hosting ProviderOG

    @schnafi said: Don't know why it happened, but it fixed the server freezes permanently.

    It happened because you were probably trying to squeeze so much in that there was barley any room for anything else, 256mb ram on kvm/xen is significantly less than is required to run unless you are prepared to do some expert level tuning.

    So you essentially downloaded more ram lol.

    You cant add swap on containers/vz

    Thanked by (1)schnafi

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • tonytony

    @schnafi said:
    I had the same problem on another machine with 256mb ram I am using. Even ssh was not usable and the system hung every couple of seconds. I fixed it by adding 512MB swap. Don't know why it happened, but it fixed the server freezes permanently.

    I already have 512MB of swap and even with very low resource intensive tasks like iperf (low on CPU and RAM, not on network) i get freezes. For instance this is the output of iperf :

    [  3] 23.0-23.5 sec  7.88 MBytes   132 Mbits/sec
[  3] 23.5-24.0 sec  5.38 MBytes  90.2 Mbits/sec
[  3] 24.0-24.5 sec  5.25 MBytes  88.1 Mbits/sec
[  3] 24.5-25.0 sec  5.14 MBytes  86.3 Mbits/sec
[  3] 25.0-25.5 sec   764 KBytes  12.5 Mbits/sec
[  3] 25.5-26.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 26.0-26.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 26.5-27.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 27.0-27.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 27.5-28.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 28.0-28.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 28.5-29.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 29.0-29.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 29.5-30.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 30.0-30.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 30.5-31.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 31.0-31.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 31.5-32.0 sec  0.00 Bytes  0.00 bits/sec>

    Why does it freezes even if the CPU is used at less than 20% and RAM is at 24MB used of the 512MB and swap is empty ?

    I have the feeling that something else on the Server (outside of my container) is trying to get resources and then some containers are freezed.

  • tonytony

    By limiting the bandwidth to 10Mb I get much more reliable performances and almost no freeze but there is still (fewer and shorter) freezes.

    $ iperf -c vps -p 19003 -i 0.5 -t 300 -b 10M 
[  3] 75.0-75.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 75.5-76.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 76.0-76.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 76.5-77.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 77.0-77.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 77.5-78.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 78.0-78.5 sec   768 KBytes  12.6 Mbits/sec
[  3] 78.5-79.0 sec   512 KBytes  8.39 Mbits/sec
[  3] 79.0-79.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 79.5-80.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 80.0-80.5 sec   768 KBytes  12.6 Mbits/sec
[  3] 80.5-81.0 sec  28.2 KBytes   462 Kbits/sec
[  3] 81.0-81.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 81.5-82.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 82.0-82.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 82.5-83.0 sec  63.6 KBytes  1.04 Mbits/sec
[  3] 83.0-83.5 sec  1.25 MBytes  21.0 Mbits/sec
[  3] 83.5-84.0 sec  3.00 MBytes  50.3 Mbits/sec
[  3] 84.0-84.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 84.5-85.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 85.0-85.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 85.5-86.0 sec   640 KBytes  10.5 Mbits/sec

    At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

  • WSSWSS Retired

    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    My pronouns are asshole/asshole/asshole. I will give you the same courtesy.

  • InceptionHostingInceptionHosting Hosting ProviderOG

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • tonytony

    @WSS said:
    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    @AnthonySmith said:

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable :(

  • InceptionHostingInceptionHosting Hosting ProviderOG

    @tony said:

    @WSS said:
    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    @AnthonySmith said:

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable :(

    what do you get when not trying artificial benchmarks e.g. :

    wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • vimalwarevimalware OG

    I doubt you need more than 7-8 threads. 6 should cover almost every provider I have.

  • tonytony

    @AnthonySmith said:

    @tony said:

    @WSS said:
    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    @AnthonySmith said:

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable :(

    what do you get when not trying artificial benchmarks e.g. :

    wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

    100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

  • InceptionHostingInceptionHosting Hosting ProviderOG
    edited May 2020

    @tony said:

    @AnthonySmith said:

    @tony said:

    @WSS said:
    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    @AnthonySmith said:

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable :(

    what do you get when not trying artificial benchmarks e.g. :

    wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

    100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

    It is more than enough, I am trying to get you to test and post the results of that because it simulates real world use, in what world are you downloading a 10GB file on to a VPS that cost less than your lunch?

    Honestly if I had to guess based on your comments so far i would assume you have been IO de-prioritized due to the sorts of tests you are running.

    Thanked by (1)vimalware

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • tonytony

    @AnthonySmith said:

    @tony said:

    @AnthonySmith said:

    @tony said:

    @WSS said:
    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    @AnthonySmith said:

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable :(

    what do you get when not trying artificial benchmarks e.g. :

    wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

    100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

    It is more than enough, I am trying to get you to test and post the results of that because it simulates real world use, in what world are you downloading a 10GB file on to a VPS that cost less than your lunch?

    Honestly if I had to guess based on your comments so far i would assume you have been IO de-prioritized due to the sorts of tests you are running.

    I don't understand what you are trying to prove. The problem was here way before I started the tests. If doing apt dist-upgrade made me IO de-prioritized then the service is not cheap anymore regarding what is provided :s

  • schnafischnafi OG

    @AnthonySmith said:
    You cant add swap on containers/vz

    Interesting, I did not know that. Another (new) limitation of OVZ. Regardless, I still prefer OVZ over KVM on low ram machines, where only a minimum of services are running, as they are mostly cheaper.

  • tonytony

    @tony said:

    @AnthonySmith said:

    @tony said:

    @AnthonySmith said:

    @tony said:

    @WSS said:
    As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

    Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    @AnthonySmith said:

    @tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

    The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

    Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

    I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable :(

    what do you get when not trying artificial benchmarks e.g. :

    wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

    100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

    It is more than enough, I am trying to get you to test and post the results of that because it simulates real world use, in what world are you downloading a 10GB file on to a VPS that cost less than your lunch?

    Honestly if I had to guess based on your comments so far i would assume you have been IO de-prioritized due to the sorts of tests you are running.

    I don't understand what you are trying to prove. The problem was here way before I started the tests. If doing apt dist-upgrade made me IO de-prioritized then the service is not cheap anymore regarding what is provided :s

    There is also the 500GB limit per month to prevent abuse.

  • InceptionHostingInceptionHosting Hosting ProviderOG

    @tony said: I don't understand what you are trying to prove.

    OK I will opt out of this discussion then.

    https://inceptionhosting.com
    Please do not use the PM system here for Inception Hosting support issues.

  • Hetzner_OLHetzner_OL Hosting ProviderOG

    @tony - If you think there is a problem on Hetzner's end, please write back to our support team using the same support ticket as you mentioned at the beginning of this thread. Explain why you think the issue is on Hetzner's end. Provide our team with some details of your use case and what troubleshooting you have done. Perhaps they will give you similar feedback to what you have seen here or maybe they will have other insights. -Katie

    We're Katie and Lea and we'll do our best to answer questions you have about Hetzner Online. We and not our employer are responsible for any horrible puns and dated cultural references.

  • tonytony

    @Hetzner_OL said:
    @tony - If you think there is a problem on Hetzner's end, please write back to our support team using the same support ticket as you mentioned at the beginning of this thread. Explain why you think the issue is on Hetzner's end. Provide our team with some details of your use case and what troubleshooting you have done. Perhaps they will give you similar feedback to what you have seen here or maybe they will have other insights. -Katie

    Thank you, I am also discussing with the customer support trying to catch the problem. As I know these offers are supposed to be support-less I don't wan't do be too invasive regarding support .

  • WSSWSS Retired

    @tony said:
    The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

    You are correct. I meant -P for parallel. I am far too use to -n being number of processes.

    @tony said:
    Thank you, I am also discussing with the customer support trying to catch the problem. As I know these offers are supposed to be support-less I don't wan't do be too invasive regarding support .

    So, instead you'll shit all over them on the forum trying to find a specific way to get 1000jiggabits on a 256MB plan? Did you get kicked off of Virmach already?

    If you didn't tag me like 8 fucking times in every response due to bandwidth waste, I would have just ignored you.

    My pronouns are asshole/asshole/asshole. I will give you the same courtesy.

  • tonytony

    Sorry for tagging you untimely, it was not my intention, I just wanted to quote what I was answering for clarity. Also I did not shit on anybody, if I identified the provider, its because I think that it is a useful information to identify the problem. I am not trying to get thousands of Gb continuously, sorry if the benchmark to highlight the problem was not appropriate but even 1Mb for 2 minutes is problematic ... Finally I did not get ban from anywhere as far as I remember, if the "tacit" rules of low end VPS changed since I used one a few month ago, then fine, I am just trying to understand.

  • WSSWSS Retired

    You can likely just take care of this by adding more RAM. If they're looking from the node and there's no issue on the node, the bottleneck is obviously your instance. I'm not sure of ANYTHING being happy in 256MB amymore - I use almost that with my TinyKVM nameservers with no other external-facing services on them.

    My pronouns are asshole/asshole/asshole. I will give you the same courtesy.

  • mikhomikho AdministratorHosting ProviderOG

    It’s always fun trying to measure speed/connectivity during a ddos attack.

    Reading this thread I assume I’m the provider as there are som signs.

    Any who, one of my DE Nodes has been under daily DDoS attacks for roughly a week now.

    Hetzner is doing a great job keeping the node online but its not as stable as it used to be.

    Hope it will end soon.

    https://clients.mrvm.net

  • tonytony

    @mikho said:
    It’s always fun trying to measure speed/connectivity during a ddos attack.

    Reading this thread I assume I’m the provider as there are som signs.

    Any who, one of my DE Nodes has been under daily DDoS attacks for roughly a week now.

    Hetzner is doing a great job keeping the node online but its not as stable as it used to be.

    Hope it will end soon.

    No problem, I guess the timing was unfortunate as I ordered the VPS at the beginning of the DDoS attack. Please let me know when it is over so I can see if the situation effectively improved.

Sign In or Register to comment.

© LowEndSpirit 2024

ipv6 ready