benchmarks — LowEndSpirit

The Problem with Generalizations: A Response to "The Problem with Benchmarks" by raindog308

Mason — Wed, 12 May 2021 17:56:05 +0000

I was going to post this in the Rants category, but would rather not confine my thoughts on this to only LES members (the Rants category requires you to be signed in to view the threads).

YABS – and many others like it over the years – attempts to produce a meaningful report to judge or grade the VM. It reports CPU type and other configuration information, then runs various disk, network, and CPU tests to inform the user if the VPS service he’s just bought is good, bad, or middling. But does it really?

I stumbled upon a blog post recently from raindog308 on the Other Green Blog and was amused that YABS was called out. Raindog states that YABS (and other benchmark scripts/tests like it) may be lacking in its ability to "produce a meaningful report to judge or grade the VM". Some of the reasons for discrediting and proposed alternatives had me scratching my head. I notice that raindog has been hard at work lately pumping up LEB with good content. But is he really?

I'm going to cherry pick some quotes and arguments to reply to below -

It’s valid to check CPU and disk performance for outliers. We’ve all seen overcrowded nodes. Hopefully, network performance is checked prior to purchase through test files and Looking Glass.

I'd argue that not all providers have readily available test files for download and/or a LG. Additionally, it can also be misleading when hosts simply link to their upstream's test files or have their LG hosted on a different machine/hardware that may not have the same usage patterns and port speeds that one would expect in the end-user's VM. However, the point is noted to do some due diligence and research the provider a bit, as that's certainly important.

I'd also argue that iperf (which YABS uses for the network tests) is much more complex than a simple test file/LG. If all you care about is a quick, single-threaded, single-direction, HTTP download then sure use the test file to your heart's content. BUT if you actually care about overall capacity and throughput to different areas of the world in BOTH directions (upload + download), then a multi-threaded, bi-directional iperf test can be much more telling of overall performance.

But other than excluding ancient CPUs and floppy-drive-level performance, is the user really likely to notice a difference day-in and day-out between a 3.3Ghz CPU and a 3.4Ghz one? Particularly since any operation touches many often-virtualized subsystems.

I actually laughed out loud at this comment. I guess I didn't realize that people use benchmarking scripts/tools to differentiate between a "3.3Ghz CPU and a 3.4Ghz one"... (Narrator: "they don't").

Providers have different ways that they fill up their nodes -- overselling CPU, disk space, network capacity, etc. is, more often than not, mandatory to keep prices low. Most (all?) providers are doing this in some form and the end-user most of the time is none the wiser as long as the ratios are done right and the end-user has resources available to meet their workload.

A benchmarking script/tool does help identify cases where the provider's nodes are oversubscribed to excess and is immediately obvious if disk speeds, network speeds, or CPU performance are drastically lower than they should be for the advertised hardware. Could this be a fluke and resolve itself just a few minutes later? Certainly possible. On the flip side, could performance of a system with good benchmark results devolve into complete garbage minutes/hours/days after the test is ran? Certainly possible as well. Multiple runs of a benchmark tool spread out over the course of hours, days, or weeks could potentially help identify if either of these cases are true.

On a personal note, I've seen dozens of instances where customers of various providers post their benchmark results and voice some concerns of system performance. And a large percentage of the time, the provider is then able to rectify the issues presented by fixing hardware issues, identifying abusers on the same node that are impacting performance, etc. From this, one can notice patterns starting to emerge where you can see the providers that take criticisms (via posts containing low-performing benchmarks) to better improve their services and ensure their customers are happy with the resources that they paid for. Other trends help identify providers to avoid where consistent low network speeds, CPU scores, etc. go unaddressed, indicating unhealthy overselling. But I digress...

If I could write a benchmark suite, here is what I would like it to report.

Here we get a rapid-fire of different un-quantifiable metrics that would be in raindog's ideal benchmarking suite:

Reliability
Problems
Support
Security
"Moldy Oldies" (outdated VM templates)
"Previous Residents" (previous owners of IPs)
"My Neighbors" (anybody doing shitcoin mining on the same node)

Raindog realizes it'd be hard to get at these metrics -

Unfortunately, all of these things are impossible to quantify in a shell script.

They'd be impossible to quantify by any means (shell script or fortune teller)... Almost all of the above metrics are subject to personal opinions and preference. Some of these can be investigated by other means -- reliability: one could check out a public status page for that provider (if available); problems: one could search for public threads of people having issues and noting how the provider responds/resolves the issues; moldy oldies: a simple message/pre-sales ticket to the provider could alleviate that concern.

Anyways, the above metrics are highly subjective and are of varying importance to perspective buyers (someone might not give a shit about support response times or if their neighbor is having a party on their VM).

But what's something that everyone is actually concerned about? How the advertised VM actually performs.

And how do we assess system performance in a non-subjective manner? With benchmarking tests.

If you ask me which providers I recommend, the benchmarks that result from VMs on their nodes are not likely to factor into my response. Rather, I’ll point to the provider’s history and these “unquantifiables” as major considerations on which providers to choose.

That's great and, in fact, I somewhat agree here. Being an active member of the low end community, I have the luxury of knowing the providers that run a tight ship and care about their customers and which ones don't based on their track record. But not everyone has the time to dig through thousands of threads to assess a provider's support response or reliability and not everyone in the low end community has been around long enough to develop opinions and differentiate between providers that are "good" or "bad".

I also found it highly amusing that a related post on the same page links to another post regarding a new benchmarking series by jsg. This post is also written by raindog. The framing is a bit different in that post, where benchmarks are described in a way where they aren't represented as entirely useless and lacking in their ability to "produce a meaningful report to judge or grade the VM". I'm not really sure what happened in the few months between those two posts and they both talk about the limitations of benchmarking tool, but just would like to note the changes in tone.

My main point in posting this "response" (read: rant) is that benchmark tests aren't and shouldn't be the all-in-one source for determining if a provider and a VM are right for you. I don't advertise the YABS project in that manner. In the description of the tool I even state that YABS is "just yet another bench script to add to your arsenal." So I'm not really sure of the intent of raindog's blog post. Should users not be happy when they score a sweet $7/year server that has top-notch performance and post a corresponding benchmark showing how sexy it is? Should users not test their system to see if they are actually getting the advertised performance that they are expecting? Should users not use benchmarking tools as a means of debugging and bringing a provider's attention to any resource issues? Those "unquantifiables" that are mentioned certainly won't help you out there.

This response is now much longer than the original blog post that I'm responding to, so I'll stop here.

Happy to hear anyone else's thoughts or face the pitch forks on this one.

A Couple of BSD VPS Questions (FreeBSD, OpenBSD)

flips — Thu, 17 Dec 2020 23:15:06 +0000

@AnthonySmith said in another thread:
As it is simple to find a good 512mb deal it kind of makes this thread redundant so I have span one up myself on IH infrastructure but because I hate things being easy I am doing it on FreeBSD instead.

Love this, "I hate things being easy" ...
Speaking of FreeBSD. I recently spun up both a FreeBSD and a OpenBSD node.

After my first purpose is accomplished, my curiosity will have me test other stuff as well. Wondering:

What kinda benchmark script/tool would you guys run on BSD? (yabs.sh by @Mason looks kinda Linux specific)
Of the many available httpd's available, which ones of these should I give a shot? OpenBSD's own/default (ob)httpd, (s)thttpd? Others?

DigitalOcean, Vultr and Hetzner - Benchmarked each plan!

cyberduck — Sat, 11 Jul 2020 19:49:03 +0000

Hi guys,

This is my first post!

I have benchmarked most VPS plans on DigitalOcean, Vultr and Hetzner.

Here are my results:

Benchmark comand used:

curl -s https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/yabs.sh | bash

Hope you find this interesting and useful.

Cyberduck

edit:
12/07/2020 - Vultr High Frequency plans have been added!
14/07/2020 - Hostworld.uk VPS-3 added

TetaHost?

moonswamp — Tue, 21 Jul 2020 22:23:54 +0000

Saw they are very new hosting provider since 2019, and several offers provided in this forum !

Any have vps from them ? any benchmarks ?

1blu vServer 4R 1CPU/4G/60GSSD 12euro/y + .DE Domain

pr0lz — Wed, 22 Jan 2020 15:21:20 +0000

Hi all,

1blu is a german provider that offers pretty cheap vps from time to time. This time I could not resist and wanted to check them out. Price non recurring, only first 12 month for this price.
https://www.1blu.de/server/vserver/

General

They have a manual provisioning process and you need to call them to approve the order. They do not offer IPv6 as far as I know.
Edit: Also included: 1 .DE Domain for the year.

Benchmarks

Done with preinstalled centos7 image.

bench

----------------------------------------------------------------------
CPU model : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
Number of cores : 1
CPU frequency : 2399.926 MHz
Total size of Disk : 59.0 GB (0.9 GB Used)
Total amount of Mem : 4096 MB (46 MB Used)
Total amount of Swap : 0 MB (0 MB Used)
System uptime : 0 days, 4 hour 14 min
Load average : 0.00, 0.06, 0.10
OS : CentOS 7.7.1908
Arch : x86_64 (64 Bit)
Kernel : 3.10.0
----------------------------------------------------------------------
I/O speed(1st run) : 538 MB/s
I/O speed(2nd run) : 531 MB/s
I/O speed(3rd run) : 526 MB/s
Average I/O speed : 531.7 MB/s
----------------------------------------------------------------------
Node Name IPv4 address Download Speed
CacheFly 205.234.175.175 76.5MB/s
Linode, Tokyo2, JP 139.162.65.37 4.87MB/s
Linode, Singapore, SG 139.162.23.4 5.16MB/s
Linode, London, UK 176.58.107.39 76.2MB/s
Linode, Frankfurt, DE 139.162.130.8 65.1MB/s
Linode, Fremont, CA 50.116.14.9 6.94MB/s
Softlayer, Dallas, TX 173.192.68.18 12.1MB/s
Softlayer, Seattle, WA 67.228.112.250 11.3MB/s
Softlayer, Frankfurt, DE 159.122.69.4 83.3MB/s
Softlayer, Singapore, SG 119.81.28.170 9.19MB/s
Softlayer, HongKong, CN 119.81.130.170 6.55MB/s
----------------------------------------------------------------------

nench

-------------------------------------------------
nench.sh v2019.07.20 -- https://git.io/nench.sh
benchmark timestamp: 2020-01-22 12:49:57 UTC
-------------------------------------------------
`Processor: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz` `CPU cores: 1` `Frequency: 2399.926 MHz` `RAM: 4.0G` `Swap: -` `Kernel: Linux 3.10.0 x86_64`
Disks:
ploop55056 60G HDD
`CPU: SHA256-hashing 500 MB` `3.007 seconds` `CPU: bzip2-compressing 500 MB` `8.970 seconds` `CPU: AES-encrypting 500 MB` `2.678 seconds`
ioping: seek rate
min/avg/max/mdev = 39.6 us / 61.8 us / 20.9 ms / 130.3 us
ioping: sequential read speed
generated 13.3 k requests in 5.00 s, 3.24 GiB, 2.66 k iops, 664.0 MiB/s
`dd: sequential write speed` `1st run: 525.47 MiB/s` `2nd run: 460.62 MiB/s` `3rd run: 544.55 MiB/s` `average: 510.22 MiB/s`
IPv4 speedtests
your IPv4: 178.254.18.xxxx
`Cachefly CDN: 70.67 MiB/s` `Leaseweb (NL): 68.27 MiB/s` `Softlayer DAL (US): 11.36 MiB/s` `Online.net (FR): 59.34 MiB/s` `OVH BHS (CA): 14.20 MiB/s`
No IPv6 connectivity detected
-------------------------------------------------
`-------------------------------------------------` `nench.sh v2019.07.20 -- https://git.io/nench.sh` `benchmark timestamp: 2020-01-22 12:50:53 UTC` `-------------------------------------------------` `Wed Jan 22 11:50:00 CET 2020`
Processor: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
CPU cores: 1
Frequency: 2399.926 MHz
RAM: 4.0G
Swap: -
Kernel: Linux 3.10.0 x86_64
`Disks:` `ploop55056 60G HDD`
CPU: SHA256-hashing 500 MB
3.345 seconds
CPU: bzip2-compressing 500 MB
8.477 seconds
CPU: AES-encrypting 500 MB
2.600 seconds
`ioping: seek rate` `min/avg/max/mdev = 44.4 us / 63.8 us / 12.1 ms / 122.5 us` `ioping: sequential read speed` `generated 13.7 k requests in 5.00 s, 3.35 GiB, 2.75 k iops, 686.7 MiB/s`
dd: sequential write speed
1st run: 516.89 MiB/s
2nd run: 544.55 MiB/s
3rd run: 563.62 MiB/s
average: 541.69 MiB/s
`IPv4 speedtests` `your IPv4: 178.254.18.xxxx`
Cachefly CDN: 59.38 MiB/s
Leaseweb (NL): 88.35 MiB/s
Softlayer DAL (US): 11.60 MiB/s
Online.net (FR): 93.48 MiB/s
OVH BHS (CA): 17.38 MiB/s
``
No IPv6 connectivity detected
-------------------------------------------------

yabs

Have not done anything with the machine but wanted to share if someone is interested. Look at the benches yourself and see if its something of interest for you.
Please request more benchmarks if you want.

Not affiliated with the hoster in any way btw.

Benchmark & Docker

Dream — Sun, 22 Dec 2019 14:01:12 +0000

Hey Guys,

We all love to show our virtual Dick on bench marking the shit out of our servers. But i hate to always search for the URL's because I'm too lazy to save them down.

So i had the idea on creating a Docker image which contains the most of the Scripts.
Now my questions,

Is there any interest in something like that?
If yes, which scripts would you like to see?
What are the downsides in this approach?

I would love to hear your opinions about this Topic.

New Intel Core i9-10980XE and AMD Threadripper 3960X/3970X processors

ITLabs — Mon, 25 Nov 2019 16:36:39 +0000

Today (11.25) Intel is rolling out the Core™ i9-10980XE Extreme Edition as their new Cascade Lake X-Series processor and AMD is also launching their new Threadripper 3960X/3970X processors.

The specs are:

1) Intel Core™ i9-10980XE

Cores: 18
Threads: 36
Processor Base Frequency: 3.00 GHz
Max Turbo Frequency: 4.60 GHz
TBM 3.0 frequency: 4.80 GHz
Cache: 24.75 MB Intel® Smart Cache
DDR4-2933 quad channel memory support
TDP: 165 W
L1TF/Meltdown hardware mitigations in place

2) AMD Ryzen™ Threadripper™ 3960X

Cores: 24
Threads: 48
Base Clock: 3.8GHz
Max Boost Clock: Up to 4.5GHz
Total L1 Cache: 2.25MB
Total L2 Cache: 12MB
Total L3 Cache: 128MB
DDR4-3200
Default TDP: 280W

3) AMD Ryzen™ Threadripper™ 3970X

Cores: 32
Threads: 64
Base Clock: 3.7GHz
Max Boost Clock: Up to 4.5GHz
Total L1 Cache: 3MB
Total L2 Cache: 16MB
Total L3 Cache: 128MB
DDR4-3200
Default TDP: 280W

In order to be competitive with the new AMD processors (3960X = $1399 USD and 3970X = $1999 USD), the Core i9 10980XE is launching at $979 USD.

Phoronix released linux benchmarks for the new processors here and here.

TL;DR Threadripper dominate in performance.