DigitalOcean, Vultr and Hetzner - Benchmarked each plan!

vimalware · July 2020

@vimalware said:
That reminds me. I need to go check up on my ultravps instance.
I'm curious to see how much of that 2 week old 4GB test file I created, is still present in the Host Node's ZFS ARC(ram).

Here we go, for the curious. Looks like everything is travelling the full ZFS access path/stack.
1 x Full Random read test with 128k blocksize :

./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=128k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=100
test: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.20
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=445MiB/s][r=3563 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=27879: Fri Jul 24 02:16:33 2020
  read: IOPS=3957, BW=495MiB/s (519MB/s)(4096MiB/8281msec)
   bw (  KiB/s): min=257277, max=744750, per=100.00%, avg=508770.19, stdev=124657.26, samples=16
   iops        : min= 2009, max= 5818, avg=3974.31, stdev=973.96, samples=16
  cpu          : usr=13.99%, sys=74.13%, ctx=1187, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=495MiB/s (519MB/s), 495MiB/s-495MiB/s (519MB/s-519MB/s), io=4096MiB (4295MB), run=8281-8281msec

Disk stats (read/write):
  vdb: ios=32399/2, merge=16/2, ticks=58520/0, in_queue=55212, util=84.67%

I used 128k blocksize to bet that ultravps are using the default ZFS recordsize. I might be proven wrong in next test after a week.
Summary: read: IOPS=3957, BW=495MiB/s (519MB/s)(4096MiB/8281msec)

dstat output for the duration of the full Random read test. (to get some better visibility into CPU iowait under zfs.)

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
--snipped--
  2   3  95   0   0|   0     0 |2039B  271B|   0     0 |  88   299 
  1   2  97   0   0|   0     0 |1611B  173B|   0     0 |  60   220 
 11  37  51   1   0|1672k  108k|3036B  338B|   0   108k| 229   541 
 23  67  10   0   0|  18M  628k|2511B  650B|   0   628k| 408   645 
 15  85   0   0   0| 657M    0 |2130B  720B|   0     0 |2585   572 
 17  82   1   0   0| 597M    0 |2512B  223B|   0     0 |2108   516 
 20  80   0   0   0| 552M    0 |2509B  396B|   0     0 |2318   554 
 18  82   0   0   0| 532M   32k|1807B  363B|   0     0 |2013   554 
 20  80   0   0   0| 392M  136k|2526B  372B|   0     0 |1435   582 
 19  81   0   0   0| 454M    0 |2335B  458B|   0     0 |1655   567 
 22  78   0   0   0| 344M    0 |2214B  383B|   0     0 |1303   646 
 22  78   0   0   0| 447M    0 |1769B  563B|   0     0 |1588   532 
 10  47  43   0   0| 103M    0 |2680B 1135B|   0     0 | 508   701 
  3   4  93   0   0|   0     0 |2333B  432B|   0     0 |  87   270 
  6   4  90   0   0|   0     0 |2346B 2078B|   0     0 |  94   332
--snipped--

willie · July 2020

Shouldn't be affected by caching. Random read/writes and direct I/O flag on.

I have to wonder whether that flag really works under virtualization. It's sometimes doubtful even on bare metal, because of the many layers of caches in the controller, the drive, etc.

ViridWeb · July 2020

Wow! Hetzner's €2.99 plan is hard to defeat specially that performance at that price.

Nice effort BTW,

willie · July 2020

outdated, please delete.

willie · July 2020

I just ran the test on the 16 vcore Epyc instance. Multicore cpu was 5694 which I guess is disappointing since it's less than 10x the single core speed. It still beats the 8 vcore Intel instance by a fair amount. It's less than half the speed of a Hetzner Ryzen 3900 (AX61 I guess, 3.1 ghz, slower than the 3900X) dedi which scored 11656. So I guess it's slower even than a dedicated AX41. Oh well I guess it's still nice to be able to spin something like this up for a quick compute task, using the hourly billing. Other Hetzner results are all over the place:

https://browser.geekbench.com/v5/cpu/search?utf8=✓&q=hetzner

The AX51 (Ryzen 3700X) is consistently in the 9000+ range and the AX61 is 11000+. I haven't found an AX41 result that makes any sense yet but I'd guess it around 6000-7000.

My bench results for the CPX51 are here:

https://bin.privacytools.io/?32874f2c0832178f#w2UUx4hxYt/wEjltIRwc9eMUYbXtNW+KHjWBJyGy9rw=

@cyberduck , please feel free to put this link in the main post.

Hmm, here someone got 7449 on a CPX51 which is a fair bit better than what I got:

https://browser.geekbench.com/v5/cpu/3006334

Here, someone got 990 single, 12995 multi, so clearly an unloaded node. Better than 2x my instance's speed.

https://browser.geekbench.com/v5/cpu/2787981

So I guess these things are fairly loaded by now. My instance was at Falkenstein which might be the busiest site, fwiw.

vimalware · July 2020

@willie said:

my instance was at Falkenstein which might be the busiest site, fwiw.

Yes I'd imagine Finland would be the place to try the one off 16cpu compute jobs

willie · July 2020

Yeah I might try Finland sometime but these things are too inconsistent and not that fast in the first place. I also wonder what happens in terms of throttling if you do a long-running compute task. The product is less interesting than I'd hoped. I still might use one now and then. Meanwhile my now ancient i7-3770 auction server is fast enough for most things (around 3500 on that bench).

The €4.19 instance (3.49 ex vat) looks great as a general purpose vps fwiw.

DigitalOcean, Vultr and Hetzner - Benchmarked each plan!

Comments