Does anyone have experience with SSD based Ceph?
In the last year we have migrated all HDD based vServers to a Ceph storage and are excited about the possibilities and the performance the system offers.
Now I would like to migrate our SSD vServer to a Ceph based storage and I am facing some questions that I have answered myself with the HDD based system through many tests. But that cost a lot of time and money, which I only accepted because I also wanted to familiarize myself intensively with Ceph before I start running customers here.
Which SSDs do you use in your Ceph storage?
How important is the RAM speed (I could imagine that this plays an upscale role with SSDs)?
Any good or negative experiences you can share?
I would be extremely happy if a lively exchange of experiences and opinions takes place here, without anyone attacking anyone else.
Thanks a lot for your support.
Comments
I know Silicloud and Prepaid-Host are using Ceph-based storage with SSD. Maybe they can make a suggestion.
(But SSD performed worse than normal in Ceph)
CEPH is great but only makes sense if you can tolerate the lower performance compared to native local storage.
Ideally, all servers in your CEPH cluster have the same hardware setup, otherwise the slowest box will be the bottleneck.
Oh, and you'll need fast NICs to get the most out of it, speaking 25G or 40G+.
In terms of SSD selection you'll want to go with datacenter SSDs.
I've made great experiences with Samsung PM9A3 if you're looking for NVMe SSDs or Micron 5300 Pro SATA SSD
https://v6node.com
@iandk
Thanks a lot.
We already running Ceph on a 40G based storage network wich we would also use for our SSD ceph.
I've been told extremely often now to just go for datacenter SSDs and wonder: why exactly?
It's not like we don't already use a lot of Samsung PM1635 and other datacenter ssds, but I have had very good experiences with some cheaper SSDs in RAID10 operation and wonder if I could use them in Ceph as well.
Is it not recommended because the IOPS are significantly lower or because there are more frequent failures? As long as it does not affect the performance, I would try to continue using the hardware I have already purchased. If it does affect performance, we'll soon have SSDs for inexpensive dedicated servers.
PS: Micron 5300 Pro was not on my radar. Any more Enterprise grade hardware that I don't know yet?
Consumer SSDs often have a much lower write endurance and aren't meant for longer sustained writes.
Depending on the model they might have a high initial performance which will drop a lot once the SLC cache fills up.
They also lack features like power loss protection which at least combined with ZFS is quite a performance hit.
Intel SSD DC S3700 is also a great choice
https://v6node.com