Boost Ceph block performance

Boost Ceph block performance with Rhel 7.5, Ceph Luminous and the Micron 9200 Max NVMe SSD

Hi, everybody,

Usually a point release in an OS or storage solution is no big deal, but this time is different. I tested Red Hat Enterprise Linux 7.5 and Ceph Luminous 12.2.5, both point releases from my previous blog on Bluestore vs. Filestore Performance, and found a surprising improvement in block performance.

4KB random write IOPS performance increases by 12%, average latency decreases by 10%, and 99.99% tail latency decreases by 24%.

4KB random read IOPS and average latency are similar, and 99.99% tail latency decreases by 20% to 43%.

This solution is optimized for block performance. Random small block testing using the Rados Block Driver in Linux saturates platinum-level 8168 Intel Purley processors in a 2-socket storage node.

With 4 storage nodes and 10 drives per storage node, this architecture has a usable storage capacity of 232TB that can be scaled out by adding additional 1U storage nodes.

Reference Design – Hardware

Test Results and Analysis

Ceph Test Methodology

Ceph Luminous (12.2.4 & 12.2.5) is configured with Bluestore with 2 OSDs per Micron 9200 MAX NVMe SSD. RocksDB and WAL data are stored on the same partition as data.

There are 10 drives per storage node and 2 OSDs per drive, 80 total OSDs with 232TB of usable capacity.

The Ceph storage pool tested was created with 8192 placement groups and 2x replication. Performance is tested with 100 RBD images at 75GB each, providing 7.5TB of data on a 2x replicated pool, 15TB of total data.

4KB random block performance was measured using FIO against the Rados Block Driver. We used 10 load generation servers (Dual-CPU Xeons w/ 50GbE networking) and ran multiple FIO processes per load generation server. Each FIO process accessed a unique RBD image and FIO processes were distributed evenly across the 10 load generation servers. For example, the 100 FIO clients test used 10 FIO processes per load generation server.

We are CPU limited in all tests, even with 2x Intel 8168 CPUs per storage node. All tests were run 3 times for 10 minutes with a 5-minute ramp up per test.

RBD FIO 4KB Random Write Performance: RHEL 7.4 + Ceph 12.2.4 vs. RHEL 7.5 + Ceph 12.2.5

RHEL 7.5 + Ceph Luminous 12.2.5 provides a 12% increase in IOPS and a 10% decrease in average latency.

Tail latency is improved with RHEL 7.5 and Ceph Luminous 12.2.5, decreasing by 25% at 100 FIO clients.

RBD FIO 4KB Random Read Performance: RHEL 7.4 + Ceph 12.2.4 vs. RHEL 7.5 + Ceph 12.2.5

4KB random read performance is similar between RHEL 7.4 + Ceph Luminous 12.2.4 and RHEL 7.5 + Ceph Luminous 12.2.5. There’s a slight increase in IOPS with a maximum of 2.23 Million IOPs.

Tail latency is improved with RHEL 7.5 and Ceph Luminous 12.2.5, decreasing by 43% at queue depth 16 and 23% at queue depth 32.

Would You Like to Know More?

Ceph + the Micron 9200 MAX NVMe SSD on the Intel Purley platform is super fast. The latest reference architecture for Micron Accelerated Ceph Storage Solutions is available now. I presented details about the reference architecture and other Ceph tuning and performance topics during my session at OpenStack Summit 2018. A recording of my session is available here.

Have additional questions about our testing or methodology? Email us ssd@micron.com.