Measuring OpenEBS Local Volume Performance Overhead in Kubernetes

I wrote about the usage of OpenEBS to manage local storage in Kubernetes in my previous posts:

Deploying Percona Kubernetes Operators with OpenEBS Local Storage

OpenEBS for the Management of Kubernetes Storage Volumes

The primary reason to use the local storage is performance and the logical question is then: How big is the performance overhead if we use OpenEBS to manage the local storage?

To look into this I will use sysbench fileio test on the NVMe storage Intel® SSD DC P4610.

Direct Storage

The command to prepare data:

sysbench fileio prepare --file-num=25 --file-total-size=100G --file-extra-flags=direct

The command to run test:

sysbench fileio run --threads=32 --file-async-backlog=1024  --file-num=25 --file-total-size=100G --file-io-mode=async --file-test-mode=rndrw --file-extra-flags=direct --time=120 --report-interval=1 --file-fsync-freq=0  --file-fsync-end=off

This means I will use Async IO to perform read-write IO, using O_DIRECT and by default, sysbench is using 16KiB block size. The result on the direct storage (without Kubernetes):

Throughput:
        read, MiB/s:                    1691.30
        written, MiB/s:                 1129.30

OpenEBS local-pv

Now, let’s run this on the volume managed by OpenEBS local-pv (hostpath). For this, we need to do some prep work.

In the first step, I prepared a sysbench docker image which we can use to start a pod in Kubernetes and it will include sysbench and sysbench-tpcc benchmarks

In the second step, I define a volume:

cat pvc.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-hostpath-pvc2
spec:
  storageClassName: local-hostpath
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 203G

And define a pod to start an image with sysbench:

cat startpod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: sysbench-demo
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: local-hostpath-pvc2
  containers:
  - name: sysbench
    image: vadimtk/ubu-mysql-sysbench:latest
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello; sleep 10;done"]
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage
  restartPolicy: OnFailure

When the pod is running, we can shell into it as:

kubectl exec -it sysbench-demo -- bash

And execute the same commands as previously in  /mnt/store path. The results in this case:

Throughput:
        read, MiB/s:                    1700.06
        written, MiB/s:                 1134.10

As we can see, there is no measurable difference from the previous result, so, to be clear, there is no performance difference when using OpenEBS local-pv versus direct local storage.

OpenEBS zfs-localpv

Now let’s test OpenEBS zfs-localpv. ZFS adds extra features, but it comes with a performance overhead and I would like to measure it also. The full documentation on how to install and enable ZFS in your Kubernetes is here:

https://github.com/openebs/zfs-localpv

And I will just show the StorageClass I am using ( I am using 16k recordsize to match sysbench and InnoDB IO):

cat zfs-storageclass.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-zfspv
parameters:
  recordsize: "16k"
  compression: "off"
  dedup: "off"
  fstype: "zfs"
  poolname: "zfspv-pool"
provisioner: zfs.csi.openebs.io

And then running a similar experiment as previously described, I get the following results:

Throughput:
        read, MiB/s:                    1149.17
        written, MiB/s:                 767.14

And to summarize all results in the single table:

 

	Throughput, MiB/sec	Ratio to direct
Direct read	1691.30	–
Localpv read	1700.06	1
Zfs-localpv read	1149.17	0.67
Direct write	1129.30	–
Localpv write	1134.10	1
Zfs-localpv write	767.14	0.66

 

Conclusions

I want to state that there is no performance penalty on IO when using OpenEBS localpv volume manager. ZFS-localpv is a different story, and ZFS comes with additional features, but it comes with the cost of about 33% of the performance hit I observe in my results.

by Vadim Tkachenko via Percona Database Performance Blog