Measuring OpenEBS Local Volume Performance Overhead in Kubernetes
I wrote about the usage of OpenEBS to manage local storage in Kubernetes in my previous posts:
The primary reason to use the local storage is performance and the logical question is then: How big is the performance overhead if we use OpenEBS to manage the local storage?
To look into this I will use sysbench fileio test on the NVMe storage Intel® SSD DC P4610.
The command to prepare data:
sysbench fileio prepare --file-num=25 --file-total-size=100G --file-extra-flags=direct
The command to run test:
sysbench fileio run --threads=32 --file-async-backlog=1024 --file-num=25 --file-total-size=100G --file-io-mode=async --file-test-mode=rndrw --file-extra-flags=direct --time=120 --report-interval=1 --file-fsync-freq=0 --file-fsync-end=off
This means I will use Async IO to perform read-write IO, using O_DIRECT and by default, sysbench is using 16KiB block size. The result on the direct storage (without Kubernetes):
Throughput: read, MiB/s: 1691.30 written, MiB/s: 1129.30
Now, let’s run this on the volume managed by OpenEBS local-pv (hostpath). For this, we need to do some prep work.
In the first step, I prepared a sysbench docker image which we can use to start a pod in Kubernetes and it will include sysbench and sysbench-tpcc benchmarks
In the second step, I define a volume:
cat pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-hostpath-pvc2 spec: storageClassName: local-hostpath accessModes: - ReadWriteOnce resources: requests: storage: 203G
And define a pod to start an image with sysbench:
cat startpod.yaml apiVersion: v1 kind: Pod metadata: name: sysbench-demo spec: volumes: - name: local-storage persistentVolumeClaim: claimName: local-hostpath-pvc2 containers: - name: sysbench image: vadimtk/ubu-mysql-sysbench:latest command: ["/bin/sh"] args: ["-c", "while true; do echo hello; sleep 10;done"] volumeMounts: - mountPath: /mnt/store name: local-storage restartPolicy: OnFailure
When the pod is running, we can shell into it as:
kubectl exec -it sysbench-demo -- bash
And execute the same commands as previously in /mnt/store path. The results in this case:
Throughput: read, MiB/s: 1700.06 written, MiB/s: 1134.10
As we can see, there is no measurable difference from the previous result, so, to be clear, there is no performance difference when using OpenEBS local-pv versus direct local storage.
Now let’s test OpenEBS zfs-localpv. ZFS adds extra features, but it comes with a performance overhead and I would like to measure it also. The full documentation on how to install and enable ZFS in your Kubernetes is here:
And I will just show the StorageClass I am using ( I am using 16k recordsize to match sysbench and InnoDB IO):
cat zfs-storageclass.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-zfspv parameters: recordsize: "16k" compression: "off" dedup: "off" fstype: "zfs" poolname: "zfspv-pool" provisioner: zfs.csi.openebs.io
And then running a similar experiment as previously described, I get the following results:
Throughput: read, MiB/s: 1149.17 written, MiB/s: 767.14
And to summarize all results in the single table:
|Throughput, MiB/sec||Ratio to direct|
I want to state that there is no performance penalty on IO when using OpenEBS localpv volume manager. ZFS-localpv is a different story, and ZFS comes with additional features, but it comes with the cost of about 33% of the performance hit I observe in my results.
by Vadim Tkachenko via Percona Database Performance Blog