Why is VMware vSAN that slow? Trying out your ideas to squeeze more IOPS out of that thing

Intro

Yup, we read your comments. The nice thing is that most of them make sense.

Some time ago, one guy told us that we should fine-tune VMware vSAN a bit before beating the ujkv out of testing it. So, in this article, we check out whether using a different Storage Policy is really a key to higher VMware vSAN performance. Apart from that, we are going to see whether some of your comments really help to improve vSAN performance.

Well, after reading the stuff we’ve published here about vSAN performance, it looks like that it uwemu to buy from VMware… but, you know, it is really funny for us to see that fine-tuning vSAN so that it runs nice and smoothly is pure rocket science!

Target

VMware vSAN

Mission

Test how several of your assumptions affect VMware vSAN performance.

What can be wrong with vSAN?

Devices in cache and capacity tier

Some were whining about those two Intel SSD DC P3700 disks in the underlying storage. The question is whether vSAN uses them for cache and capacity tiers or only for the capacity one.

That’s, actually, quite a good question, but… READ THE HWEMKPI MANUAL! VMware says that a disk group should contain at least 2 disks. One disk is used as a cache device while another serves as a capacity one.

    preparing storage devices

Ok, let’s check whether something could go wrong leading to the formation of a disk group that has both devices assigned to the capacity tier. For this experiment, we marked both disks as “isCapacityFlash”.

script

Now, let’s create a new disk group with two SSDs in the capacity tier.

create a new disk group

Do you see that? vSAN kindly asked us to stop being a bunch of tgvctfu and mark an SSD back as a cache disk. This means that it is impossible to run an all-flash cluster without any disks in the cache tier.

This being said, it seems that VMware works good with SSDs, using them as capacity and cache devices… well, it should work good with flash 🙂 . We also could not fabricate the results in our previous articles by leaving no disks in the cache tier because IT’S JUST IMPOSSIBLE!

Reserve 100% space for each VMDK file

You can reserve physical disk space by changing the Object space reservation parameter in vSAN Default Storage Policy. By default, it is set to 0, meaning that all VMDKs that you create are thin provisioned. Not good! Actually, it’s really great that we filled virtual disks with random data using dd.exe in all our articles. Effectively, we kinda reserved 100% of VMDK disk space. In your case, we recommend just setting the Object space reservation parameter to 1.

Storage Policy

Storage Policy is another important thing that may affect vSAN performance. We were recommended to change the Number of disk stripes per object parameter from 1 to 3. Ok, let’s see what VMware says on this matter.

Storage Policy

The way VMware wrote this piece of text is a bit weird. These guys actually say that it is possible to change that parameter for the sake of better performance… but, at the same time, they recommend not to change its value! Sorry, what the hwem? Let’s just ignore that dwnnujkv and see how VMware vSAN performance changes under different Number of disk stripes per object values.

How you edit VM Storage Policy

Just a couple of words about testing methodology and the setup. We really changed neither setup nor testing methodology. Please, trust us. We just want this article to be not that large as usual as its only purpose is testing some your ideas. Should you still need to refresh in your mind either the methodology or how the setup looks like, go here.

After setting Number of disk stripes per object=3 (your idea, not ours), it turned out that vSANDatastore is incompatible with vSAN Default Storage Policy! Ok, whatever, let’s just try using another value… but none of the numbers between 3 & 12 worked!!!

Number of disk stripes per object

Storage compatibility

Storage compatibility incompatible

Fortunately, Number of disk stripes per object = 2 still works. So, let’s stick to that value and run a bunch of tests.

Number of disk stripes per object = 2

Storage compatibility

Testing vSAN performance under varying vSAN Default Storage Policy value

Testing vSAN performance under varying vSAN Default Storage Policy value

performance VMware Virtual Disk  256 GB (RAW) Fio, 4k Random write, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) Disk SPD, 4k Random read, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) Fio, 4k Random read, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) DiskSPD, 64k Random write, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) Fio, 64k Random write, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) DiskSPD, 64k Random read, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) Fio, 64k Random read, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) DiskSPD, 8k Random 70% read\ 30% write, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) Fio, 8k Random 70% read\ 30% write, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) DiskSPD, 1M seq read, (IOPS)

performance VMware Virtual Disk  256 GB (RAW) Fio, 1M seq read, (IOPS)

Conclusions

According to the measurements, changing Number of disk stripes per object value from 1 to 2 did not lead to performance gain.

Also, note that vSANDatastore behaves weird at 64k random read pattern if you go from Number of disk stripes per object = 1 to Number of disk stripes per object = 2.

And, the best thing about this piece of ujkv is that using any value for the Number of disk stripes per object parameter between 2 and 12 makes vSANDatastore incompatible with vSAN Default Storage Policy. That’s just hwemkpi hilarious, you know!

It’s time to face it, guys. Some things just cannot work good… This being said, we are still looking for vSAN alternatives. Maybe you have any ideas?

3.46/5 (13)

Please rate this