Does Number of disk stripes per object have something to do with VMware vSAN performance on the mixed NVMe-SSD underlying storage?

We got a comment on our previous article about VMware vSAN. The reader said that setting a different Number of disk stripes per object value is a key to higher VMware vSAN performance. Today, we check whether this parameter can alter vSAN performance… even though VMware warns against doing that!

comics

Introduction

There was a guy on https://www.cultofanarchy.org/can-vmware-vsan-recoup-mixed-nvme-ssd-underlying-storage-will-just-mess-things/ comments who said that if we did some black magic on VMware vSAN settings, its performance wouldn’t be a mediocre piece of ujkv. He referred vSAN stripes (How, When and Why?) trying to convince us to try these things:

  • Changing the Number of disk stripes per object value to 4
  • Increasing test disk volume to 255 GB
  • Installing ESXi 6.7 U1
  • Disabling checksumming

Ok, he did it… let’s see what happens if we apply [some of] these changes! Yes, we do not give a ujkv about what VMware says. We played around with the number of disk stripes per object some time ago, and nobody was hurt (except Paul: he was drunk as hwem and had a terrible headache in the morning after 🙂 ).

So, today, we are going to redo our tests with changing the number of disk stripes per object. In the next article, we’ll try out Automatic Striping.

The brief: What we did (and what we didn’t)?

Today, we investigated how the overall number of IOPS that a vSAN cluster can deliver changes under Number of disk stripes per object = 2; 4; 8. The performance measured today was compared to performance observed before with Number of disk stripes per object = 1.

We did not disable checksumming. WHY?! We wanted to see the maximum possible VMware vSAN performance under the close-to-production conditions NOT IN A PERFECT SETUP! That’s why we left the Disable object checksum option intact.

Edit VM

We carried out this study on ESXi 6.5 U1 to avoid incomparability with any other in this series.

We did not change the size of the tested disk either since we set the Number of disk stripes per object manually. Let’s hope this parameter alone really does some miracles to vSAN performance.

This all being said, today, we have an article addressing the impact of the number of disk stripes per object on vSAN performance. Let’s roll!

Hardware toys

ESXi Host #1, ESXi Host #2, ESXi Host #3, ESXi Host #4

Dell R730, CPU 2x Intel Xeon E5-2683 v3 @ 2.00 GHz (14 stones/CPU), RAM 64GB

Host Storage: 1x Intel SSD DC P3700 2TB, 4x Intel SSD DC S3500 480GB

LAN: 2x Mellanox ConnectX-4 100Gbit/s CX465A

Hypervisor: VMware ESXi 6.5 Update 1

Here’s the setup interconnection diagram.

VMware vSAN

Methodology

We’ve carried out this study just as we did it before. The things which we did not do this time:

  • Finding the optimal test utility launching parameters (the number of threads and outstanding I/O). We used threads = 4 and Outstanding I/O = 8.
  • Measuring single Intel SSD DC P3700 2TB performance. We had nothing going wrong in our lab for a while, so let’s just reuse these numbers!
  • Measuring Intel SSD DC S3500 480GB performance in the bare-metal Windows Server 2016 environment.

The testing methodology also did not change. We’ve judged on VMware vSAN performance based on the overall cluster performance observed while populating it with VMs. The more VMs – the better! We kept on spawning VMs in the cluster until its performance hit the saturation point. It was populated with VMs according to the “circle lgtm” principle 🙂 : every new VM was pinned to a different node than a previous one. Each VM had its own 80GB virtual disk (“data” disk) on a dedicated vsanDatastore.

Just as usual, today, we’ve carried out performance measurements using DiskSPD v2.0.20a and FIO v3.6.

All tests were done under these patterns:

  • 4k random write
  • 4k random read
  • 64k random write
  • 64k random read
  • 8k random 70%read/30%write
  • 1M sequential read

Does the number of disk stripes per object matter?

Number of disk stripes per object = 1;2;4;8. DiskSDP

Performance VMware Virtual Disk 80GB (RAW)4k random write (IOPS), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)4k random write (MB/s), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)4k random read (IOPS), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)4k random read (MB/s), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)64k random write (IOPS), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)64k random write (MB/s), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)64k random read (IOPS), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)64k random read (MB/s), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)8k random 70%read/30%write (IOPS), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)8k random 70%read/30%write (MB/s), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)1M seq read (IOPS), DiskSPD

Performance VMware Virtual Disk 80GB (RAW)1M seq read (MB/s), DiskSPD

Stripe = 1

DiskSPD, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 37321 146 1.22 74805 292 0.43 5288 331 6.05 25996 1625 1.23 32721 256 0.98 1270 1270 25.19
2x VM 16474 64 3.88 127485 498 0.50 5176 323 12.72 35167 2198 1.82 44164 345 1.45 2189 2189 29.24
3x VM 21320 83 4.51 155134 606 0.63 6768 423 14.19 48181 3011 2.00 56843 444 1.69 3167 3167 31.54
4x VM 24003 94 7.12 175230 684 0.76 7170 448 18.44 45502 2844 2.87 58657 458 2.65 3784 3784 35.32
5x VM 31700 124 6.81 180237 704 0.93 9453 591 20.21 59389 3712 2.78 79683 623 2.89 4681 4681 35.68
6x VM 27479 107 7.86 174365 681 1.17 8095 506 25.73 64816 4051 3.05 64252 502 3.21 5183 5183 38.35
7x VM 33937 133 7.58 170299 665 1.42 9705 607 25.13 70022 4376 3.34 69360 542 3.46 5102 5102 46.04
8x VM 34920 136 8.99 164924 644 1.67 9860 616 30.59 66758 4172 4.07 63058 493 4.37 5009 5009 54.44
9x VM 34787 136 8.92 201961 789 2 10639 665 30.17 81830 5114 3.77 86490 676 3.91 5129 5129 62.78
10x VM 36610 143 10.57 210931 824 1.82 11183 699 36.36 82664 5166 4.62 74597 583 5.75 5676 5676 64.62
11x VM 38620 151 10.47 203692 796 1.83 11222 701 33.37 88178 5511 4.23 74275 580 5.62 6048 6048 59.54
12x VM 44384 173 9.98 213144 833 1.84 12665 792 35.62 89424 5589 4.47 74270 580 6.26 6329 6329 61.79
13x VM 50620 198 10.71 304747 1190 1.40 13577 849 32.24 99280 6205 4.27 90044 703 5.67 6245 6245 68.03
14x VM 65040 254 10.74 283837 1109 1.59 14021 876 33.88 102274 6392 4.44 118865 929 4.82 6007 6007 75.18
15x VM 62352 244 12.48 275757 1077 1.76 13479 842 38.32 103343 6459 4.76 105093 821 6.41 6184 6184 81.80
16x VM 61615 241 11.00 282526 1104 1.84 14088 881 38.39 95404 5963 5.49 110421 863 5.62 6042 6042 87.13
17x VM 64725 253 9.77 282576 1104 1.72 14491 906 39.96 103583 6474 4.79 81841 639 8.96 7598 7598 73.71
18x VM 69179 270 13.68 272617 1065 2.37 14146 884 47.26 99097 6194 6.47 105000 820 8.16 6627 6627 96.17
19x VM 69321 271 11.91 275787 1077 1.88 13953 872 46.69 99858 6241 5.63 81771 639 10.73 7598 7598 84.50
20x VM 61662 241 16.32 245407 959 2.63 13775 861 47.86 91760 5735 7.10 114115 892 6.86 6090 6090 107.14

Stripe = 2

DiskSPD, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 6193 24 5.17 62638 245 0.51 4729 296 6.77 25423 1589 1.26 22184 173 1.44 1177 1177 27.19
2x VM 24059 94 2.97 129779 507 0.49 8849 553 7.23 48884 3055 1.31 50448 394 1.27 2620 2620 24.44
3x VM 29422 115 4.00 190188 743 0.51 9307 582 11.02 62151 3884 1.55 64724 506 1.51 3420 3420 29.00
4x VM 50116 196 3.07 224903 879 0.59 10709 669 12.11 69334 4333 1.88 92601 723 1.43 4696 4696 28.49
5x VM 53137 208 5.38 235951 922 0.78 10913 682 14.87 77289 4831 2.16 93773 733 1.76 4899 4899 36.44
6x VM 73770 288 3.42 274081 1071 0.79 15509 969 13.48 92447 5778 2.16 123052 961 1.64 5564 5564 37.25
7x VM 72092 282 6.50 230837 902 0.85 12243 765 17.06 78225 4889 2.28 95709 748 2.41 5384 5384 32.36
8x VM 64688 253 10.29 243573 951 1.23 12784 799 22.14 103440 6465 2.61 103102 805 2.83 5996 5996 45.45
9x VM 48298 189 7.14 257227 1005 1.18 12923 808 22.91 94250 5891 3.11 105350 823 3.09 6118 6118 50.60
10x VM 63383 248 7.71 253599 991 1.33 13319 832 24.82 88732 5546 3.71 109132 853 3.24 6016 6016 54.12
11x VM 56928 222 9.87 249894 976 1.46 11791 737 30.99 92366 5773 3.89 109559 856 3.78 5934 5934 61.49
12x VM 70892 277 9.28 264511 1033 1.50 15235 952 26.93 98559 6160 4.00 136457 1066 3.64 6469 6469 59.96
13x VM 68725 268 10.24 254820 995 1.83 13770 861 34.20 101640 6352 4.62 112745 881 5.32 6764 6764 67.59
14x VM 72277 282 12.55 260911 1019 1.78 14757 922 33.18 100206 6263 4.82 121922 953 4.81 6947 6947 65.62
15x VM 58915 230 11.38 253029 988 1.98 13265 829 36.99 99864 6242 5.19 112064 875 5.53 6920 6920 71.52
16x VM 65440 256 10.21 257379 1005 2.06 14326 895 36.10 101539 6346 5.33 100258 783 5.53 7216 7216 72.69
17x VM 54536 213 11.92 244396 955 2.35 14156 885 38.58 94818 5926 5.87 103594 809 6.49 7005 7005 78.53
18x VM 54584 213 13.60 249442 974 2.46 13314 832 45.38 97020 6064 6.17 109687 857 7.42 7029 7029 83.28
19x VM 72484 283 13.75 251305 982 2.59 14917 932 41.97 98223 6139 6.53 93341 729 8.21 7837 7837 79.26
20x VM 59434 232 15.39 239593 936 2.78 14093 881 47.55 95112 5944 6.91 87298 682 9.11 7141 7141 90.80

Stripe = 4

DiskSPD, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 13525 53 2.37 64761 253 0.49 9759 610 3.28 26519 1657 1.21 25242 197 1.27 1141 1141 28.05
2x VM 44841 175 1.43 137465 537 0.47 15232 952 4.20 45401 2838 1.41 59053 461 1.08 2243 2243 28.54
3x VM 74692 292 1.29 185674 725 0.52 15211 951 6.31 69227 4327 1.39 99737 779 0.97 3430 3430 28.01
4x VM 80433 314 1.63 215865 843 0.60 15222 951 8.88 81462 5091 1.58 98100 766 1.33 5071 5071 25.28
5x VM 82055 321 2.01 219057 856 0.75 15058 941 10.63 80192 5012 2.00 118356 925 1.35 5643 5643 28.62
6x VM 88698 346 2.34 220366 861 0.91 14584 912 13.87 79699 4981 2.42 118623 927 1.62 5441 5441 36.68
7x VM 88354 345 2.87 238677 932 0.98 16622 1039 15.08 88902 5556 2.65 125447 980 1.91 5659 5659 42.41
8x VM 84194 329 3.94 257547 1006 1.00 16828 1052 16.02 88852 5553 2.96 137537 1075 1.95 6270 6270 42.50
9x VM 84759 331 4.82 259066 1012 1.14 15236 952 19.48 90881 5680 3.37 123698 966 2.60 6329 6329 47.34
10x VM 75953 297 6.37 228766 894 1.45 13527 845 24.59 90509 5657 3.82 101235 791 3.93 6615 6615 51.24
11x VM 65583 256 7.39 239342 935 1.64 14196 887 30.67 95400 5962 4.02 93396 730 5.05 6856 6856 56.84
12x VM 71169 278 7.36 245456 959 1.72 13442 840 31.70 94019 5876 4.41 95509 746 5.28 6987 6987 58.98
13x VM 72284 282 8.63 229314 896 2.03 14350 897 30.51 94402 5900 4.86 96524 754 5.81 7263 7263 57.97
14x VM 59596 233 9.00 233934 914 2.13 12916 807 35.75 92107 5757 5.87 92893 726 6.63 7206 7206 63.60
15x VM 54139 211 10.16 233892 914 2.21 13320 833 36.84 75617 4726 7.64 88022 688 7.32 6679 6679 73.34
16x VM 57256 224 10.83 227351 888 2.37 13912 870 37.42 78610 4913 7.93 93055 727 7.35 7493 7493 69.67
17x VM 45531 178 12.01 229575 897 2.48 14135 883 45.60 75384 4712 8.19 97199 759 7.44 7274 7274 75.71
18x VM 49456 193 11.82 227912 890 2.64 14883 930 42.86 74308 4644 8.40 97193 759 7.40 7180 7180 81.05
19x VM 49610 194 12.85 225222 880 2.80 15552 972 44.13 68088 4256 10.92 102549 801 7.73 7411 7411 82.87
20x VM 50978 199 13.52 194460 760 3.46 15343 959 43.91 54373 3398 15.28 91521 715 13.02 7675 7675 86.21

Stripe = 8

DiskSPD, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 16799 66 1.90 67719 265 0.47 11076 692 2.89 26593 1662 1.20 35161 275 0.91 1144 1144 27.98
2x VM 27516 107 2.33 136272 532 0.47 22086 1380 2.90 52692 3293 1.22 83512 652 0.77 2233 2233 28.66
3x VM 65629 256 1.46 198827 777 0.48 19772 1236 4.85 71995 4500 1.34 113426 886 0.85 3346 3346 28.71
4x VM 89125 348 1.44 225186 880 0.57 18464 1154 6.93 88904 5556 1.44 136063 1063 0.94 4522 4522 28.32
5x VM 87168 341 1.84 260658 1018 0.61 16874 1055 9.48 93513 5845 1.71 143151 1118 1.12 5767 5767 27.84
6x VM 74566 291 2.57 252396 986 0.76 16909 1057 11.36 77189 4824 2.49 138630 1083 1.38 6621 6621 29.08
7x VM 68581 268 3.27 243876 953 0.92 15499 969 14.52 90829 5677 2.47 120846 944 1.85 6838 6838 33.32
8x VM 66950 262 3.83 237187 927 1.08 14490 906 17.93 89787 5612 2.85 100367 784 2.55 6706 6706 39.08
9x VM 64914 254 4.44 207220 809 1.39 15784 987 18.42 71575 4473 4.03 89184 697 3.23 7132 7132 41.40
10x VM 65385 255 4.90 216163 844 1.48 16795 1050 19.27 59492 3718 5.39 86355 675 3.71 7580 7580 42.62
11x VM 59645 233 5.91 206529 807 1.71 17981 1124 19.74 42736 2671 8.27 72170 564 4.88 7742 7742 45.62
12x VM 56384 220 6.87 179156 700 2.14 19050 1191 20.21 41005 2563 9.38 40111 313 9.87 7453 7453 51.66
13x VM 49625 194 8.62 187046 731 2.23 20366 1273 20.43 39209 2451 10.62 35095 274 11.98 7335 7335 56.89
14x VM 54614 213 8.57 201999 789 2.22 20710 1294 21.63 38511 2407 11.64 33813 264 13.28 7631 7631 58.93
15x VM 52928 207 9.28 173310 677 2.78 21143 1321 22.74 36746 2297 13.10 33926 265 14.63 7431 7431 65.27
16x VM 50255 196 10.99 153865 601 3.33 21248 1328 24.22 38556 2410 13.29 33582 262 15.52 7233 7233 71.39
17x VM 49156 192 12.27 158482 619 3.44 21127 1320 25.99 38268 2392 14.24 34081 266 16.39 7387 7387 74.48
18x VM 49598 194 13.61 188159 735 3.06 21494 1343 27.19 36818 2301 15.68 34322 268 17.07 7460 7460 78.40
19x VM 45967 180 15.20 177181 692 3.44 21468 1342 28.70 36718 2295 16.60 34443 269 17.83 7810 7810 79.05
20x VM 47554 186 29.35 172310 673 3.92 21416 1339 34.71 36968 2311 17.55 33800 264 37.95 7391 7391 88.06

Number of disk stripes per object = 1;2;4;8. FIO

Performance VMware Virtual Disk 80GB (RAW)4k random write (IOPS), FIO

Performance VMware Virtual Disk 80GB (RAW)4k random write (MB/s), FIO

Performance VMware Virtual Disk 80GB (RAW)4k random read (IOPS), FIO

Performance VMware Virtual Disk 80GB (RAW)4k random read (MB/s), FIO

Performance VMware Virtual Disk 80GB (RAW)64k random write (IOPS), FIO

Performance VMware Virtual Disk 80GB (RAW)64k random write (MB/s), FIO

Performance VMware Virtual Disk 80GB (RAW)64k random read (IOPS), FIO

Performance VMware Virtual Disk 80GB (RAW)64k random read (MB/s), FIO

Performance VMware Virtual Disk 80GB (RAW)8k random 70%read/30%write (IOPS), FIO

Performance VMware Virtual Disk 80GB (RAW)8k random 70%read/30%write (MB/s), FIO

Performance VMware Virtual Disk 80GB (RAW)1M seq read (IOPS), FIO

Performance VMware Virtual Disk 80GB (RAW)1M seq read (MB/s), FIO

Stripe = 1

FIO, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 36728 143 1.31 73839 288 0.42 5887 368 5.41 24459 1529 1.29 44465 347 0.83 1228 1229 25.96
2x VM 17334 68 3.68 124033 485 0.50 5453 341 11.71 29422 1839 2.17 42826 335 2.03 2104 2107 30.74
3x VM 21387 84 4.59 156965 613 0.61 7381 462 12.99 43852 2741 2.19 58303 456 2.26 2982 2987 33.18
4x VM 24936 97 6.95 176985 691 0.73 7647 478 17.42 42044 2628 3.08 63771 498 3.64 3576 3585 36.97
5x VM 33144 130 6.68 173445 678 0.95 10027 627 17.86 50661 3167 3.25 76416 597 3.80 4405 4416 37.53
6x VM 27637 108 7.33 160409 627 1.27 9151 573 21.61 52961 3311 3.76 67238 525 4.21 4820 4831 41.94
7x VM 34261 134 7.77 176534 690 1.37 10373 649 23.50 56066 3505 4.20 74560 583 4.33 4897 4910 47.59
8x VM 36868 144 8.45 166919 652 1.62 11003 689 24.94 60193 3763 4.42 67104 524 5.04 4760 4774 57.50
9x VM 36592 143 8.31 200380 783 1.64 12039 754 24.89 72125 4509 4.26 94778 741 4.80 5021 5039 61.34
10x VM 42598 166 7.98 214333 837 1.60 13342 835 25.13 76218 4765 4.30 88089 689 5.26 5661 5677 58.41
11x VM 42272 165 8.48 210388 822 1.75 12902 808 27.54 83447 5217 4.29 87940 687 5.42 5757 5779 63.55
12x VM 48477 189 9.38 229942 898 1.73 13685 857 31.95 85426 5340 4.60 90987 711 5.88 6484 6510 61.35
13x VM 59633 233 8.70 279691 1093 1.51 16004 1002 27.36 90761 5674 4.64 133062 1040 4.82 5501 5528 76.18
14x VM 71341 279 9.30 268295 1048 1.71 15347 961 30.83 95124 5947 4.84 128947 1008 5.55 5820 5848 78.20
15x VM 67758 265 10.11 274764 1073 1.79 14895 933 33.35 89817 5615 5.46 117456 918 6.94 5900 5930 83.93
16x VM 59186 231 10.30 274303 1072 1.96 14748 924 35.96 83899 5246 6.25 112571 880 7.01 6043 6077 89.58
17x VM 62515 244 10.86 278976 1090 2.04 14649 918 39.16 94839 5930 5.83 109312 854 8.39 6201 6234 92.259
18x VM 63708 249 11.96 275047 1075 2.28 14440 905 43.49 96645 6043 6.07 104115 814 10.28 6310 6348 95.24
19x VM 60042 235 14.02 263738 1030 2.43 14747 924 43.90 85272 5332 7.17 109609 857 9.71 5707 5744 107.75
20x VM 68223 266 13.19 285043 1114 2.38 14477 907 46.05 93482 5845 7.00 104276 815 10.23 6288 6330 106.40

Stripe = 2

FIO, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 25306 99 10.10 53257 208 4.79 6732 421 38.00 19227 1202 13.30 35560 278 7.65 1168 1171 218.42
2x VM 51192 200 10.82 131374 513 3.89 9252 579 55.30 38990 2437 13.12 71963 562 7.51 2329 2333 219.35
3x VM 48562 190 19.98 196375 767 3.95 9854 616 82.85 49140 3072 15.72 84745 662 11.20 3400 3408 225.55
4x VM 47740 187 23.89 234702 917 4.71 10645 666 97.12 60893 3806 17.52 103878 812 11.39 4563 4569 231.23
5x VM 53867 210 31.71 239913 937 6.33 11621 727 110.43 67589 4225 20.78 109624 857 13.00 4774 4783 295.82
6x VM 73749 288 23.46 281947 1101 6.15 16585 1037 100.31 82480 5156 19.96 138683 1084 11.95 5474 5490 302.29
7x VM 79407 310 46.00 259383 1013 7.69 14220 890 145.64 78614 4914 24.38 118311 925 20.74 5980 5995 309.23
8x VM 71077 278 61.02 278451 1088 7.96 14157 886 150.77 90380 5650 23.86 124226 971 19.88 6176 6188 351.10
9x VM 54492 213 49.24 260516 1018 9.52 13993 876 169.92 92564 5786 25.71 118217 924 24.46 5986 6001 400.15
10x VM 68159 266 54.47 272965 1066 10.32 14132 884 187.32 89350 5586 29.93 114143 892 26.44 6291 6311 413.56
11x VM 64486 252 64.57 253523 990 12.20 13625 853 208.49 86322 5396 35.53 101357 792 33.82 6772 6795 432.66
12x VM 74021 289 70.08 269711 1054 11.62 14668 918 212.38 97159 6074 32.98 130803 1022 31.62 6906 6931 447.77
13x VM 73615 288 65.28 262136 1024 13.94 15507 971 226.85 87783 5488 42.67 123875 968 47.08 7075 7100 474.54
14x VM 73958 289 80.04 274548 1073 13.57 14716 922 260.52 98955 6186 39.25 121177 947 41.94 7149 7176 504.84
15x VM 65493 256 83.67 250586 979 16.65 14580 913 275.13 95843 5992 44.50 113317 886 51.50 7531 7555 515.83
16x VM 63116 247 75.44 264877 1035 17.37 14441 905 289.60 91086 5695 47.42 107672 842 53.07 7694 7729 533.20
17x VM 56719 222 88.58 254950 996 17.65 14362 900 307.82 90346 5649 49.08 102785 803 57.62 7755 7790 564.77
18x VM 62646 245 91.06 261369 1021 18.30 14475 907 331.08 83689 5233 58.11 105785 827 64.23 7770 7806 597.23
19x VM 65562 256 101.41 243557 952 22.34 13597 852 367.74 91903 5746 53.81 106450 832 62.70 8105 8145 604.52
20x VM 67557 264 103.08 261536 1022 21.17 13938 874 381.27 92860 5806 56.38 106786 835 68.93 8358 8399 615.80

Stripe = 4

FIO, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 11535 45 22.18 65157 255 3.92 9669 604 26.45 19625 1227 13.03 36340 284 7.48 1172 1176 217.40
2x VM 47648 186 10.76 137108 536 3.74 14070 880 36.48 39724 2483 12.87 75872 593 7.15 2248 2251 227.11
3x VM 79726 311 9.63 195598 764 3.92 13726 858 55.91 59899 3744 12.83 108306 846 7.67 3593 3596 213.30
4x VM 83992 328 12.56 207359 810 5.25 15459 967 73.60 75229 4702 13.60 119906 937 8.70 4886 4895 209.29
5x VM 89482 350 14.83 223727 874 5.82 13605 851 94.01 73664 4605 17.48 139039 1086 9.13 5491 5501 234.16
6x VM 85770 335 18.07 226887 886 7.20 14975 937 113.44 75277 4706 22.31 142594 1114 10.98 5755 5769 273.42
7x VM 93162 364 21.14 253434 990 7.45 17052 1067 121.12 81164 5074 24.86 139973 1094 14.49 5789 5802 323.98
8x VM 93555 366 26.32 259623 1014 7.98 16946 1060 122.32 81852 5117 26.23 130105 1017 18.84 6095 6110 347.63
9x VM 87206 341 36.47 253276 989 9.53 14990 938 158.58 85796 5363 28.64 104625 818 31.43 6273 6288 380.94
10x VM 80084 313 45.01 234107 915 13.17 13434 841 199.35 80264 5018 35.22 101935 797 38.15 6512 6531 412.84
11x VM 73855 289 49.41 240786 941 14.23 15536 972 200.98 68151 4261 50.21 101312 792 41.23 7316 7335 403.34
12x VM 75416 295 53.41 246039 961 14.14 13978 875 222.25 71547 4473 50.47 103740 811 43.07 7065 7092 449.47
13x VM 76222 298 55.85 217686 850 16.87 15059 943 221.69 68489 4282 60.67 107959 844 43.46 7274 7302 462.76
14x VM 65937 258 64.39 232814 910 17.20 14983 938 239.12 51547 3223 104.49 106277 831 52.77 7329 7355 490.39
15x VM 59397 232 76.25 215072 840 22.18 15389 964 249.73 50350 3149 115.19 75926 594 133.40 7333 7365 526.60
16x VM 66651 260 76.78 228005 891 21.01 16370 1025 251.09 50062 3131 115.81 73986 578 135.76 7926 7957 524.13
17x VM 64347 251 75.88 225066 879 21.09 16498 1033 294.33 45128 2823 117.91 93326 730 116.35 8253 8292 527.86
18x VM 66588 260 75.73 230272 900 20.97 17372 1088 292.88 44432 2779 111.99 104253 815 87.80 8535 8572 541.42
19x VM 67058 262 82.94 218929 855 23.15 17752 1112 299.56 42099 2634 133.33 94578 739 129.65 8526 8561 576.07
20x VM 69465 271 91.43 225271 880 24.90 17897 1121 312.33 37294 2333 149.51 97153 760 138.85 8600 8639 580.90

Stripe = 8

FIO, threads=4, Outstanding I/O=8
4k random write 4k random read 64k random write 64k random read 8k random 70%read/30%write 1M seq read
IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms) IOPS MB/s Latency (ms)
1x VM 16079 63 15.91 69308 271 3.68 15196 950 16.83 20757 1297 12.32 42462 332 6.36 1188 1190 214.79
2x VM 50026 195 10.22 137933 539 3.70 20523 1283 24.93 39900 2494 12.86 84703 662 6.31 2252 2254 226.93
3x VM 72339 283 10.61 200526 783 3.82 18294 1144 41.95 56579 3537 13.57 119553 934 6.64 3308 3315 231.47
4x VM 86921 340 11.77 264596 1034 3.86 17238 1078 59.36 69831 4365 14.67 156249 1221 6.54 4363 4369 234.31
5x VM 88977 348 14.37 272411 1064 4.73 16353 1023 78.32 76590 4788 16.73 156825 1225 8.18 6081 6090 210.49
6x VM 90702 354 16.93 229870 898 6.67 15291 957 100.37 86583 5412 17.74 105374 823 14.68 6563 6575 233.67
7x VM 78875 308 22.80 247010 965 7.26 15714 983 114.08 86071 5380 20.81 94030 735 19.29 7172 7186 250.92
8x VM 73212 286 28.10 226582 885 9.04 16809 1051 121.85 45171 2824 45.38 80546 629 25.78 7404 7427 278.42
9x VM 72816 284 31.90 202447 791 11.53 17745 1110 130.10 43054 2692 53.55 79829 624 29.31 7440 7457 312.80
10x VM 66428 260 39.37 166012 649 15.43 18048 1129 142.96 33475 2093 76.53 52531 411 54.35 7532 7553 339.76
11x VM 67618 264 42.82 163269 638 17.26 18420 1153 155.07 32095 2007 87.68 61608 482 47.02 7631 7656 368.07
12x VM 59573 233 54.34 185757 726 16.79 18590 1164 169.42 35498 2220 86.52 72697 568 44.02 7639 7666 400.91
13x VM 60066 235 59.54 151829 593 21.91 18689 1170 183.21 32925 2060 101.14 39444 308 90.97 7755 7781 428.98
14x VM 57924 226 67.70 165616 647 21.65 18781 1176 197.67 33506 2096 107.22 30916 242 123.86 7990 8021 449.90
15x VM 51755 202 82.21 187323 732 20.71 18562 1162 212.79 34123 2135 112.66 32072 251 126.74 8115 8150 473.13
16x VM 57004 223 94.26 169496 662 24.64 19845 1242 223.10 40341 2523 101.71 34521 269 70.87 8592 8619 482.02
17x VM 55593 217 81.00 147731 577 29.48 18879 1182 237.42 36690 2295 120.96 34544 279 86.57 8177 8213 429.97
18x VM 53329 208 105.44 159933 625 29.43 18711 1172 256.02 33043 2067 139.38 32408 254 152.90 8562 8593 540.66
19x VM 51964 203 110.03 178166 696 28.10 18285 1145 272.67 33471 2094 145.21 35588 279 147.35 8549 8586 570.86
20x VM 50927 199 147.02 186881 730 29.08 18551 1162 312.55 33515 2097 152.74 37298 292 241.06 8750 8790 589.24

Conclusions

All plots above clearly show that performance does depend on the number of disk stripes per object. Here, we used the performance with 1 disk stripe per object as a reference.

The parameter value impacts remarkably 4k random writes performance. With Number of disk stripes per object = 4 (matches the number of disks in host capacity tier), we got a great gain of IOPS until the 13th VM joined the cluster. Cluster performance peaked to 88-93 IOPS (6-7 VMs in the cluster). Well, that’s higher than vSAN performance with Number of disk stripes per object = 1! After spawning the 13th VM, though, cluster performance becomes even lower than while Number of disk stripes per object=1. With Number of disk stripes per object = 8, performance grows until the 4th VM gets on board (peak is around 90K IOPS). If you keep on adding VMs to the cluster under this parameter value, its overall performance just degrades.

Setting the Number of disk stripes per object to 2 or 4 is more beneficial for performance under 1M sequential reads than having 8 disk stripes per object. With parameter value set to 2 or 4, we observed the clear performance growth up to 19 VMs in the cluster.

Now let’s take a look at the performance under 64k random writes and 64k random reads. We observed the highest number of IOPS having Number of disk stripes per object = 8. But, under that parameter value, performance drops from 90K IOPS to just 40K once the 9th VM joins the cluster. With Number of disk stripes per object = 2, the cluster exhibited the most consistent performance growth under this pattern.

For 8k 70% read 30% write, things look the same. We observed the highest number of IOPS with Number of disk stripes per object = 8, but the most consistent performance gain was seen for 2 and 4 disk stripes per object.

All this being said, you should set the Number of disk stripes per object parameter value with respect to the number of disks in the host capacity tier. At this point, we’d like to go back to vSAN documentation warning against playing around with the parameter. TESTS ABOVE HAVE JUST PROVEN THAT IT IS BULLUJKV!!! How do you like that now, VMware? When vSAN stops being a just piece of ujkv that needs a techwizard to run smoothly? Maybe, introducing the mechanism for picking automatically the Number of disk stripes per object parameter will change the game…

4.88/5 (16)

Please rate this