SDDC

Bug in VSAN 6.2: De-dupe scanning running on hybrid datastores

UPDATE
VMware has posted a KB about this, which I did not realize at the time of writing the blog. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146267

We’ve been testing out VSAN here at work and noticed that one of the clusters we rolled out had serious latency issues. We initially blamed the application running on the hosted VMs, but when it continued to get worse we finally opened a case with VMware. Here’s a chart of the kind of stats we were seeing (courtesy of SexiGraf):

Read latency in particular was very high on the datastore level, IOPS weren’t great, and Read Cache Hit Rate was low. We also saw that read and write latency was high on the VM level. After we opened a ticket with VMware, they discovered an undocumented bug in VSAN 6.2 where deduplication scanning is running even though deduplication is turned off (and actually unsupported in hybrid mode VSAN altogether). They provided the following solution:

For each host in the VSAN cluster: 1. Enter maintenance mode 2. SSH to the host and run: "esxcfg-advcfg -s 0 /LSOM/lsomComponentDedupScanType" 3. Reboot the host

After we applied the fix, the cluster rebalanced for a little while and came back looking much, much better. In the below graph, you can see right when the fix was applied and see read latency drop, IOPS increase, and read cache hit rate jump to the high 90-percents:

And for good measure, this is how it’s looked since:

So to summarize, if you are running hybrid VSAN 6.2, you should definitely check your latency and read cache hit rate. If you’re experiencing high latency and poor read cache hit rate, go through and change /LSOM/lsomComponentDedupScanType on all your hosts to 0. I can’t take credit for actually discovering this, so thank you to my coworker @per_thorn for tracking it down. And thank you @thephuck for letting me write it up on this blog!

–@vDudeJon

How to create an NSX CLI user, API user & set up NSX Plugin for vROps

TL-DR: See below for details on these commands

Create a local user in the NSX Manager’s CLI, then use the API to grant CLI privileges to that user.

Here’s how using a linux machine:
ssh admin@[nsxmanagerIP] enable config t user vrops-readonly password plaintext notrealpassword user vrops-readonly privilege web-interface
Log out of the NSX Manager (type exit) and stay logged into the linux machine.
Create cli-auditor.xml that contains this (replace brackets with greater/less than):
[?xml version="1.0" encoding="ISO-8859-1" ?] [accessControlEntry] [role]auditor[/role] [resource] [resourceId]globalroot-0[/resourceId] [/resource] [/accessControlEntry]
Add the user as an auditor in the NSX Manager as a CLI user:
curl -i -k -u 'admin:password' -H "Content-Type: application/xml" -X POST --data "@cli-auditor.xml" https://nsxmanagerip/api/2.0/services/usermgmt/role/vrops-readonly?isCli=true
Add your domain/vCenter user as an auditor in the NSX Manager (NOT as a CLI user):
curl -i -k -u 'admin:password' -H "Content-Type: application/xml" -X POST --data "@cli-auditor.xml" https://nsxmanagerip/api/2.0/services/usermgmt/role/[email protected]?isCli=false

Details for creating the NSX CLI user for vROps

Here’s the error

While building a new environment for my lab, I ran across an interesting thing yesterday.

I looked at my cluster’s VSAN health and saw this error:

It’s complaining that my hosts don’t have matching Virtual SAN advanced configuration items.

If you click on that error, you’ll see at the bottom where it shows comparisons of hosts and the advanced configurations:

It shows VSAN.DomMaxLeafAssocsPerHost and VSAN.DomOwnerInflightOps as being different between a few of my hosts. Looking at the image above, you’ll see node 09 has values of 36000 and 1024, respectively, while the other nodes 10-12 show 12000 and 0.

I immediately went to the host configuration advanced settings in the web client, searched VSAN and don’t see either of those. I even checked through PowerCLI and can’t see those:

ThepHuck