Bug in VSAN 6.2: De-dupe scanning running on hybrid datastores

UPDATE
VMware has posted a KB about this, which I did not realize at the time of writing the blog. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146267

We’ve been testing out VSAN here at work and noticed that one of the clusters we rolled out had serious latency issues. We initially blamed the application running on the hosted VMs, but when it continued to get worse we finally opened a case with VMware. Here’s a chart of the kind of stats we were seeing (courtesy of SexiGraf):

Read latency in particular was very high on the datastore level, IOPS weren’t great, and Read Cache Hit Rate was low. We also saw that read and write latency was high on the VM level. After we opened a ticket with VMware, they discovered an undocumented bug in VSAN 6.2 where deduplication scanning is running even though deduplication is turned off (and actually unsupported in hybrid mode VSAN altogether). They provided the following solution:

For each host in the VSAN cluster: 1. Enter maintenance mode 2. SSH to the host and run: "esxcfg-advcfg -s 0 /LSOM/lsomComponentDedupScanType" 3. Reboot the host

After we applied the fix, the cluster rebalanced for a little while and came back looking much, much better. In the below graph, you can see right when the fix was applied and see read latency drop, IOPS increase, and read cache hit rate jump to the high 90-percents:

And for good measure, this is how it’s looked since:

So to summarize, if you are running hybrid VSAN 6.2, you should definitely check your latency and read cache hit rate. If you’re experiencing high latency and poor read cache hit rate, go through and change /LSOM/lsomComponentDedupScanType on all your hosts to 0. I can’t take credit for actually discovering this, so thank you to my coworker @per_thorn for tracking it down. And thank you @thephuck for letting me write it up on this blog!

–@vDudeJon

The world v. CrossFit – Can we stop the hate?

I’ve been a fan of fitness for many many years. Am I an “athlete”? I’m sure that depends on the interpretation, but the word athlete is defined as a person who is proficient in sports and other forms of physical exercise. I played football in middle school, by high school, everyone was bigger than me, so I played golf my entire high school career.

After high school, I began going to the gym, mostly light strength training, I would run a little here in there. And by a little, I mean 1-2 miles. I ran a 5k in high school when I was maybe 16, and think I finished with a time of roughly 26 minutes & change, by no means fast.

Fast forward to today. I still hit the gym, although more focused on strength & high intensity. Every once in a while, Google Now will throw something into my feed that’s interesting. A while back, I saw an article from Runners World: The CrossFit Workout Runners Should Actually Try. It outlined some workouts regularly found in CrossFit to help runners.

Where am I going with this?

TL-DR: See below for details on these commands

Create a local user in the NSX Manager’s CLI, then use the API to grant CLI privileges to that user.

Here’s how using a linux machine:
ssh admin@[nsxmanagerIP] enable config t user vrops-readonly password plaintext notrealpassword user vrops-readonly privilege web-interface
Log out of the NSX Manager (type exit) and stay logged into the linux machine.
Create cli-auditor.xml that contains this (replace brackets with greater/less than):
[?xml version="1.0" encoding="ISO-8859-1" ?] [accessControlEntry] [role]auditor[/role] [resource] [resourceId]globalroot-0[/resourceId] [/resource] [/accessControlEntry]
Add the user as an auditor in the NSX Manager as a CLI user:
curl -i -k -u 'admin:password' -H "Content-Type: application/xml" -X POST --data "@cli-auditor.xml" https://nsxmanagerip/api/2.0/services/usermgmt/role/vrops-readonly?isCli=true
Add your domain/vCenter user as an auditor in the NSX Manager (NOT as a CLI user):
curl -i -k -u 'admin:password' -H "Content-Type: application/xml" -X POST --data "@cli-auditor.xml" https://nsxmanagerip/api/2.0/services/usermgmt/role/ReadOnly@THEPHUCK.COM?isCli=false

Details for creating the NSX CLI user for vROps

Automatic Plex Media Server update script for Linux/Ubuntu

I’m sure many of you know of Plex Media Server (PMS) and how awesome it can be for letting your kids watch your movies on the go. It likely needs no introduction, but if you’d like to learn more, please click Plex Media Server to be taken to their site.

In my home lab running ESXi through my VMUG Advantage EVALexperience (shameless plug, I know), I have an ubuntu VM I built specifically for PMS with 2 vCPUs & 4GB of RAM.

I then set up the mounts for my Synology NAS where my movie folders are so they’re mounted at boot, installed PMS, configured libraries, did some customizations, and BOOM! Kids’ movies on my phone on the go!

Sounds awesome! Why are you writing a post?

My phone is spying on me!

Today at lunch we were talking about the primaries and how everyone was doing in the races. I was eating my lunch with my phone sitting on the table off to the side.

As our conversations continued, I picked up my phone and decided to check Google Now. There was nothing really out of the ordinary. As I scrolled, I saw this:

I thought how that’s a strange coincidence, then brought it up to the lunch crowd and joked how my phone was eavesdropping on our conversation. We all laughed and continued the joke as I put my phone down.

A few minutes later, I went back in to Google Now and it had a tile titled “Is your smartphone listening to you?”

Wow!! Really?? Yes, it was there, but too bad I didn’t screenshot it. After I clicked the link, then came back to Google Now and it refreshed, tile gone 😢

Coincidence? I think not!

vSphere Fault Tolerance Role Privilege names have changed from vSphere 5.5 to 6.0

I was playing in my lab today and ran across something I thought was strange. I exported the privileges from a test role in one lab, which happened to be vSphere 5.5, then tried to create a new role in vCenter 6.0 with the privileges I just pulled. It worked fine for almost everything, except these two:

Could not find Privilege with name 'Enable Fault Tolerance'. Could not find Privilege with name 'Disable Fault Tolerance'.

I thought that was kind of strange, so I ran a quick

Get-VIPrivilege | ? {$_.name -like "*fault*"} | select Name,Id

1	Get-VIPrivilege \| ? {$_.name -like "fault"} \| select Name,Id

and looked for something similar. Below is the comparison of 5.5 & 6.0:

vSphere 5.5
Name - Id ------ Turn On Fault Tolerance - VirtualMachine.Interact.CreateSecondary Turn Off Fault Tolerance - VirtualMachine.Interact.TurnOffFaultTolerance Disable Fault Tolerance - VirtualMachine.Interact.DisableSecondary Enable Fault Tolerance - VirtualMachine.Interact.EnableSecondary Query Fault Tolerance compatibility - VirtualMachine.Config.QueryFTCompatibility

vSphere 6.0
Name - Id ------ Turn On Fault Tolerance - VirtualMachine.Interact.CreateSecondary Turn Off Fault Tolerance - VirtualMachine.Interact.TurnOffFaultTolerance Suspend Fault Tolerance - VirtualMachine.Interact.DisableSecondary Resume Fault Tolerance - VirtualMachine.Interact.EnableSecondary Query Fault Tolerance compatibility - VirtualMachine.Config.QueryFTCompatibility

The difference is not drastic, but one simply word, or even one character, out of place will cause your script to fail. It’s easy to see that “Turn On” and “Enable” sound the same, so the need to rename “Enable” to “Resume” makes sense to me. Same with Disable & Suspend. These are just the two I know about, I really should write another article listing which ones have changed, but that’s for another day :)

Just something to watch out for I wanted to share.

Happy scripting!

VMware Virtual SAN Health failed Cluster health test

Here’s the error

While building a new environment for my lab, I ran across an interesting thing yesterday.

I looked at my cluster’s VSAN health and saw this error:

It’s complaining that my hosts don’t have matching Virtual SAN advanced configuration items.

If you click on that error, you’ll see at the bottom where it shows comparisons of hosts and the advanced configurations:

It shows VSAN.DomMaxLeafAssocsPerHost and VSAN.DomOwnerInflightOps as being different between a few of my hosts. Looking at the image above, you’ll see node 09 has values of 36000 and 1024, respectively, while the other nodes 10-12 show 12000 and 0.

I immediately went to the host configuration advanced settings in the web client, searched VSAN and don’t see either of those. I even checked through PowerCLI and can’t see those:

VMware vSphere 5.5 Web Client authentication fails with ‘cannot connect to the vCenter Single Sign On server.’

Earlier this week we were greeted with this awesome message:

It’s so descriptive we knew exactly where to start! Okay, yeah, not really. Sarcasm aside, you’d think the culprit would be SSO. I began checking the two SSO servers we have in an HA configuration and they appeared fine. What’s even more strange is the fat clients were all authenticating fine. I started checking logs on the SSO servers and saw several things similar to this:

2015-08-25 23:20:49,538 INFO [ActiveDirectoryProvider] Failed to find user snip@snipPrincipal id not found: {Name: snip, Domain: snip} via ldap search
and
2015-08-26 00:29:37.709:t@21945040:ERROR: ldap simple bind failed. Error(4294967295)

So I assumed it was SSO again, maybe related to the domain we auth against.

Great! So now what?

vSphere Replication 5.8 lets you violate VSAN Storage Policies

I’m sure many of you know VSAN’s Failures To Tolerate, or FTT, is something that adds overhead to both your cluster & your data. It’s no secret FTT of 1 doubles your data, think of it as N+1 copies of your data. You could essentially have two, three, or four copies of your data, redundancy is a good thing!

When you look at the cluster side of it, there is another ‘gotcha’. The host needs becomes 2N+1. Let’s look at FTT of one, that’s saying you need 2(1)+1 hosts, so 2+1 = 3. And of course, FTT2 requires 5 hosts, and FTT3 requires 7.

What’s the problem?

VMworld 2015 Public Voting is OPEN!

It’s that time of year again, and I know you’re as excited as I am, VMworld has opened up the sessions to public voting!

There are TONS of sessions to choose from, and I submitted a handful listed here:

The last one is something special with GS Khalsa, so please vote for us!

If you like any of these and would like to see these presentations on the big stage, please vote for them!

Go to http://vmw.re/1Pj6Lcc and you can either search for my last name or session ID.

ThepHuck