vCenter Server Appliance fails with EXT4-fs journal errors

vCenter’s not responding properly

Scroll down to bottom for TL;DR version

I got a text message this evening from a colleague of mine (@FrankRax) stating our lab was down. I tried to hit the vCenter and the hosts & clusters view wouldn’t load in the web client, just left me with the spinning wheel:

Okay, that’s fine, so I’ll check the VAMI, or Management UI of the VCSA, but then I got really scared when I saw this:

This isn’t a fresh install, it’s been a lab for a long time, actually even upgraded to 6.5u1 not that long ago. Now I know for a fact something’s gone wrong, so I launched the host client on each node in the cluster until I found the vCenter Server Appliance VM and launched the console, and was pretty much horrified at what I saw
the following content may be disturbing to some audiences, viewer discretion is advised

Nutanix AHV VM Reporting

I was reading VCDX56’s post Nutanix AHV VM Reporting Via REST API authored by Magnus Andersson @magander3 where, as the title suggests, he discusses a script he wrote to gather information about VMs running on a Nutanix AHV cluster using the REST APIs. At the end of the post, he mentioned that he would like to change the script from Bash to Python.

I have recently been doing quite a bit REST API scripting with Python, so I took a crack at it last night.

Error when Deploying a New VMware VM from Template: “you do not have permissions to assign this network”

I was trying to deploy a new VM from a vSphere template, but I kept getting the error: “you do not have permissions to assign this network”

Backup and Restore a VM in AHV

I had a need that required me to completely wipe my Intel NUC running Nutanix Community Edition (CE), including all VMs. Unfortunately, a couple of those VMs have become pets, not cattle so I had to find a way to backup and restore those VMs. Poking around online I found various parts of what I wanted to do most of which centered around backing up the VMs or exporting them to another hypervisor. I couldn’t find anything on completing the process of actually restoring the VMs that were backed up. So I set out to finish the loop.

Install Nutanix Community Edition Nested in KVM

This guide will show you how to install Nutanix Community Edition (CE) nested in a KVM environment. While running Nutanix CE on physical hardware is preferred, being able to run it as a VM or four could prove to be invaluable to you for certain lab testing, learning, or training.

All in a completely license fee free environment!

Getting Started with VMware vSAN: Hybrid or All Flash?

Everyone hears about VMware’s Virtual SAN and how awesome it is. It’s a very compelling offering and is only overshadowed by their software defined networking solution NSX.

The biggest hurdle: how to get started.

The truth is it’s extremely simple to enable and start using, but that’s not the “getting started” I’m talking about. I wanted to cover off some things to think about when you’ve decided you’re going down the VSAN path.

How do you know how many IOPS to expect, or how much storage you will have or need, should you go hybrid or all flash, and what resiliency or protection options you have, and the impact of those.

First things first: Hybrid or All Flash?

Bug in VSAN 6.2: De-dupe scanning running on hybrid datastores

UPDATE
VMware has posted a KB about this, which I did not realize at the time of writing the blog. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146267

We’ve been testing out VSAN here at work and noticed that one of the clusters we rolled out had serious latency issues. We initially blamed the application running on the hosted VMs, but when it continued to get worse we finally opened a case with VMware. Here’s a chart of the kind of stats we were seeing (courtesy of SexiGraf):

Read latency in particular was very high on the datastore level, IOPS weren’t great, and Read Cache Hit Rate was low. We also saw that read and write latency was high on the VM level. After we opened a ticket with VMware, they discovered an undocumented bug in VSAN 6.2 where deduplication scanning is running even though deduplication is turned off (and actually unsupported in hybrid mode VSAN altogether). They provided the following solution:

For each host in the VSAN cluster: 1. Enter maintenance mode 2. SSH to the host and run: "esxcfg-advcfg -s 0 /LSOM/lsomComponentDedupScanType" 3. Reboot the host

After we applied the fix, the cluster rebalanced for a little while and came back looking much, much better. In the below graph, you can see right when the fix was applied and see read latency drop, IOPS increase, and read cache hit rate jump to the high 90-percents:

And for good measure, this is how it’s looked since:

So to summarize, if you are running hybrid VSAN 6.2, you should definitely check your latency and read cache hit rate. If you’re experiencing high latency and poor read cache hit rate, go through and change /LSOM/lsomComponentDedupScanType on all your hosts to 0. I can’t take credit for actually discovering this, so thank you to my coworker @per_thorn for tracking it down. And thank you @thephuck for letting me write it up on this blog!

–@vDudeJon

The world v. CrossFit – Can we stop the hate?

I’ve been a fan of fitness for many many years. Am I an “athlete”? I’m sure that depends on the interpretation, but the word athlete is defined as a person who is proficient in sports and other forms of physical exercise. I played football in middle school, by high school, everyone was bigger than me, so I played golf my entire high school career.

After high school, I began going to the gym, mostly light strength training, I would run a little here in there. And by a little, I mean 1-2 miles. I ran a 5k in high school when I was maybe 16, and think I finished with a time of roughly 26 minutes & change, by no means fast.

Fast forward to today. I still hit the gym, although more focused on strength & high intensity. Every once in a while, Google Now will throw something into my feed that’s interesting. A while back, I saw an article from Runners World: The CrossFit Workout Runners Should Actually Try. It outlined some workouts regularly found in CrossFit to help runners.

Where am I going with this?

TL-DR: See below for details on these commands

Create a local user in the NSX Manager’s CLI, then use the API to grant CLI privileges to that user.

Here’s how using a linux machine:
ssh [email protected][nsxmanagerIP] enable config t user vrops-readonly password plaintext notrealpassword user vrops-readonly privilege web-interface
Log out of the NSX Manager (type exit) and stay logged into the linux machine.
Create cli-auditor.xml that contains this (replace brackets with greater/less than):
[?xml version="1.0" encoding="ISO-8859-1" ?] [accessControlEntry] [role]auditor[/role] [resource] [resourceId]globalroot-0[/resourceId] [/resource] [/accessControlEntry]
Add the user as an auditor in the NSX Manager as a CLI user:
curl -i -k -u 'admin:password' -H "Content-Type: application/xml" -X POST --data "@cli-auditor.xml" https://nsxmanagerip/api/2.0/services/usermgmt/role/vrops-readonly?isCli=true
Add your domain/vCenter user as an auditor in the NSX Manager (NOT as a CLI user):
curl -i -k -u 'admin:password' -H "Content-Type: application/xml" -X POST --data "@cli-auditor.xml" https://nsxmanagerip/api/2.0/services/usermgmt/role/[email protected]?isCli=false

Details for creating the NSX CLI user for vROps

Automatic Plex Media Server update script for Linux/Ubuntu

I’m sure many of you know of Plex Media Server (PMS) and how awesome it can be for letting your kids watch your movies on the go. It likely needs no introduction, but if you’d like to learn more, please click Plex Media Server to be taken to their site.

In my home lab running ESXi through my VMUG Advantage EVALexperience (shameless plug, I know), I have an ubuntu VM I built specifically for PMS with 2 vCPUs & 4GB of RAM.

I then set up the mounts for my Synology NAS where my movie folders are so they’re mounted at boot, installed PMS, configured libraries, did some customizations, and BOOM! Kids’ movies on my phone on the go!

Sounds awesome! Why are you writing a post?

ThepHuck