Skip to content

Tag: ESXi

PowerCLI – Quick Stats Not Up To Date Error

Recently rebooted one of my vCenters and came across an error on my ESXi hosts stating “Quick stats on *vmhost* is not up-to-date”. A couple seconds worth of googling brought me to VMware KB2061008 which helped to resolve the issue.
Quick Stats Error Message

However, the KB only went through the GUI process of adding in the requisite parameters but there’s no fun in clicking through the GUI so I came up with a short script that’s applicable to vCenter 6.0 which can also accomplish performing the parameter creation/updates for the KB’s workaround.

Once you run the script, you’ll still need to restart the vCenter service on your VCSA or Windows server. If you happen to be in the web client:
Go to “Administration” then to the “Deployment” area to select “System Configuration”
Select “Services”, then select “VMware vCenter Server”
Select “Actions” followed by “Restart”

Once the service is back up and running the error will no longer be present.
vCenter Service Restart

Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.

PowerCLI – Removing and Re-creating Unknown and/or Orphaned VMs

Ran into an issue lately where I found a host had lost its storage. For whatever reason HA didn’t kick in to bring all the VMs back to life and there was a need to recreate ~75 VMs whom were in an unknown and/or orphaned state. It looked a bit like this:
Unknown VMs

PowerCLI to the rescue!

Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.

PowerCLI – Reset NTP settings on VMHosts per Cluster

As daylight savings time is almost upon us, there are always questions revolving around what hosts use what NTP sources for time synchronization. To address this issue, I created a script to go through and remove the current NTP sources, add the desired NTP sources and then restart the NTP service on all of the hosts in the environment. However, to be safe, the preference was that it wasn’t done all at the same time. To achieve this, I broke it down and applied it to hosts one by one in a specified cluster.

When running the script, it will ask for the cluster name and four NTP servers so you don’t have to modify the code unless you want to add more or less NTP servers.


#Reset NTP settings on VM Hosts per Cluster

$InputCluster = Read-Host "Cluster Name:"
$InputNTP1 = Read-Host "First NTP Server:"
$InputNTP2 = Read-Host "Second NTP Server:"
$InputNTP3 = Read-Host "Third NTP Server:"
$InputNTP4 = Read-Host "Fourth NTP Server:"

#Select Cluster to change NTP Settings
$Cluster = Get-Cluster $InputCluster

#NTP servers to be changed to
$ntp1 = $InputNTP1
$ntp2 = $InputNTP2
$ntp3 = $InputNTP3
$ntp4 = $InputNTP4

#Grabbing VMHosts for desired Cluster
$allVMhost = $Cluster | Get-VMHost | sort Name

#Reseting NTP servers one by one
foreach ($vmhost in $allVMhost){

#Remove existing NTP servers
Write-Host "Removing all NTP Servers from $vmhost"
$allNTPList = Get-VMHostNtpServer -VMHost $vmhost
Remove-VMHostNtpServer -VMHost $vmhost -NtpServer $allNTPList -Confirm:$false | out-null
Write-Host "All NTP Servers from $vmhost have been removed"
Write-Host ""

#Setting NTP servers
Write-Host "Adding NTP servers to $vmhost"
Add-VmHostNtpServer -NtpServer $ntp1,$ntp2,$ntp3,$ntp4 -VMHost $vmhost -Confirm:$false | out-null
Write-Host "The following NTP servers have been added to $vmhost : $ntp1, $ntp2, $ntp3, $ntp4"
Write-Host ""

#Checking NTP Service on the ESXi host
$ntp = Get-VMHostService -vmhost $vmhost| ? {$_.Key -eq 'ntpd'}
Set-VMHostService $ntp -Policy on | out-null

if ($ntp.Running ){
Restart-VMHostService $ntp -confirm:$false
Write-Host "$ntp Service on $vmhost was On and was restarted"
}
Else{
Start-VMHostService $ntp -confirm:$false
Write-Host "$ntp Service on $vmhost was Off and has been started"
}

Write-Host ""

}

Click here to download a text file containing the script.

Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.

PowerCLI – Rescan HBAs for a Cluster on a Host by Host Basis

After having some bad luck with rescanning HBAs for entire clusters or datacenters all at the same time (Cliff’s Notes: the LUNs ended up looking like array based snapshots and therefore unusable), it was decided that any rescans should be done on an individual host basis. Below is the script I created to achieve this goal.

When you run the script, it will ask for the Cluster name so you don’t have to modify the code.


#Rescan HBAs for a cluster on a host by host basis

$InputCluster = Read-Host "Cluster Name"

$vmhosts = get-cluster $InputCluster | get-vmhost

foreach ($i in $vmhosts) {

Write-Host "Starting rescan of all HBAs on $i"
$i | Get-VMHostStorage -RescanAllHba | Out-Null
Write-Host "Rescan of all HBAs on $i is complete"
Write-Host ""

}

Click here to download a text file containing the script.

Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.

PowerCLI – Analyze a Cluster to Move VMs to a Resource Pool

After someone deployed a bunch of VMs, we let them know about the Resource Pool they were supposed to be deployed to. Oops. To correct this, and to avoid a couple hours of dragging and dropping VMs into a resource pool, I was able to create a script that detects if a VM is outside of a Resource Pool and then move it to the specified Resource Pool.

When you run the script, it will ask for the Cluster name and the Resource Pool name so you don’t have to modify the code.


#Analyze the Clusters and check for systems outside of Resource Pools

$InputCluster = Read-Host "Cluster Name"
$InputRP = Read-Host "Resource Pool Name"

#Cluster which will be analyzed
$cluster = Get-Cluster $InputCluster

#Resource Pool where VMs should be moved
$rp = Get-ResourcePool $InputRP

#Detection of VMs not in Resource Pools
$norp = Get-Cluster $cluster | Get-VM | where {$_.ResourcePool.Name -eq "Resources"}

foreach ($i in $norp) {

Write-Host "Moving $i to Resource Pool $rp"
Get-VM $i -Location $cluster | Move-VM -Destination $rp | Out-Null
Write-Host "Move complete"
Write-Host ""

}

Click here to download a text file containing the script.

Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.

Fusion-IO Caching Tests…

After some issues with the setup and configuration of the Fusion-IO ioCache cards we picked up, I finally got to dig in and do some basic testing with IOMeter.

To setup the test, I configured a new 20GB VMDK on it’s own paravirtualized SCSI Controller. The drive was formatted NTFS in a full/non-quick method as the F: drive. The IOMeter test was run on a single worker against the entire F: drive. The All-in-one test was selected and run for 20 minutes.

First up, I tested the drive all by itself with no caching enabled:

IOps: 1348.69 Read IOps: 670.1738 Write IOps: 678.5159

Next, I tested Volume based caching. I started off by making the following modifications to the Fusion-IO tab within vCenter as follows to add only the F: drive to the Volume Caching Filter:
Volume Based Caching

Then I reset the F: drive by formatting it again as NTFS in a full/non-quick method. Once the format was complete, I reran the IOMeter test and received these results:

IOps: 1486.163 Read IOps: 737.5608 Write IOps: 748.6018

Lastly, I tested the Drive based caching. I went back to the Fusion-IO tab within vCenter and removed the Volume Caching Filter on the F: drive and then set the Drive Caching Filter to Drive1 (Drive0 was the drive the OS was installed on, Drive2 was the drive which is attached by FusionIO automatically):
Drive Based Caching

Then I reset the F: drive by formatting it again as NTFS in a full/non-quick method. Once the format was complete, I reran the IOMeter test and received these results:

IOps: 1509.644 Read IOps: 748.7889 Write IOps: 760.8555

I also managed to grab a shot of the Performance graphs for the disk during the tests via vSphere client: (pardon the lapse between 2PM and 3PM on the graph)
Performance Graph on Disk

So to review and put the results all on the same table:

No Caching Enabled IOps: 1348.69 Read IOps: 670.1738 Write IOps: 678.5159
Volume Caching Enabled IOps: 1486.163 Read IOps: 737.5608 Write IOps: 748.6018
Drive Caching Enabled IOps: 1509.644 Read IOps: 748.7889 Write IOps: 760.8555

Remember, these are just initial results with nothing but having the card installed, drivers installed, firmware upgraded, ioTurbine installed, and the guest package installed. While some of the results weren’t exactly what I was expecting, I’m pretty excited to dig in deeper to see what kind of performance we can gain out of these cards.

Small update…

While this particular blog post is about caching, since that’s how these cards will be used in this environment, I couldn’t help but go back, mount the Fusion-IO card as VMFS storage, SvMotion the F: drive over to the Fusion-IO VMFS datastore and run the test again. So once again, the F: drive was formatted as NTFS in a full/non-quick method. Once the format was complete, I reran the IOMeter test and received these results:

IOps: 5443.904 Read IOps: 2700.201 Write IOps: 2743.703

Updating the Fusion-IO firmware…

We were fortunate enough to pick up some Fusion-IO IOcache cards to be implemented in our new infrastructure.

Installed the Fusion-IO drivers on the ESXi box and went to check the cards out. We do a “fio-status” command vis SSH connection to pull the information on the cards in each system.
fio-status printout

Towards the bottom, you can see the Firmware listed as v5.0.7 and at the time the current release was v7.1.13. So head out to the Fusion-IO support site (http://support.fusionio.com/), log in with your support account credentials, click on the “Downloads” button, select the card installed, select the OS being used, and look for the firmware menu. Grab the firmware file, should be a file ending with an FFF extension.
Downloading Firmware

Once it’s downloaded, upload the firmware to a place the ESXi host can read from. In this case, I uploaded the firmware to a VMFS datastore so I can use the same file for every server.
Firmware in a VMFS datastore

At this point, we can get the update installing. To do that, SSH into the desired host (remember to have the SSH service running), and issue a “fio-update-iodrive *firmware location*” command.
Update Firmware

After about 12 minutes, the install was complete and a reboot was required.
Firmware Update Complete

Once the ESXi host was back up, SSH in to the ESXi host and run another “fio-status” command and show that the Firmware is now indeed v7.1.13
New Status Check

If you happen to run into the same issue where the “Active Warnings” indicate a message of: The ioMemory is currently running in a minimal state. The problem I found was that the driver version was incompatible. Check out my other posting on how to update the driver: http://www.thatcouldbeaproblem.com/?p=602

Installing and/or Updating the Fusion-IO Drivers on an ESXi Host…

We were fortunate enough to pick up some Fusion-IO IOcache cards to be implemented in our new infrastructure.

In this case, Fusion-IO drivers on the ESXi box were slipstreamed into the install image. To check the driver version we do a “fio-status” command via SSH connection to pull the information on the cards in each system. If the drivers are not yet installed, move on the the next step.

Grab the drivers from the Fusion-IO support site (http://support.fusionio.com/), log in with your support account credentials, click on the “Downloads” button, select the card installed, select the OS being used, and look for the “Software Binaries” section and download the offline_bundle zip file. Once it’s downloaded, upload it to somewhere the ESXi host can read it and run the following command: esxcli software vib install -d *full path to offload_bundle zip*

Once the install process has completed, reboot the host. Once it comes back up, SSH back into the ESXi host and run the “fio-status” command. It should now be updated to the version you just installed. In this case, version 3.2.2.

Using the Cold Clone 3.0.3 ISO with vSphere 5

As many know, the Cold Clone ISOs have been discontinued and support has been removed from VMware. This is quite unfortunate especially when you get into the business of P2V’ing items like domain controllers, SQL servers, and other pesky boxes.

I understand the converters from VMware have improved by leaps and bounds over what it was in version 3, but there’s still a reasonable amount of security in P2V’ing a box while it’s completely down with no running services whatsoever.

First issue I ran into while attempting to P2V an old SQL server was finding the ISO. It has almost been removed from the internet as a whole. So I’ve uploaded it here: Cold Clone 3.0.3 Link

Second issue, boot time. It took me literally 15 minutes from the point of “hitting a button to boot to disc” to the point of accepting the EULA.

Third issue, drivers. I was lucky enough for the internal NICs to be found but the add-on NICs were not. To put this into perspective, the server I was working on was an HP G5. I tested with an HP G6 and it did not find any NICs at all.

Fourth issue, disappearing network settings. Every time I went to the networking settings, they would be cleared. The settings held as long as I clicked apply and then ok, but the second you went back to the network config menu I had to renter all the info.

Fifth issue, vCenter integration. It’s not exactly shocking that it didn’t work with vCenter, but I was hopeful. The converter would go through and recognize everything out of vCenter, but then ran into a bunch of issues as soon as the clone actually starts. Such as:

Couldn’t find the Distributed vSwitch vNIC:
[managedVMCreator,2657] No network named “SYSMGMT” was found
[imageProcessingTaskStep,194] VmiImportTask::task{9} step “create VM” destroyed
[vmiImportTask,439] Error during creation of target VM
[imageProcessingTaskStep,194] VmiImportTask::task{9} step “create and clone to VM” destroyed
[imageProcessingTaskStep,194] VmiImportTask::task{9} step “Clone VM” destroyed
[imageProcessingTaskImpl,552] VmiImportTask::task{9}: Image processing task has failed with MethodFault::Exception: vim.fault.NotFound
[imageProcessingTaskImpl,154] VmiImportTask::task{9}: SetState to error

Couldn’t find the FC attached storage on a host:
[NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Host address lookup for server esx01 failed: The requested name is valid, but no data of the requested type was found
NBD_ClientOpen: Couldn’t connect to esx01:902 Host address lookup for server esx01 failed: The requested name is valid, but no data of the requested type was found
DISKLIB-DSCPTR: : “vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6” : Failed to open NBD extent.
DISKLIB-LINK : “vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6” : failed to open (NBD_ERR_NETWORK_CONNECT).
DISKLIB-CHAIN : “vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6” : failed to open (NBD_ERR_NETWORK_CONNECT).
DISKLIB-LIB : Failed to open ‘vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6’ with flags 0x2 (NBD_ERR_NETWORK_CONNECT).
[diskHandleWrapper,87] DiskLib_Open failed on vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6 with error NBD_ERR_NETWORK_CONNECT.
[imageProcessingTaskImpl,552] BlockLevelCloning::task{21}: Image processing task has failed with MethodFault::Exception: sysimage.fault.DiskLibConnectionFault
[imageProcessingTaskImpl,154] BlockLevelCloning::task{21}: SetState to error

How I actually got it to work was by sending the P2V directly to a specific ESXi 5 host. Once I did that, I could P2V the system to the locally attached storage or the FC attached storage. I attached the vNIC to a standard vSwitch.

Playing with ESXi and Synology in the home lab…

So I finally bit the bullet and bought a NAS for my home lab. Ended up with a Synology DS411 due to some good timing from a NewEgg sale.

I went with Synology for two reasons: 1. I could supply my own disks (I don’t want the “Green” drives that came with most other NAS units) 2. I always hear REALLY good things about them, they’re always highly recommended.

I happened to already have a set of 4 Hitachi Deskstars sitting around so I tossed them in and started running some tests with IOMeter (Run all tests). I did a test on a RAID5 grouping of all 4 disks which were presented to the ESXi host as an iSCSI target and then I did the same test with a RAID 5 grouping of all 4 disks which were then presented to the ESXi host via NFS. I was amazed at the results.

From a personal standpoint, I fully expected iSCSI to blow away NFS. Wow, was I wrong. iSCSI pulled a whopping 270 IOPS from the IOMeter test. Then I ran the exact same test only via an NFS share… NFS blew iSCSI out of the water with a 1,070 IOPS. My jaw is still on the floor from that result. Here’s some of the rest of the results:
NFS vs iSCSI

I’m also seeing some excellent regular performance as well. 45MBps worth of write during an SvMotion.
Cooler

So to say the least, I’m quite impressed at what this little guy can do with 4 SATA 7200RPM drives and a single NIC.

One other tidbit to add in an unrelated fashion is how much cooler the new Hitachi Deskstar runs compared to the older ones. A full 12F cooler. That was another surprise…
Cooler