Skip to content

Month: February 2012

Snapshot stuck “In Progress” Workaround

Came in to work today to find a VM stuck at the “In Progress” status while taking a snapshot. We use vDR as a complementary subset to our backup plans, and vDR had the unfortunate task of calling the snapshot which is now hung.

The official error read “An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.” One problem with that, I couldn’t log into the system. The snapshot was far enough along to freeze the IO, so I had to jump into CLI and kill the task.

To kill the task for a VM, jump into the CLI for the host (in this case it was through the iDRAC and local terminal… bad, I know) and run a: ps | grep vmx command to see all the processes while searching for vmx’s
Login & Command

Locate the Parent Process ID (the second column) for the hung VM, and run: kill *parent process ID* to end the process. In this case, it was: kill 465724
***DISCLAIMER: Be very careful doing this, if you don’t kill the proper process, it can do harm to your ESXi host***
Kill Parent Process

Instantly, the task remaining in progress should change to have a status of “The attempted operation cannot be performed in the current state (Powered off).”
New Status

Now, it’s time to check out that error. In this case I received a “Volume Shadow Copy Service error: Unexpected error DeviceIoControl” with the rest of the error seemingly pointing at the generic floppy drive. I know this is the error because it’s pointing dead at the VMware Snapshot Provider and in a state of “DoSnapshotSet”
Windows Error

That’s incredibly weird. So I uninstall both the floppy drive and it’s controller, also remove it from the VM’s settings while it’s powered off. I boot the system back up, the floppy drive has reinstalled itself. Very odd, so I just uninstall the drive and then disable the controller.
Device Manager

So it’s snapshot time again, right? Well, not really. I retried the snapshot, it freezes again. Time for some googling after I kill the parent process for the VM again.

What I came up with is that there is apparently something with Windows Server 2008 R2 systems having SQL 2008. This was a frequent topic over many VMware Communities posts, however no one really every had an answer on what was going on internal to the VM which would cause this problem. I know we have 4 or 5 SQL servers running and this is the first system we’ve run into this problem on.

Anyways, the best workaround I found was to disable the disk.EnableUUID parameter on the VM. Please note that by disabling this, you effectively disable VSS for the snapshot (ie. no quiescing). So I maintain that this is only a workaround and not yet a true solution

To do this, shut down the VM. Right click, Edit Settings, hit the Options tab, and click on “Configuration Parameters”
Config Parameters

In the Configuration Parameter pop-up screen, look for the “disk.EnableUUID” setting and change the value to read “false”
Enable UUID False

Click OK a couple times and boot the system up. Once it’s booted up, try giving it a snapshot while checking the option for “Quiesce guest file system”. This time, everything was successful. I ran the test and I also had the vDR appliance run another snapshot to get that one up to date

Hopefully I can do some more research and turn up some better answers, and at worst I’ll create a support ticket and see if VMware Support can point me in a better direction

vMA – Change the password…

Tried getting into my vMA box today to get some CLI action going and couldn’t log in. Either I couldn’t remember the password, or I managed to fat-finger it twice. So I got to find out the hard way how to get that reset…

Start by opening up the console on the vMA system and send it for a reboot. Once it hits the CentOS screen, hit any key to gain access to the menu
vMA Boot Screen

With the CentOS option highlighted, hit “e” to edit the boot command
vMA Boot Screen

Now, highlight the “kernel” entry and hit “e” again to edit the command
vMA Boot Screen

Enter the word “single” and hit the Enter key
vMA Boot Screen

You’ll be returned to the boot command screen, hit “b” to boot the system up. Once the system finishes booting, you should be at a prompt. At this prompt enter in “passwd vi-admin” to change the password on the vi-admin account. The system will now ask for the new password twice and say that the authentication tokens have been updated.
passwd Reset

At this point, reboot the system via “shutdown -r now”. Once the system reboots, you should return to the normal vMA screen. Hit F2 to reach the terminal and make sure you can log in, it should be successful.
Login Success

vExpert 2012 applications have begun!

The vExpert award is an achievement given by VMware to those people that stand out in the virtualization community and go that extra distance to share their knowledge and their specialties.

The criteria VMware uses are as follows (also listed on their site via: http://communities.vmware.com/docs/DOC-18314)

  • Value to the community. Did you create something valuable for the greater community of VMware users, such as a blog, book, tool, event, or presentation? Both technical and non-technical contributions are considered. This is the key criterion for the vExpert award.
  • Technical merit. If your contribution was technical, was it correct and useful?
  • Effectiveness. Were your contributions effective in making VMware users more successful?
  • Professionalism. Did you communicate in a professional manner?
  • Reach. Was your contribution useful to a large number of people? This award weighs public contributions that the entire community can benefit from.
  • Effort. vExperts typically put forth a great deal of time and effort over the course of the year, often beyond the boundaries of their normal day job. Did you demonstrate sustained effort in your virtualization projects in the past calendar year?

Couple new things for this years vExpert talent pool. First off there is the new additions of the new Paths. Those are the Evangelist Path, the Customer Path, and the VMware Partner Network Path.

  • Evangelists are classified as “book authors, bloggers, tool builders, public speakers, and other IT professionals who share their knowledge and passion with others”.
  • The Customer Path is classified for “internal evangelists and community leaders from VMware customer organizations”.
  • The VMware Partner Network (VPN) Path is classified as being for “employees of our partner companies who lead with passion and by example, who are committed to continuous learning and to making their technical knowledge and expertise available to many”.

More information about these different paths are available: http://communities.vmware.com/docs/DOC-18313

Another new item for this year is that everyone will have to apply. Nominations are now no more than invitations to apply.

So if you happen to know of someone that stands out, whether that’s a blogger, a company’s representative, a VMUG leader, or any IT professional that goes above and beyond to help out the virtualization community, please invite them via: http://www.jotform.com/form/20442343290

Or if you think you are a good candidate, apply here: https://docs.google.com/spreadsheet/viewform?formkey=dDdCbGNtanVuVW5iblM1dVNIMkpDVXc6MQ

Best of luck to all the applicants!

Windows Firewall just keeps fighting me…

I’m just trying to get some stuff to pass traffic, that’s all. However Windows Firewall on Server 2008R2 has decided that it’s too much to ask and is going out of it’s way to block the traffic.

Case in Point #1: I create a firewall rule to allow 443 in from a couple other subnets, everything works just fine. I turn on another system which resides in one of those subnets and it fails. I get a “Windows Filtering Platform has blocked a packet”.

Very weird, but maybe there’s something I’m missing. I turn on logging to see where we’re getting the failure. Here’s the funny part, the pfirewall.log file shows that it is allowing the traffic!
*NOTE: I highlighted the wrong one, so the arrow is pointing out the correct one.*
Firewall Allow, Filtering block

Probably a problem with the firewall policy right? Nope.
Firewall Policy
Firewall Policy

Some of the other things I’ve done to attempt to resolve the issue:

  • “netsh advfirewall reset” followed by recreating the rule via admin templates
  • “netsh advfirewall reset” followed by recreating the rule via Security Settings
  • “netsh advfirewall reset” followed by recreating the rule locally
  • Block Policy Inheritance, and retry all the above
  • Set the scope to any and the port to 443
  • Set the scope to any and the port to any

Everything else on the subnet is allowed in, so why would it be isolated to just this one system? I’m petitioning for a Microsoft Support Case to be made, hopefully they can get to the bottom of it.

Case in Point #2: Windows Firewall sees outgoing traffic and allows it. However, I get a “Windows Filtering Platform has blocked a packet” error again. The real head scratcher is that WFP sees the direction as being “Inbound”.
Send but Inbound is blocked

I’ve gone through and created the proper firewall rules to allow a source port exception of 5989, and even a rule to allow all traffic from the offending system. Still no luck. So I’m hoping to add that to the case and finally get to the bottom of these problems.

Any thoughts or insight is much appreciated…