Tag: VMware vSphere CLI

[Solved] ESX VM shutdown stopped at 95% – VM fails to restart

We have several customers on sketchy hardware and on occasion the VM crash due to a SCSI card issue with the mother board used, that aside we have from time to time a need to force a hard reboot of the server running in a VM. Some times it works great and sometimes we have a lockup at 95% and have to force a kill of process that runs the VM to get it to restart.

So here is the process we take to get this to free up and reboot the VM on ESXi 5.0 and later VMware hosts.

 

  1.  Make sure if you do not already have it turned on, to turn on SSH on the ESXi Host. This can be done via the [Configuration -> [Security Profile] using the VMware client.
  2. Using your favorite SSH Client  (Putty), connect to your VMWare ESXi 5 Host.
  3. We now need to get and kill the process group for the VM that has failed. To do this we will look for the process group ID using this command.

    execute -> ps -g|grep “VMName”You should get a return that looks similar to this.

    3372 vmm0:MyVMSystem
    3374 3368 vmx-vthread-4:MyVMSystem 3368 3368 /bin/vmx
    3375 3368 vmx-mks:MyVMSystem 3368 3368 /bin/vmx
    3376 3368 vmx-vcpu-0:MyVMSystem  3368 3368 /bin/vmx

    We are looking for the common number across all processes and in this case that would be “3368” as seen near the end of each line.

  4. Now will need to kill the process. To do this we need to execute -> kill -9 3368   Replace “3368” with the ID number of your system.
  5. Now we need to do some clean up, We need to delete the swap file in the directory where the VM is stored. To get to where we store the swap file you will need to do the following.execute ->cd /vmfs/volumes/<YourDataStore>/<VMName>

    Next we need to make sure what our swap file name is so execute -> ls

    This will give you a directory listing find your swap file by looking for the file extension “.vswp”.  Now we will remove it with this command.

    execute -> rm –r <YourSwapFile.vswp>

     

  6. Now lets restart our VM services, This will not affect any running VM and is safe to run while VMs are active on host.execute -> /sbin/services.sh restart

     

  7. Reconnect your VMware client to the host and complete the process to power on the VM by first removing the VM from the inventory (Do not Delete from Disk) and re-adding it back in. This will reset the VM fully and allow you to restart it. After you remove your VM from inventory you can re-add it by browsing the datastore in your VMWare client finding your VM directory and right clicking on the “.vmx” file.  A menu will pop up and you can click “Add to Inventory” which will place VM back into the available VMs list. Now select the VM and click the Boot up arrow button to get started again.

 

Enjoy,

Cubert 😎

 

[Kaseya Agent Procedure] VMWare ESXi Hardware Health Monitor

VMWare ESXi Hardware Health Monitor Script

From the skunk works here at Squidworks comes another great monitoring script for Kaseya.

 

 

This script uses the SDK provided by VMWare to query the  ESX host and return a good or bad variable.  If the hardware test fails then the script grabs the log of the test and uploads it to Kaseya Server then places it under the “Get File” area for the host that ran the test. You can run this script on any windows box, I have also included the current vSphere SDK installer and a Kaseya script to install it if it is not found on the Windows Host.

Upload the SDK installer and Import the scripts to your public files area in Kaseya under the directory “VMWare”. If you place files anywhere else you will need to edit script for the new location of files.

The script then makes a unique event log entry into the Windows Application Event Log under the Application Events that can then be picked up by Kaseya’s Event Log Monitor. When Kaseya picks up this event you can instruct the monitor to create an alarm, create a ticket, run another agent procedure or email the alarm to an address(s).  Just schedule the agent procedure to run a couple of times a day to keep an eye on your customers VMware vSphere ESXi Hardware health.

This script links to the CIM information provided by the hardware to the ESX host. You will see CPU, Memory, Fans, RAID and Controller Health. The log file that is uploaded will only show failures and will tell you what failed and on what ESX host.

Download -> Kaseya VMWare ESXi Hardware Monitor

 

 

Enjoy

Cubert 😎

XYMON – ESX Hardware Health Monitor

Grab the Hardware Health of your VMware vSphere ESXi Host

Here is RC1 of my ESX Health Script for the XYmon monitoring server, not to be compared to the ESXi script going around for VM and Snapshot Health.  ESXHealth monitors the physical health of the hardware running the ESX hyper-visor  This script uses the xymoncgimsg.cgi to send status reports from a remote network to your XYmon Display Server thus allowing you to monitor ESX Host from anywhere really easily. Using this CGI allows you to run it from any windows box and send the notifications through port (http) friendly firewalls.

The only prereqs are, you must have installed the VMWare vSphere Perl SDK and have installed CURL or place the curl-nossl.exe provided in zip in the your PATH on Windows. Your XYmon server must have the xymoncgimsg.cgi moved from the xymon/server/bin folder to your xymon-cgi folder to allow web based status messages.

I have include the curl-nossl portable EXE with zip. Just drop it in the path for windows so you can call it by name. You will need to edit the script and update the URL to send the notifications to. I show an example in the script on how to use with a htpasswd protecting your web CGI for thoes who use that layer of security. If not then just place the standard URL for your xymoncgimsg.cgi and your good to go.

The script is very simple.

If all is good you get a “All’s OK” else if anything is bad it spits out any relevant info about that issue. The last error I got was for redundant Power lost to Supply 1 and was reported with plenty of detail to know what is wrong.

Just schedule windows to run every 15 minutes or so, see script for command line syntax for task scheduler

Download  -> ESXHealth-1.1  Monitor for XYmon

 

The monitor will look like the following examples: (Never mind my XYMon theme)

 

 

 

 

I hope this helps someone out there.

Cubert 😎

How-to get SNMP Data from VMWare ESXi 5 with Dell OpenManage, Cacti, Nagios, Munin or Ganglia

 

So you want to access your VMware ESX 5 host over SNMP? Here is the quick steps to get Dell OpenManage and SNMP on your ESX 5 host up and running fast.  You will need the VIB’s from Dell, also will need to download and install the VMware CLI client for accessing ESX hosts from remote. For the SNMP setups below we are using a generic password inside of quotes(“password”) you should remove the quotes and place the root password of the esx host where you see the quoted password text. We also are setting up SNMP to send traps to 192.168.24.120, you will need to change this to any host you want to receive traps. You will need to replace the server IP address with the address of your ESX5 host. If you use network monitoring applications like Cacti, Nagios, Munin or Ganglia then this is just what you need to start capturing network statistics and graphs.

 

Steps  to install OpenManage.

  1. Downloaded OpenManage VIB from Dell http://www.dell.com/support/drivers/us/en/g_5/DriverDetails/DriverFileFormats?DriverId=N3R0H&FileId=2773322235
  2. Uploaded VIB to ESX host and placed in /var/log/vmware/
  3. SSH to host as root
  4. ran command esxcli software vib install -d /var/log/vmware/OM-SrvAdmin-Dell-Web-6.5.0-542907.VIB-ESX50i_A02.zip
  5. After software install, I rebooted the host.

Steps to configure SNMP

  1. Started vmWare CLI
  2. vicfg-snmp.pl –server 192.168.24.10 –username root –password “password” -c public
  3. vicfg-snmp.pl –server 192.168.24.10 –username root –password “password” -t 192.168.24.120@161/public
  4. vicfg-snmp.pl –server 192.168.24.10 –username root –password “password” -E
    Current SNMP agent settings:
    Enabled  : 1
    UDP port : 161
    Communities :
    public
    Notification targets :
    192.168.24.120@161/public
    Options :
    EnvEventSource=indications

Openmanage Web Interface:

  1. Download OpenManage Server Administrator Managed Node http://www.dell.com/support/drivers/us/en/555/DriverDetails/DriverFileFormats?DriverId=R300521
  2. Navigate to https://localhost:1311, type in ESX host IP address, login and password.
  3. You should be able to view the all the good stuff like “Main System Chassis”, Software, Storage

 

I hope this helps someone out there in cyber space!

Cubert 😎

[How To] – Patching Standalone vSphere 5 ESXi Hosts

So you have a standalone ESXi host or 2 that needs to be patched with the latest updates coming out of VMWare and you do not have a VCenter or are otherwise using the VMWare Update Manager. Here is the basic steps to get you updated and all patched up.

 

 Here are the steps to getting your ESXi 5 Host patched.

  1. Download the patch file you want directly -> Get VMware Patches Here
  2. Enable SSH for remote support on the ESXi Host by accessing the (Security Profile) under Host Configuration.
  3. Place ESXi host is in Maintenance Mode.
  4. Using the vShpere Client browse your local datastore and make a directory called “Updates”, select this new folder and upload the patch zip file bundle to this folder.
  5. SSH into Host and execute the following command.  (Replace [DATASTORE] and [PATCH_FILE] with your actual file information.

 

esxcli software vib install -d /vmfs/volumes/[DATASTORE]/Updates/[PATCH_FILE].zip

You should get the following installation result.

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Afterwards you should reboot the ESX host and upon it’s return it should now be patched with the build you applied above.

NOTE: If you try and use the old esxupdate command, which  is still there, you will get this message:

~ # esxupdate Usage: This program is not meant to be used directly. Please use esxcli image/software instead.
esxupdate: error: Command must be specified Usage: This program is not meant to be used directly. Please use esxcli image/software instead.

 
 

[Solved]-VMware Install operation failed: The metadata or bundle option must be specified

Getting error message: Install operation failed: The metadata or bundle option must be specified ?

This error popup up after running the following cmd when trying to update our VMware vSphere server.

 C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -url https://192.168.xxx.xxx/sdk/webservice -i –bundle c:\temp\upgrade-from-esxi4.0-to-4.1-update01-348481.zip

I am using vSphere CLI 4.0 on a Windows 2003 server, logged in as a domain admin.

I tried several things I found off Google then it occured to me that it may be a issue with permissions. I restarted the CLI with Admin priviledges and Wamo!!

 Please wait patch installation is in progress …
The update completed successfully, but the system needs to be rebooted for the changes to be effective.

 So if your getting this error this is worth a try.

Cubert