Monitoring Your Clients VMWare Infrastructures.

Who manages your VMWare ESX Hardware Health?

If you are using ConnectWise Automate or LabTech RMM tool then this is a must have plugin for your environment. Plugins4Automate has a great plugin for managing ESXi Health monitoring in ConnectWise Automate and LabTech.

Visit them at www.plugins4automate.com

VMWare ESX Hardware Health monitor for Labtech

headerlogo

 

Plugins4LabTech has created a new plugin that will monitor the CIM data announced by the hardware running VMWare ESX software. We can report on any hardware that follows CIM standards and makes it’s hardware statuses known to VMWare. This simple to use plugin deploys in just minutes and can be setup by anyone with little or no  knowledge of LabTech functions and processes. If you want to see in action the easiest ESX Hardware Health Monitor available for LabTech then have a peek at our video on our YouTube Channel.

 

Never be in the dark again over hardware health!

 

Get alerts and tickets when hardware failures are detected. Using the monitor agent supplied with the plugin get quick responses to failures and warnings directly from VMWare. When alarms happen get emails, tickets and messaging alerting you to the failure that is taking place on the hardware.

Plugins4LabTech’s VMWare Health Monitor plugin for Labtech uses an agent to talk to VMware ESX hosts and retrieves the CIM data for the hardware the ESX host is running on. The plugin processes this data and stores it in LabTech’s database to be used in the views and alarms the plugin issues when failures are seen. You will see data on your RAID arrays and SCSI controllers, Power Supplies failures and system overheats. If the hardware is reporting it to VMWare we can see it.

 

Main ESX Health Monitor Console

The main view lets you see all the ESX hosts under management sorted by Client. In this view you are able to turn on and off the collection of data, set the interval of how often the data is collected and enable alerting for any seen failures.

ViewMenu

 

 

ESX Health Monitor Client Console Tab

The Client Console tab is where you would add and delete the ESX Hosts you want to monitor. You are able to force the update of ESX CIM data by selecting to rescan ESX hosts and you can view the full CIM data list of any ESX Host.

Clienttab

 

 

ESX Health Monitor CIM Data

The CIM Data view shows all the collected CIM data on a given ESX host. If any data is in a state not considered ok then those lines will be represented with warning and alarming icons. You will also get the hardware manufactures data back and in the case of Dell that is the Hardware type and Service Tag.

CIMData

 

 

ESX Health Monitor Internal Agent

Under the main view you can select to enable alerts. When you do this agent is created in the internal monitors of LabTech. You can use this monitor to email, ticket, alert or message anyone when failures are picked up.

monitor

A simple plugin all self contained and easy to deploy.  Just a few clicks to configure and you have data. So easy to use anyone can setup and have it working in just a few minutes.

 

The plugin is currently with the Squid Squad getting a final review before we release the new plugin. Follow our release notes at http://support.plugins4labtech.com

download

 

LabTech & ConnectWise ESXI Host Hardware Monitor

vmware-esx-monitor2

NEW ->LabTech ESXi Hardware Monitor v2.1

Squidwork’s ESX Hardware monitor is a set of scripts, a custom group and search for Labtech that will monitor the CIM data provided through the VMWare API for ESX 4 and 5. The probe will launch hourly and report back to Labtech any hardware failure or warning. The script will email an alarm to any email address you would like. The script can be modified to also set an alarm, create a ticket or anything else Labtech scripting will let you do.

New in version 2.1:

We added several checks for false alarms and socket errors to prevent alarms and emails on non failures.

We added alarm flood control, once a email goes out it will not send another until the system has reported a “all OK” then alerts are reset to go out on next fail.

Added extra EDFs to control processes.

Here is how it works:

Download the zip file, extract and import the  XML files into your Labtech system.

download

Addendum Update:

After you import “all” scripts in Version 2.1  Download this zip and import this script. This script should then be used in your group scheduled script  probe instead of the v2.1. This v.2.2 of that one script.  Download update here

Download extra files directly if import fails for any reason here.

After the import you should have a VMware script group that has 3 scripts in it.

Script #1 (The Installer) will install the monitor to a Windows system. You will need to provide the FQDN or IP of the ESX host you want to monitor when you execute the installer script. When the scheduler pops up make sure to add your ESX host. The only thing you should need to do is execute the installer on a Windows system. The installer will configure the system and add the system to the custom group and search. You do not need to configure anything else at this point. The ESX user and password will be fetched from the Locations password menu for VMWare.

ESX-installer-v2

The next script (The Monitor) is assigned to the custom group “Systems that monitor ESX hosts” to run every hour. You can modify this to run at what interval you like.  The Monitor will query your “Locations” passwords database table and retrieve the VMWare user listed just like the original Labtech probes do. The monitor get the CIM health data and returns it to Labtech, It also looks to see if we are “Not OK” and fires off a email if failures are picked up.

After the Monitor runs you should see data on the Info Tab -> VMWare sub tab

esxdata

You will need to edit the monitor script updating the email address that it reports to when failures are found, you can also modify monitor script to create a ticket, fire off an alarm, set an alert or anything your heart desires. Line 39 of the script ESX Hardware Monitor V2-1 needs to be edited and the example@example.com email changed to the email you want to get the alerts.

The 3rd script is a updater script that will fetch the latest build of the Nagios Plugin:”check_esxi_hardware.py” script maintained by  www.claudiokuenzler.com   You can run this script against any Windows box that has the monitor installed and it will get the latest version of this script and deploy it to that system. This way you can keep up with all the fixes they do to this script. You may want to run this script on the group once or twice a year just to make sure you have the latest fixes and updates.

Enjoy

Cubert 😎

[Solved] ESX VM shutdown stopped at 95% – VM fails to restart

We have several customers on sketchy hardware and on occasion the VM crash due to a SCSI card issue with the mother board used, that aside we have from time to time a need to force a hard reboot of the server running in a VM. Some times it works great and sometimes we have a lockup at 95% and have to force a kill of process that runs the VM to get it to restart.

So here is the process we take to get this to free up and reboot the VM on ESXi 5.0 and later VMware hosts.

 

  1.  Make sure if you do not already have it turned on, to turn on SSH on the ESXi Host. This can be done via the [Configuration -> [Security Profile] using the VMware client.
  2. Using your favorite SSH Client  (Putty), connect to your VMWare ESXi 5 Host.
  3. We now need to get and kill the process group for the VM that has failed. To do this we will look for the process group ID using this command.

    execute -> ps -g|grep “VMName”You should get a return that looks similar to this.

    3372 vmm0:MyVMSystem
    3374 3368 vmx-vthread-4:MyVMSystem 3368 3368 /bin/vmx
    3375 3368 vmx-mks:MyVMSystem 3368 3368 /bin/vmx
    3376 3368 vmx-vcpu-0:MyVMSystem  3368 3368 /bin/vmx

    We are looking for the common number across all processes and in this case that would be “3368” as seen near the end of each line.

  4. Now will need to kill the process. To do this we need to execute -> kill -9 3368   Replace “3368” with the ID number of your system.
  5. Now we need to do some clean up, We need to delete the swap file in the directory where the VM is stored. To get to where we store the swap file you will need to do the following.execute ->cd /vmfs/volumes/<YourDataStore>/<VMName>

    Next we need to make sure what our swap file name is so execute -> ls

    This will give you a directory listing find your swap file by looking for the file extension “.vswp”.  Now we will remove it with this command.

    execute -> rm –r <YourSwapFile.vswp>

     

  6. Now lets restart our VM services, This will not affect any running VM and is safe to run while VMs are active on host.execute -> /sbin/services.sh restart

     

  7. Reconnect your VMware client to the host and complete the process to power on the VM by first removing the VM from the inventory (Do not Delete from Disk) and re-adding it back in. This will reset the VM fully and allow you to restart it. After you remove your VM from inventory you can re-add it by browsing the datastore in your VMWare client finding your VM directory and right clicking on the “.vmx” file.  A menu will pop up and you can click “Add to Inventory” which will place VM back into the available VMs list. Now select the VM and click the Boot up arrow button to get started again.

 

Enjoy,

Cubert 😎

 

[Kaseya Agent Procedure] VMWare ESXi Hardware Health Monitor

VMWare ESXi Hardware Health Monitor Script

From the skunk works here at Squidworks comes another great monitoring script for Kaseya.

 

 

This script uses the SDK provided by VMWare to query the  ESX host and return a good or bad variable.  If the hardware test fails then the script grabs the log of the test and uploads it to Kaseya Server then places it under the “Get File” area for the host that ran the test. You can run this script on any windows box, I have also included the current vSphere SDK installer and a Kaseya script to install it if it is not found on the Windows Host.

Upload the SDK installer and Import the scripts to your public files area in Kaseya under the directory “VMWare”. If you place files anywhere else you will need to edit script for the new location of files.

The script then makes a unique event log entry into the Windows Application Event Log under the Application Events that can then be picked up by Kaseya’s Event Log Monitor. When Kaseya picks up this event you can instruct the monitor to create an alarm, create a ticket, run another agent procedure or email the alarm to an address(s).  Just schedule the agent procedure to run a couple of times a day to keep an eye on your customers VMware vSphere ESXi Hardware health.

This script links to the CIM information provided by the hardware to the ESX host. You will see CPU, Memory, Fans, RAID and Controller Health. The log file that is uploaded will only show failures and will tell you what failed and on what ESX host.

Download -> Kaseya VMWare ESXi Hardware Monitor

 

 

Enjoy

Cubert 😎

How-to get SNMP Data from VMWare ESXi 5 with Dell OpenManage, Cacti, Nagios, Munin or Ganglia

 

So you want to access your VMware ESX 5 host over SNMP? Here is the quick steps to get Dell OpenManage and SNMP on your ESX 5 host up and running fast.  You will need the VIB’s from Dell, also will need to download and install the VMware CLI client for accessing ESX hosts from remote. For the SNMP setups below we are using a generic password inside of quotes(“password”) you should remove the quotes and place the root password of the esx host where you see the quoted password text. We also are setting up SNMP to send traps to 192.168.24.120, you will need to change this to any host you want to receive traps. You will need to replace the server IP address with the address of your ESX5 host. If you use network monitoring applications like Cacti, Nagios, Munin or Ganglia then this is just what you need to start capturing network statistics and graphs.

 

Steps  to install OpenManage.

  1. Downloaded OpenManage VIB from Dell http://www.dell.com/support/drivers/us/en/g_5/DriverDetails/DriverFileFormats?DriverId=N3R0H&FileId=2773322235
  2. Uploaded VIB to ESX host and placed in /var/log/vmware/
  3. SSH to host as root
  4. ran command esxcli software vib install -d /var/log/vmware/OM-SrvAdmin-Dell-Web-6.5.0-542907.VIB-ESX50i_A02.zip
  5. After software install, I rebooted the host.

Steps to configure SNMP

  1. Started vmWare CLI
  2. vicfg-snmp.pl –server 192.168.24.10 –username root –password “password” -c public
  3. vicfg-snmp.pl –server 192.168.24.10 –username root –password “password” -t 192.168.24.120@161/public
  4. vicfg-snmp.pl –server 192.168.24.10 –username root –password “password” -E
    Current SNMP agent settings:
    Enabled  : 1
    UDP port : 161
    Communities :
    public
    Notification targets :
    192.168.24.120@161/public
    Options :
    EnvEventSource=indications

Openmanage Web Interface:

  1. Download OpenManage Server Administrator Managed Node http://www.dell.com/support/drivers/us/en/555/DriverDetails/DriverFileFormats?DriverId=R300521
  2. Navigate to https://localhost:1311, type in ESX host IP address, login and password.
  3. You should be able to view the all the good stuff like “Main System Chassis”, Software, Storage

 

I hope this helps someone out there in cyber space!

Cubert 😎

Error during the configuration of the host: Failed to update disk partition information

VMware ESX 4 -> Failed to update disk partition information.

After creating a new LUN on a Dell MD3000i, I wanted to add the new iSCSI LUN to my ESXi server. Everything went fine, the LUN was presented and the Add storage wizard has found the new LUN. But when i wanted to complete the wizard, the following error came up:

VMware ESX 4 -> Failed to update disk partition information.

After creating a new LUN on a Dell MD3000i, I wanted to add the new iSCSI LUN to my ESXi server. Everything went fine, the LUN was presented and the Add storage wizard has found the new LUN. But when i wanted to complete the wizard, the following error came up:

Now if you go on the Internet a search around you will find several references to this issue. Most if not all of them tell you to:

1. Use fdisk on the /dev/sd[a-z]* device. Create a partition with type 0xfb.
2. Format the VMFS datastore using
vmkfstools -C vmfs3 vmhbaI:T:L:P
In this example, I corresponds to the initiator of the VMware host bus adapter, T corresponds to the target number of the disk, L is the LUN number, and P is the partition number of the newly created partition.

3. Reboot ESX server

This is not always required, It may be just a hickup in your ISCSI registration on the ESX servers. Here is how I fixed mine and it worked with out any reboots or major adjustments.

Steps to take first before following the CLI instructions:

  1. Go create another 5Gb LUN on Power Vault.
  2. Set the access group assigned to new LUN to allow ESX servers
  3. On ESX Host GUI go back to configuration menu and select Storage Adapters menu
  4. Select your ISCSI Software Adapter from Storage Adapter list and select “Rescan” (even if you see your LUN)
  5. Go back to the Storage menu and select add storage, select Disk/LUN and follow through the menus to mount and format the 5gb LUN and it should go through now.
  6. Remove the 5gb LUN from the ESX Storage menu.
  7. Remove the 5gb LUN from Dell Vault.
  8. Go back to your ESX and rescan HBAs and make sure the 5gb LUN is gone and you see your original LUN you want to mount.
  9. Go back to the Storage menu and select add storage, select Disk/LUN and follow through the menus to mount and format the  LUN and it should go through now.

If this does not work then change the access to the LUN on the vault to a new access group. Rescan for LUN on ESX then go reset the LUN back to the original access group you want to use and then rescan again from ESX.  Try to remount LUN and format. This should now connect and no errors should appear. If you still are getting errors then you may have other issues and you can now try the CLI method or call in support.

Good Luck, I hope  this helps a few of you out there from dealing with production systems and not wanting to spend all day VMotioning, reboots and such.

Enjoy