Network Systems Engineers Consortium
If you are using ConnectWise Automate or LabTech RMM tool then this is a must have plugin for your environment. Plugins4Automate has a great plugin for managing ESXi Health monitoring in ConnectWise Automate and LabTech.
Visit them at www.plugins4automate.com
I was recently tasked with an issue where our CIM probe was failing during CIM requests to new VMWare ESXi 6.5 servers we deployed. We were getting connection rejected failures from our probes which resulted in no valuable data being returned. We started following the breadcrumbs which lead us back to the ESXi host. We opened the UI and checked the health monitor in the UI and found it was showing “No sensor data available”. The first thing we checked was to see if the sfcbd-watchdog was running, and it was not. By default, this service was turned off, or so we thought! We turned on the service and the UI reported that the service was now running.
Even after several refreshes of the UI it stilled showed running but we still received a connection rejected. We rebooted the ESXi host and after it came back we tested the connections again and are still failing. We reopen the web UI and looked at the services again and there was our watchdog service stopped. We had set the service to autostart with host so this lead us to believe it must be dying at some point.
The best way to see what a service doesn’t like is to login to ESXi host using SSH and manually start the process and see what it’s output is. A quick /etc/init.d/sfcbd-watchdog start showed us that the service was “Administratively disabled”.
After digging around Google for some reference to this new data we came across a blurb about setting an option to allow CIM manager to run.
The command esxcli system wbem set –enable true followed by /etc/init.d/sfcbd-watchdog start allowed the sfcb-HTTPS-Daem process to start. This process is the TCP Listener that takes CIM requests from probes like ours and returns the health of the hardware.
You should get an output like the following
/etc/init.d/sfcbd-watchdog start
sfcbd-init: Getting Exclusive access, please wait…
sfcbd-init: Exclusive access granted.
sfcbd-init: Request to start sfcbd-watchdog, pid 69438
sfcbd-config[69448]: No third party cim providers installed
sfcbd-init: snmp has not been enabled.
sfcbd-init: starting sfcbd
sfcbd-init: Waiting for sfcb to start up.
sfcbd-init: Program started normally.
Invoking lsof -nPV | awk {‘count[$2]++}END{for(i in count)print count[i], i’} | sort -n in the SSH console will produce a list of running processes minus all the junk. You can use this list of processes to determine what is running on the ESXi Host.
We also used esxcli network ip connection list to get a list of ports the ESXi host was listening on to help determine if the port 5989 was active.
If you are deploying VMWare ESXi 6.5 in your environments and need CIM health data, remember to enable it and do not just assume that the WebUI is telling you it is active.
Check out our ESXi Health Monitor for LabTech (Automate) here
Well it’s finally here, the plugin we have all been waiting for, Vmware ESXi Health Monitor plugin for Labtech. This plugin installs into your Labtech system as a Location plugin to monitor the CIM data available in most hardware. Easy to configure controls and full view of the CIM data collected is just part of what this plugin can do. We have incorporated new functions into this plugin that are stark differences from our earlier version 2 and version 1 ESXi Health Monitors. We are now supporting multiple usernames and passwords per location for ESX hosts and only require 1 probe system to monitor all the ESX hosts at a single location. I could talk all day about the plugin but maybe it’s better if I show you.
Here is the main view of the ESXi Health Monitor plugin.
The hosts get listed with status face based on current status and when you select a host the CIM data is displayed so you can see all the reported statuses of the hardware.
This is the ESXi host configuration tab of the ESXi health Monitor plugin.
This is where you will add and edit your host systems to monitor, you can set the system you want to be responsible for probing the ESXi hosts for the CIM data. Then you select a system and “Set” it as the probe the plugin will launch a script against the probe to prep it for monitoring automatically. You will not need to “install” the probe software manually as this is handled by the plugin when the probe system is selected.
The ESXI Health Monitor plugin uses a custom group to locate all the probes available by using a custom search to locate all systems with the EDF of “VMware Master CIM Scanner” selected as seen it the example below. You will find this setting under the Info tab -> VMWare on any system console. Just checking it will not install probe software so you must “Set” the probe via the plugin.
The Custom group should look like this and have the custom search setup as seen in this example.
The custom group also has a scheduled script to run every 2-4 hours and I use the exclude time range as I do not need this data so bad that I can’t sleep with out it running every hour or 2. This is just my personal preference but saves on CPU cycles during backup windows and maintenance schedules. You will need to reset this when you import your group as this seems to always get rest to nothing during imports.
You can see your probe run on the system by watching the script logs on the probe systems console. This helps when troubleshooting common issues.
This version is in Beta! this is the first release of this plugin and as such may have odd behavior issues and or may not work for you as expected. I seriously doubt you will have any issues but as this is Beta expect a few minor glitches. We are actively working on updates and with your help we can make this a great plugin!
You can run the version 2 and version 3 side by side and they will not effect each other.
Changed Version number back to what it should of started at. It was a typo starting with (3)
Added Internal monitor called CIM – VMWare ESXi Health Monitor
Added Client View and Global View of System Statuses
Corrected a few minor coding mistakes
Added Linux Probe Support
Updated the data views
Added Last Scan Time Stamps
Added color coded data views
Added new images
Fixed Scan Time not updating
Updated Python Packages to 2.7.8
Fixed several SQL issues with table creation.
Minor enhancements
Added new instant host probe from the configure tab
Fixed minor issues in plugin
Fixed issues with installer script
Fixed CIM Monitor in LT
Fixed Versioning context
Fixed password bug when resaving the same password for esx probe.
Fixed Issues with Installer where latest PY script requires several new Python modules.
New Client Level View
This view give you a look at all systems under the clients control. you can select the system and review the CIM data returned.
This plugin monitors the condition all the systems and will alarm when a system is found to be in error. You can customize the alarms and the methods of alerts received through the Monitors management interface in Labtech .
You can download the latest version here.
If you like what we are doing then please donate to our cause, help keep our software free.
Please post here any comments and issues you may have so we can get them fixed and out in the next release.
Enjoy
Cubert 😎
Wouldn’t it be really cool if you could somehow safely access any VMWare vSphere ESX 5 host directly just using the local vSphere 5 Client installed on your workstation without porting and NATing traffic through your customers firewalls? With Labtech that is no problem, by setting up a Application Redirector you can create a new proxy that will pass all your traffic to any ESX or vCenter host and allow you to fully manage that host using your own installed vClient on your workstation, using the Labtech server and an agent, wow is it fast.
I want to thank the guys over at www.labtechgeek.com for creating the outline I am following here.
1.) You will need to create a new redirected app named vSphere Client.
Goto [System Dashboard] -> [Config] -> [Redirected Apps]
2.) In the program field, enter the location of your local vShpere client like -> C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe
3.) Now we will redirect the following ports.
Local Port: 443 Local IP: 127.0.0.1 Remote Port: 443 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 902 Local IP: 127.0.0.1 Remote Port: 902 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 903 Local IP: 127.0.0.1 Remote Port: 903 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 8080 Local IP: 127.0.0.1 Remote Port: 8080 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 9443 Local IP: 127.0.0.1 Remote Port: 9443 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 10080 Local IP: 127.0.0.1 Remote Port: 10080 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 10443 Local IP: 127.0.0.1 Remote Port: 10443 Remote IP: %RemoteIP% Type: TCP Local Listen
Local Port: 902 Local IP: 127.0.0.1 Remote Port: 902 Remote IP: %RemoteIP% Type: UDP Local Listen
Redirector type should have the “Computer” box checked, this makes the redirector show up on computer consoles along side the other redirectors.
4.) Now we need to create an entry in your hosts file for the redirector to work. Add -> “127.0.0.1 vsphere-redir” to your local host file. If you just use 127.0.0.1 or localhost when the client pops up then the client may actually try to connect to the NetBIOS name of your PC, which will not work.
5.) Reload your systems cache and you should see the redirector show up under Computer consoles Redirectors menu.
To connect directly to an ESXi host, while holding the Shift click on the vSphere Client redirector from meun. This will prompt you for an IP address for the remote ESX host you want to control – enter the IP of the ESXi host.
Once you place your IP of the ESX host in the IP for Redirection box click OK. Give it a few seconds to get the proxy up and app to launch then you should see your VMware client pop up. You now type in to IPaddress/Name area “vsphere-redir” as the host IP and then the user and pass needed to log in to the ESX host.
***Note – When you’re connecting directly to a host, or to vCenter, you must always enter vsphere-redir as the IP Address/Name in the VMWare vSphere client.
***Note – If you’re connecting to a vCenter server, you won’t be able to view the console of any VMs (MKS) – this is because the vSphere Client makes a direct connection to the ESXi server on port 902. If you connect directly to the ESXi host, MKS works fine.
Enjoy,
Cubert 😎
From the skunk works here at Squidworks comes another great monitoring script for Kaseya.
This script uses the SDK provided by VMWare to query the ESX host and return a good or bad variable. If the hardware test fails then the script grabs the log of the test and uploads it to Kaseya Server then places it under the “Get File” area for the host that ran the test. You can run this script on any windows box, I have also included the current vSphere SDK installer and a Kaseya script to install it if it is not found on the Windows Host.
Upload the SDK installer and Import the scripts to your public files area in Kaseya under the directory “VMWare”. If you place files anywhere else you will need to edit script for the new location of files.
The script then makes a unique event log entry into the Windows Application Event Log under the Application Events that can then be picked up by Kaseya’s Event Log Monitor. When Kaseya picks up this event you can instruct the monitor to create an alarm, create a ticket, run another agent procedure or email the alarm to an address(s). Just schedule the agent procedure to run a couple of times a day to keep an eye on your customers VMware vSphere ESXi Hardware health.
This script links to the CIM information provided by the hardware to the ESX host. You will see CPU, Memory, Fans, RAID and Controller Health. The log file that is uploaded will only show failures and will tell you what failed and on what ESX host.
Download -> Kaseya VMWare ESXi Hardware Monitor
Enjoy
Cubert 😎
VMware vSphere vCenter Server 5.1 now uses a new SSO (Single Sign On) service to authenticate with Microsoft Active Directory when deploying vCenter. If you do not install this services and configure it for AD then you will not be able to use your domain accounts with vCenter 5.1 During the initial install you may get errors when installing SSO. KB 2034374 reports that a error of ” Error 29155 Identity source discovery error” is due to a failed attempt to automatically discover your Active Directory domain. Verify that the domain name and DNS are setup correctly.
Now lets setup an AD server in vCenter to allow our Domain Accounts. First we will login to vCenter Web Client (https://127.0.0.1:9443) if you used the default ports for the web client installs. The default login is admin@system-domain and the password you set for SSO during the install process. Once you are logged in to the web client you can continue.
Now Select [Administration]
Now Select [Sign-On and Discovery] -> [Configuration]
Under the Identity Sources Tab in the right pane select the PLUS symbol to add a new AD source. This will pop up a “Add Identity Source” window, select the active directory radio button and fill out the requested information with you AD Domain name and the “OU” the holds your users and groups.
Here is the generic information you will need just replace the sesenviron.local with your domain and then place your AD credentials at the bottom.
Now that we have a AD server assigned as a source we must now add this newly created connector to our “Default Domains ” list.
Now that we have it in our Default Domains list lets move it up to be our primary source. To do this highlight the AD domain name and select the blue arrow head pointing up and move the domain name to the top of the list .
Now lets select the small floppy disk icon to save the changes to the default domain list box. Once this is complete we should be able to open up the vSphere client application and log in with domain access. You should be using a domain level admin to access vCenter.
I hope this helps some people out there.
Cubert 😎
Getting error message: Install operation failed: The metadata or bundle option must be specified ?
This error popup up after running the following cmd when trying to update our VMware vSphere server.
C:\Program Files\VMware\VMware vSphere CLI\bin>vihostupdate.pl -url https://192.168.xxx.xxx/sdk/webservice -i –bundle c:\temp\upgrade-from-esxi4.0-to-4.1-update01-348481.zip
I am using vSphere CLI 4.0 on a Windows 2003 server, logged in as a domain admin.
I tried several things I found off Google then it occured to me that it may be a issue with permissions. I restarted the CLI with Admin priviledges and Wamo!!
Please wait patch installation is in progress …
The update completed successfully, but the system needs to be rebooted for the changes to be effective.
So if your getting this error this is worth a try.
Cubert
RetrieveStorageInfo request is ignored because the VM is in an invalid state: bad config = false could be produced as a possible error when the VMX file is invalid. During different activities that the ESX system is doing and during this activity the ISCSI disks are lost or something get in the way where a snaphot or some other process gets funked up. Can leave items in your VMX file that are invalid. When you recover from a loss you may find a system in a “Unknown” state.
Look throught your VMX file for odd spaces or items that may have invalid data in it.
Then re-attach the VM and see if it will showup correctly in your Vcenter server.
After creating a new LUN on a Dell MD3000i, I wanted to add the new iSCSI LUN to my ESXi server. Everything went fine, the LUN was presented and the Add storage wizard has found the new LUN. But when i wanted to complete the wizard, the following error came up:
After creating a new LUN on a Dell MD3000i, I wanted to add the new iSCSI LUN to my ESXi server. Everything went fine, the LUN was presented and the Add storage wizard has found the new LUN. But when i wanted to complete the wizard, the following error came up:
Now if you go on the Internet a search around you will find several references to this issue. Most if not all of them tell you to:
1. Use fdisk on the /dev/sd[a-z]* device. Create a partition with type 0xfb.
2. Format the VMFS datastore using
vmkfstools -C vmfs3 vmhbaI:T:L:P
In this example, I corresponds to the initiator of the VMware host bus adapter, T corresponds to the target number of the disk, L is the LUN number, and P is the partition number of the newly created partition.3. Reboot ESX server
This is not always required, It may be just a hickup in your ISCSI registration on the ESX servers. Here is how I fixed mine and it worked with out any reboots or major adjustments.
Steps to take first before following the CLI instructions:
If this does not work then change the access to the LUN on the vault to a new access group. Rescan for LUN on ESX then go reset the LUN back to the original access group you want to use and then rescan again from ESX. Try to remount LUN and format. This should now connect and no errors should appear. If you still are getting errors then you may have other issues and you can now try the CLI method or call in support.
Good Luck, I hope this helps a few of you out there from dealing with production systems and not wanting to spend all day VMotioning, reboots and such.
Enjoy
