I was recently tasked with an issue where our CIM probe was failing during CIM requests to new VMWare ESXi 6.5 servers we deployed. We were getting connection rejected failures from our probes which resulted in no valuable data being returned. We started following the breadcrumbs which lead us back to the ESXi host. We opened the UI and checked the health monitor in the UI and found it was showing “No sensor data available”. The first thing we checked was to see if the sfcbd-watchdog was running, and it was not. By default, this service was turned off, or so we thought! We turned on the service and the UI reported that the service was now running.
Even after several refreshes of the UI it stilled showed running but we still received a connection rejected. We rebooted the ESXi host and after it came back we tested the connections again and are still failing. We reopen the web UI and looked at the services again and there was our watchdog service stopped. We had set the service to autostart with host so this lead us to believe it must be dying at some point.
The best way to see what a service doesn’t like is to login to ESXi host using SSH and manually start the process and see what it’s output is. A quick /etc/init.d/sfcbd-watchdog start showed us that the service was “Administratively disabled”.
After digging around Google for some reference to this new data we came across a blurb about setting an option to allow CIM manager to run.
The command esxcli system wbem set –enable true followed by /etc/init.d/sfcbd-watchdog start allowed the sfcb-HTTPS-Daem process to start. This process is the TCP Listener that takes CIM requests from probes like ours and returns the health of the hardware.
You should get an output like the following
/etc/init.d/sfcbd-watchdog start
sfcbd-init: Getting Exclusive access, please wait…
sfcbd-init: Exclusive access granted.
sfcbd-init: Request to start sfcbd-watchdog, pid 69438
sfcbd-config[69448]: No third party cim providers installed
sfcbd-init: snmp has not been enabled.
sfcbd-init: starting sfcbd
sfcbd-init: Waiting for sfcb to start up.
sfcbd-init: Program started normally.
Invoking lsof -nPV | awk {‘count[$2]++}END{for(i in count)print count[i], i’} | sort -n in the SSH console will produce a list of running processes minus all the junk. You can use this list of processes to determine what is running on the ESXi Host.
We also used esxcli network ip connection list to get a list of ports the ESXi host was listening on to help determine if the port 5989 was active.
If you are deploying VMWare ESXi 6.5 in your environments and need CIM health data, remember to enable it and do not just assume that the WebUI is telling you it is active.
Check out our ESXi Health Monitor for LabTech (Automate) here
