How to fix | ESXI Virtual SAN Health service installation

I encountered an issue with the ESXi Virtual SAN Health service installation in one of the vSAN cluster,

Step 1 :

I checked whether all the ESXi hosts are running on the same version or not,
VMware ESXi 6.0.0 build-5224934 VMware ESXi 6.0.0 Update 3 on ESX1
VMware ESXi 6.0.0 build-5224934 VMware ESXi 6.0.0 Update 3 on ESX2
VMware ESXi 6.0.0 build-5224934 VMware ESXi 6.0.0 Update 3 on ESX3
VMware ESXi 6.0.0 build-5224934 VMware ESXi 6.0.0 Update 3 on ESX5
VMware ESXi 6.0.0 build-5224934 VMware ESXi 6.0.0 Update 3 on ESX4
They are in the same version so we can go check whether vSAN health VIB is installed or not.

From the KB https://kb.vmware.com/s/article/2109874,
On vSphere 6.0 Update 2 release, none of the other health checks will be conducted until all the hosts are upgraded to 6.0 Update 2 (when running the latest version, vSAN 6.2) release to avoid false alarms.  But we have all the ESXi hosts in ESXi 6.0 Update 3
"Install the vSAN Health Service VIB on all the ESXi hosts that are participating in the vSAN cluster, or update the ESXi host to the version that is indicated in the error field. "

Step 2: 

We have all the ESXi hosts in the same version so no need this option is ignored. The only available option is to check if the vSAN health service VIB installed on the host or not.
vsanhealth                6.0.0-3000000.3.0.3.58.5224738         VMware  VMwareCertified   2017-04-13   on ESX1
vsanhealth                6.0.0-3000000.3.0.3.58.5224738         VMware  VMwareCertified   2017-04-12   on ESX2
vsanhealth                6.0.0-3000000.3.0.3.58.5224738         VMware  VMwareCertified   2017-04-13   on ESX3
vsanhealth                6.0.0-3000000.3.0.3.58.5224738         VMware  VMwareCertified   2017-04-13   on ESX5
vsanhealth                6.0.0-3000000.3.0.3.58.5224738         VMware  VMwareCertified   2017-04-13   on ESX4
And the VIB is installed on all the ESXi hosts in the cluster.

Step 3 : 

So te next is to check the vsangmgtd health state on the ESXi host ESX4 where ESXi Virtual SAN Health service installation status is uknown.
[root@esx4:/var/log] /etc/init.d/vsanmgmtd status
vsanperfsvc is running
[root@esx4:/var/log] cat /var/log/vsanmgmt.log
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2017-11-28T04:48:23Z VSANMGMTSVC: ERROR vsanperfsvc[MainThread] [statsdaemon::logDaemonMemoryStats] Failed to get memory stats: Bad parameter
Traceback (most recent call last):
  File "/build/mts/release/bora-5224738/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 580, in logDaemonMemoryStats
  File "/build/mts/release/bora-5224738/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 593, in _checkAndLogMemoryStats
ValueError: Bad parameter
2017-11-28T04:48:28Z VSANMGMTSVC: ERROR vsanperfsvc[MainThread] [cliutils::GetVSANClusterInfo] Failed to query cluster info: Failed to alloc memory for text
Traceback (most recent call last):
  File "/build/mts/release/bora-5224738/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/cliutils.py", line 69, in GetVSANClusterInfo
  File "/usr/lib/vmware/hostd/hmo/VsanInternalSystem.py", line 205, in DecodeEntry
    return eval(pyCMMDS.BinToTextPolicy(e.dataStr, True, style))
RuntimeError: Failed to alloc memory for text
2017-11-28T04:48:28Z VSANMGMTSVC: ERROR vsanperfsvc[MainThread] [statsdaemon::logDaemonMemoryStats] Failed to get memory stats: Bad parameter
Traceback (most recent call last):
  File "/build/mts/release/bora-5224738/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 580, in logDaemonMemoryStats
  File "/build/mts/release/bora-5224738/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 593, in _checkAndLogMemoryStats
ValueError: Bad parameter
2017-11-28T04:48:33Z VSANMGMTSVC: ERROR vsanperfsvc[MainThread] [cliutils::GetVSANClusterInfo] Failed to query cluster info: Failed to alloc memory for text
Traceback (most recent call last):
  File "/build/mts/release/bora-5224738/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/cliutils.py", line 69, in GetVSANClusterInfo
  File "/usr/lib/vmware/hostd/hmo/VsanInternalSystem.py", line 205, in DecodeEntry
    return eval(pyCMMDS.BinToTextPolicy(e.dataStr, True, style))
RuntimeError: Failed to alloc memory for text
++++++++++++++++++++++++++++++++++++++++++++++++++++++
[root@esx4:~] /etc/init.d/vsanmgmtd restart
watchdog-vsanperfsvc: Terminating watchdog process with PID 34190
vsanperfsvc started
After restarting the vsanmgmtd service, ESXI Virtual SAN Health service installation error has been disappeared and all the vSAN healths are available now.

In case if you want to install the health service on ESXi there are four installation options for the health VIB:
1. Enable health check plugin via vSphere web client
2. RVC, the Ruby vSphere Console
3.VUM, the VMware Update Manager. Use of VUM is outside the scope of this document and users are directed to the official vSphere documentation on how to use VUM to install VIBs.
4.Manual install at the ESXi command line. This is done using ESXCLI software VIB commands.
VMware recommends option 1 for ease of install, but the steps to install via option 2 are also covered here.

Read: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/products/vsan/vmw-gdl-vsan-health-check.pdf ( From Page 12)