Service Health Check
NCache provides a Service Health Check mechanism to determine whether the caches running on a server remain accessible and responsive.
Conventional process monitoring can detect when an NCache process terminates, but it cannot always detect a process that remains running while becoming unresponsive because of an internal failure or deadlock. In such cases, the operating system can continue to consider the process active even though NCache is no longer responding through the communication paths required by clients or management operations.
Therefore, Service Health Check addresses this by monitoring NCache health independently of the process state and exposing the resulting health status to the hosting environment. Kubernetes and cloud environments can then use this status to determine when recovery action is required.
How Service Health Check Works
The NCache Service acts as the common health-monitoring component on each NCache server. It periodically checks the locally running CacheHost processes in the background and maintains their latest health status.
The health status is exposed through a REST health endpoint that can be consumed by the deployment environment. In the background, the NCache Service determines this status by verifying connectivity to the Cache Host port. Since health monitoring is performed periodically, external probes can obtain the latest status without initiating the complete health-check sequence for every request.
The checking process is as follows:
- The NCache Service monitors the locally running caches.
- It verifies the communication paths required to access those caches.
- The latest health status is maintained by the service.
- The hosting environment queries the NCache health endpoint.
- Based on its configured health policy, the hosting environment determines whether recovery action is required.
This provides a common NCache health check mechanism while allowing each platform to use its native monitoring and recovery capabilities.
Determining Cache Health
Currently, Service Health Check focuses on verifying whether the locally running CacheHost processes remain accessible through the communication paths required for normal connectivity. Essentially, the NCache Service verifies the following that the service can communicate with the locally running cache through its management interface and that the cache remains accessible through the communication path used for client connections.
If these communication paths become unavailable or unresponsive, the health check mechanism can report the cache as unhealthy even when the underlying process is still running.
Platform Integration
The same NCache health check mechanism can be used across different deployment environments. Each platform consumes the health status exposed by NCache and applies its own timing, failure threshold, and recovery policy.
Kubernetes
Kubernetes can use the NCache health check mechanism through its standard health probes:
- Startup Probe determines whether NCache has initialized successfully.
- Readiness Probe determines whether the NCache instance is ready to receive traffic.
- Liveness Probe determines whether the NCache instance remains operational.
Kubernetes periodically queries the corresponding health endpoint and uses its configured probe interval, timeout, and failure threshold to determine the health of the container. If the liveness probe continues to fail, Kubernetes can restart the NCache pod according to its configured recovery policy. This allows Kubernetes to detect an NCache process that is still running but no longer responding through its required communication paths.
Cloud Environments
Cloud environments such as Azure Virtual Machine Scale Sets (VMSS), and AWS Auto Scaling Groups (ASG) can use the NCache health status when determining whether a VM remains operational.
If NCache continues to report an unhealthy state, the cloud environment can initiate its configured recovery action. Depending on the deployment, this may include restarting the existing VM or replacing the unhealthy instance with another VM.
When a node is removed or replaced, the NCache cluster handles the resulting node leave and join through its normal scale-down and scale-up cluster behavior. The remaining cache servers continue operating, and data is redistributed according to the configured cache topology. When the recovered or replacement node rejoins the cluster, data distribution is adjusted accordingly.
