Unable to initialize Elasticsearch cluster, health status is in RED

Issue

• When the search cluster is in RED, this results in the Liferay service (DXP) not starting or starting without search capability. A RED status usually means that the search cluster has an inconsistent status because of a data lost or data corruption.
• You can see the following error in the Liferay service startup log:

  [dxp] ERROR [main][PortalInstances:271] java.lang.IllegalStateException: Unable to initialize Elasticsearch cluster: {_clusterHealthStatus=RED, _healthStatusMessage='{"cluster_name":"liferay_cluster","status":"red","timed_out":true,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":22,"active_shards":728,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":8,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":90.29629629629629}'}

• The error can be verified by accessing to the shell of the search service and using the REST api:
  curl -X GET search:9200/_cat/health
  curl -X GET search:9200/_cluster/allocation/explain?pretty

Solution

• You have a couple of ways to solve this:
  
  OPTION 1 (Recommended)
  
  
  1. Open a Shell in a search instance and check what indexes are affected by running curl -X GET search:9200/_cat/indices and for each one in red, delete it by running curl -X DELETE search:9200/name-of-the-index-in-red
     The elasticsearch cluster will go to YELLOW and then to GREEN. You can check this in the search service logs or with a  curl -X GET search:9200/_cat/health
  2. Restart Liferay service to get the search engine connection working.
  3. In the Liferay DXP instance, sign-in as an administrator and execute a full reindex from the Control Panel > Configuration > Search to recover the information in the indexes.
  
  OPTION 2
  
  If the last option does not work, this is the way to get a clean/empty search installation:
  
  1. Configure a different name for the volume mounted in the data path of the search service, you can modify the name in the search/LCP.json
  2. Delete the running search service from the console.
  3. Deploy a build with the changes.
  4. Restart Liferay if the last step does not include the Liferay service, this will get the search engine connection working.
  5. In the Liferay DXP instance, sign-in as an administrator and execute a full reindex from the Control Panel > Configuration > Search to recover the information in the indexes.
  
  The handicap of the second option is that you have to repeat the process in the other environments even if they have a healthy search cluster, this is because it involves a new build changing the search's volume and if you want to have portable builds you should align the volume names in the different environments.

Additional Information

• This fix allows Liferay DXP service to start in a situation like this, but losing the search capability: LPS-121782