Scaling down an Elasticsearch cluster

Issue

• You might face a scenario where you need to scale down your current Elasticsearch cluster to a fewer node setup

Environment

• Liferay DXP 7.0+
• Elasticsearch

Resolution

• Preparations Before scaling down
  
  • Back up your cluster to have something to restore if things go wrong down the line.
  • Check with the master nodes configuration minimum master nodes according to the
  • Stop all writes to your cluster as it will not be safe to failover after our downscaling, but not mandatory if all goes fine.
  • Make sure that you are not overloading the cluster by making it too small disk space and memory, else cluster will become read only with Low disk watermark.
  • Bring down the index replication factor to 1 in order to save space and speed up shard relocation during scaling, since less shards need to be created and moved around. Also, this saves a lot of space in duplicated data.
    curl -X PUT "localhost:9200/twitter/_settings?pretty" -H 'Content-Type: application/json' -d' { "index" : { "number_of_replicas" : 1 } } '
  • Rebalance the cluster gracefully before you start scaling down.
  • Cluster need to be healthy with green status, check with the shards and status
    • Health: curl -X GET "localhost:9200/_cluster/health?pretty"
    • Expected Output: 

      { "cluster_name" : "\"es-data-cluster\"", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

    • Shards: curl -X GET "localhost:9200/_cat/shards"
    • Expected Output: 

      twitter 2 p STARTED 0 0b 172.18.0.2 es-node twitter 1 p STARTED 0 0b 172.18.0.2 es-node twitter 0 p STARTED 0 230b 172.18.0.2 es-node

  When the cluster status is green and all shards are STARTED then you are good to go with scaling down.
   
• Steps to scale down
  
  • Remove one data node — the cluster will go into the yellow state. Now observe the:
    
    • logs of the cluster,
    • check with STARTED, and UNASSIGNED shards
  • Wait for green—then the cluster has replicated the lost shards.
  • Adjust your licenses on the remaining nodes

Additional Information

• Scaling Down an Elasticsearch Cluster:
  • Elasticsearch must be resilient to the failures of individual nodes. It achieves this resilience by considering cluster-state updates to be successful after a quorum of nodes has accepted them. A quorum is a carefully-chosen subset of the master-eligible nodes in a cluster.
  • Quorums must be carefully chosen so the cluster cannot elect two independent masters which make inconsistent decisions, ultimately leading to data loss, to know more please see:
    
    • A new era for cluster coordination in Elasticsearch