This article is designed to help system administrators troubleshoot a Liferay DXP instance deployed in a clustered environment. Some issues may arise because of too restrictive firewall settings (most, if not all, systems are protected by a firewall). And if certain ports are not opened on the firewall, this will cause some random cluster issues. This is because Liferay platform's ClusterLink uses JGroups for its underlying technology, which uses specific addresses + ports for cluster nodes communications.
Here are some potential clustering issues due to firewall configurations.
- Cache cannot be replicated between cluster nodes (updates on one node can not be replicated to the other nodes in cluster).
- A cluster cannot be formed despite correct cluster settings.
These are only symptoms; some of the root causes are listed below:
Environment
- Liferay DXP 7.1
- Liferay DXP 7.0
Resolution
Issue: Blocked Multicast Cluster Nodes Internal Communications Protocols
To make sure cluster nodes can communicate with each other, there are several nodes that need to be opened on the firewall:
#
# See the property "cluster.link.channel.properties.control".
#
multicast.group.address["cluster-link-control"]=239.255.0.1
#multicast.group.address["cluster-link-control"]=ff0e::8:8:1
multicast.group.port["cluster-link-control"]=23301
#
# See the properties "cluster.link.channel.properties.transport.0" and
# "cluster.link.channel.system.properties".
#
multicast.group.address["cluster-link-udp"]=239.255.0.2
#multicast.group.address["cluster-link-udp"]=ff0e::8:8:2
multicast.group.port["cluster-link-udp"]=23302
#
# See the property "cluster.link.channel.system.properties".
#
multicast.group.address["cluster-link-mping"]=239.255.0.3
#multicast.group.address["cluster-link-mping"]=ff0e::8:8:3
multicast.group.port["cluster-link-mping"]=23303
#
# See the properties "ehcache.multi.vm.config.location" and
# "ehcache.multi.vm.config.location.peerProviderProperties".
#
multicast.group.address["multi-vm"]=239.255.0.5
multicast.group.port["multi-vm"]=23305
Issue: Ports Are Randomly Chosen by JGroups
According to JGroups's documentation,
UDP uses IP multicast for sending messages to all members of a group and UDP datagrams for unicast messages (sent to a single member). When started, UDP opens a unicast and multicast socket: the unicast socket is used to send/receive unicast messages, while the multicast socket sends and receives multicast messages. (emphasis added)
By default, JGroups chooses the unicast port at random. For example, as indicated in the portal startup log, 55091 and 52920 are randomly chosen:
GMS: address=emma-12723, cluster=liferay-channel-control, physical address=192.168.1.115:55091 GMS: address=emma-15524, cluster=liferay-channel-transport-0, physical address=192.168.1.115:52920
Therefore these two ports need to be opened on firewall.
If administrators want to fix which ports are allowed through the firewall, they can define the bind_port property in the portal-ext.properties:
cluster.link.channel.properties.control=UDP(bind_addr=${cluster.link.bind.addr["cluster-link-control"]};bind_port=8888;mcast_group_addr=${multicast.group.address["cluster-link-control"]};mcast_port=${multicast.group.port["cluster-link-control"]}):PING:MERGE3(min_interval=10000;max_interval=30000):FD_SOCK:FD_ALL:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK2(xmit_interval=500;xmit_table_num_rows=100;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=30000;max_msg_batch_size=500;use_mcast_xmit=false;discard_delivered_msgs=true):UNICAST3(xmit_interval=500;xmit_table_num_rows=100;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=60000;conn_expiry_timeout=0;max_msg_batch_size=500):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=4M):pbcast.GMS(join_timeout=2000;print_local_addr=true;view_bundling=true):UFC(max_credits=2M;min_threshold=0.4):MFC(max_credits=2M;min_threshold=0.4):FRAG2(frag_size=60K):RSVP(resend_interval=2000;timeout=10000)
cluster.link.channel.properties.transport.0=UDP(bind_addr=${cluster.link.bind.addr["cluster-link-udp"]};bind_port=9999;mcast_group_addr=${multicast.group.address["cluster-link-udp"]};mcast_port=${multicast.group.port["cluster-link-udp"]}):PING:MERGE3(min_interval=10000;max_interval=30000):FD_SOCK:FD_ALL:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK2(xmit_interval=500;xmit_table_num_rows=100;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=30000;max_msg_batch_size=500;use_mcast_xmit=false;discard_delivered_msgs=true):UNICAST3(xmit_interval=500;xmit_table_num_rows=100;xmit_table_msgs_per_row=2000;xmit_table_max_compaction_time=60000;conn_expiry_timeout=0;max_msg_batch_size=500):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=4M):pbcast.GMS(join_timeout=2000;print_local_addr=true;view_bundling=true):UFC(max_credits=2M;min_threshold=0.4):MFC(max_credits=2M;min_threshold=0.4):FRAG2(frag_size=60K):RSVP(resend_interval=2000;timeout=10000)