Node in Cluster Fails to Start with Deadlock and Circular Reference Errors

Issue

When starting a Liferay DXP cluster, the node fails to start and gets stuck. The thread dump show a combination of errors, including deadlocks, circular references, and class not found exceptions.

Key errors in the thread dumps may include:

	- locked <0x000000040d331478> (a com.liferay.petra.concurrent.DCLSingleton)
	at com.liferay.portal.kernel.module.service.Snapshot.lambda$new$3(Snapshot.java:109)
	at com.liferay.portal.kernel.module.service.Snapshot$$Lambda$290/0x00000008005a1840.get(Unknown Source)
	at com.liferay.portal.kernel.module.service.Snapshot.get(Snapshot.java:157)
	at com.liferay.portal.kernel.dao.orm.FinderCacheUtil.getFinderCache(FinderCacheUtil.java:42)
	at com.liferay.portal.kernel.dao.orm.FinderCacheUtil.getResult(FinderCacheUtil.java:55)
	at com.liferay.portal.service.persistence.impl.ClassNamePersistenceImpl.findAll(ClassNamePersistenceImpl.java:685)
	at com.liferay.portal.service.persistence.impl.ClassNamePersistenceImpl.findAll(ClassNamePersistenceImpl.java:645)
	at com.liferay.portal.service.persistence.impl.ClassNamePersistenceImpl.findAll(ClassNamePersistenceImpl.java:610)
	at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.checkClassNames(ClassNameLocalServiceImpl.java:56)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.2/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.2/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.2/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.2/Method.java:566)
	at com.liferay.portal.spring.aop.AopMethodInvocationImpl.proceed(AopMethodInvocationImpl.java:41)
	at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:60)
	at com.liferay.portal.spring.aop.AopMethodInvocationImpl.proceed(AopMethodInvocationImpl.java:48)
	at com.liferay.portal.spring.aop.AopInvocationHandler.invoke(AopInvocationHandler.java:40)
	at com.sun.proxy.$Proxy78.checkClassNames(Unknown Source)
	at com.liferay.portal.kernel.service.ClassNameLocalServiceUtil.checkClassNames(ClassNameLocalServiceUtil.java:58)
	at com.liferay.portal.spring.context.PortalContextLoaderListener.lambda$_contextInitialized$4(PortalContextLoaderListener.java:489)
	at com.liferay.portal.spring.context.PortalContextLoaderListener$$Lambda$989/0x0000000800fae840.accept(Unknown Source)
	at com.liferay.portal.db.partition.util.DBPartitionUtil.forEachCompanyId(DBPartitionUtil.java:178)
	at com.liferay.portal.spring.context.PortalContextLoaderListener._contextInitialized(PortalContextLoaderListener.java:488)
	at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:213)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.2/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.2/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.2/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.2/Method.java:566)
	at com.liferay.shielded.container.internal.proxy.ContextClassLoaderInvocationHandler.invoke(ContextClassLoaderInvocationHandler.java:34)
	at com.liferay.shielded.container.internal.proxy.EventListenerInvocationHandler.invoke(EventListenerInvocationHandler.java:55)
	at com.sun.proxy.$Proxy6.contextInitialized(Unknown Source)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4460)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4914)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
	- locked <0x00000004008ac830> (a org.apache.catalina.core.StandardContext)

A thread dump analysis reveals a deadlock involving multiple threads waiting on locks, particularly on com.liferay.petra.concurrent.DCLSingleton.getSingleton.

Environment

Liferay DXP 2024.Q1

Resolution

The combination of errors indicates two separate issues: a deadlock caused by a deprecated JVM property and a class deserialization problem in the cluster communication, which may be due to a product defect or inconsistent module deployments.

To resolve these issues, follow these steps:

Address the Deadlock:
- In your application server's startup script (e.g., setenv.sh or setenv.bat), locate and remove the following deprecated JVM argument:
```
-Dcompany-id-properties=true
```
- This property has been deprecated since Liferay Portal 6.2 and can cause deadlocks during the startup of modern DXP versions.
Restart the Cluster:
- Perform a full shutdown of all nodes in the cluster.
- On each node, clear the following temporary directories:
  - [Liferay Home]/work
  - [Liferay Home]/osgi/state
  - [Tomcat Home]/temp
- Start the cluster nodes one at a time, allowing the first node to fully start before starting the next.

Additional Information

Clustering for High Availability