Troubleshooting Unresponsiveness Caused by Batch Import Deadlocks

Issue

After a period of activity, such as heavy content and object creation, the Liferay DXP instance becomes unresponsive. Symptoms include:

Inability to accessible Liferay via browser or cURL requests, which eventually time out.
Unresponsiveness after 2-3 hours of uptime, even if the system is idle.

Environment

Quarterly Releases

Resolution

When this issue is caused by one or more stuck entries in the BatchEngineImportTask database table, you must identify and remove the stuck tasks to resolve this issue. To attempt temporarily restoring service in the meantime, restart the Liferay server.

1. Identify Potential Root Cause

While the server is unresponsive, acquire a set of multiple thread dumps
Analyze the thread dumps (or request analysis by Liferay Support)

If the thread dump analysis indicates that the BatchEngineTaskOrphanScannerSchedulerJobConfiguration is causing a deadlock, proceed to the next section. If the analysis does not indicate this, the issue is likely due to a separate root cause.

This job is designed to find and re-trigger stuck batch import/export tasks. However, if tasks are permanently stuck, the job can result in system-wide unresponsiveness in certain situations.

2. Identify Stuck Tasks

Query the BatchEngineImportTask table in your Liferay database.
Identify rows where the endTime column has a NULL value.

These represent tasks that have not completed and are considered stuck.

3. Delete Stuck Tasks

Before proceeding, ensure that you have a recent, full database backup in case a rollback is required.

You'll use the Gogo Shell to delete stuck tasks using the identified batchEngineImportTaskId. To do so, for each stuck task:

Navigate to Control Panel → Gogo Shell.
Execute the following command for each stuck task, replacing [batchEngineImportTaskId] with the actual ID:

BatchEngineImportTaskLocalServiceUtil.deleteBatchEngineImportTask([batchEngineImportTaskId])