Failed upgrade process for module com.liferay.portal.workflow.kaleo.service: Unable to convert XML to JSON

Issue

We are currently upgrading our DXP 7.4 u92 DB to 2024.Q1.12

We configured things for DB upgrade based on https://learn.liferay.com/w/dxp/installation-and-upgrades/upgrading-liferay/upgrade-basics/using-the-database-upgrade-tool and ran the upgrade via DB upgrade tool.

However, the upgrade fails with error:
```
INFO - Failed upgrade process com.liferay.portal.workflow.kaleo.internal.upgrade.v4_0_0.KaleoDefinitionContentUpgradeProcess in 349 ms
ERROR - Failed upgrade process for module com.liferay.portal.workflow.kaleo.service
com.liferay.portal.kernel.upgrade.UpgradeException: com.liferay.portal.kernel.workflow.WorkflowException: Unable to convert XML to JSON
```
We are not using Kaleo Workflows currently in our project. However, I do see that there are a few custom Kaleo Definitions that were created and modified 6-9 years ago for test purposes. At that time, we were on Liferay DXP 7.0, and before that on Liferay 6.2 when those definitions were created.

Since then we upgraded from 6.2 to 7.0 to 7.2 to 7.4 (u92) and the upgrade never failed due to Kaleo related errors.

We also checked the source for 2024.Q1.12 and it seems that the parsing of XML to convert it into a JSON is failing which didn't happen in the earlier upgrades.
In the meantime, we managed to proceed with the DB upgrade by removing the problematic entries from DB, but we would be interested in knowing the cause behind the issue happening while upgrading to 2024.Q1.12 and not with earlier upgrades to 7.4 u92 or even to 7.2 and 7.0

Environment

Resolution

In some old versions like 7.1 there is not an XML validation before saving the workflow definition content, but after these changes the validation had to be applied to convert the XML string to a JSON object, those workflow definitions are not valid due to this syntax constraint:

"The ampersand character (&) and the left angle bracket (<) must not appear in their literal form, except when used as markup delimiters, or within a comment, a processing instruction, or a CDATA section. If they are needed elsewhere, they must be escaped using either numeric character references or the strings " & " and " < " respectively. The right angle bracket (>) may be represented using the string " > ", and must, for compatibility, be escaped using either " > " or a character reference when it appears in the string " ]]> " in content, when that string is not marking the end of a CDATA section.”

We also use now XML validation on frontend side to format appropriately it in the Process Builder UI (see attached screenshot)

At the time of writing this article, our engineers are working on a way to escape the entire XML file applying the constraints mentioned below before parsing it to a JSON object, on https://liferay.atlassian.net/browse/LPD-22890