Tomcat JDBC Connection Exhaustion

This article helps troubleshooting MySQL when executing the VerifyProcess. Small Max JDBC Connection Pool causes JDBC Connection Exhaustion if the VerifyProcess is executed concurrently; when a user has a max Tomcat JDBC Connection Pool size configured or defined as less than the numbers of the models in some of the VerifyProcess Suite, the Tomcat JDBC Connection Pool exhaustion might happen once the VerifyProcess is executed concurrently.

This could happen during the upgrade from earlier Liferay Portal version to Liferay Portal 6.2, or if user has the verify.frequency= property set with "-1" as its value.

Below are the ones from the Verify Process Suite that has the models defined:

1. VerifyAuditedModel
2. VerifyGroupId
3. VerifyResourcePermissions
4. VerifyUUID

Cause

In com.liferay.portal.verify.VerifyProcess, there is a method doVerify() with code snippet as follow:

protected void doVerify(
            Collection
                throwableAwareRunnables)
        throws Exception {

        if (throwableAwareRunnables.size() <
                PropsValues.VERIFY_PROCESS_CONCURRENCY_THRESHOLD) {

            for (ThrowableAwareRunnable throwableAwareRunnable :
                    throwableAwareRunnables) {

                throwableAwareRunnable.run();
            }
        }
        else {
            ExecutorService executorService = Executors.newFixedThreadPool(
                throwableAwareRunnables.size());

When the required amount of models from the verify suite exceed the threshold (defined by verify.process.concurrency.threshold property), based on below code from above snippet, the verify process will try to run concurrently.

else {

            ExecutorService executorService = Executors.newFixedThreadPool(

                throwableAwareRunnables.size());

This process then will try to get the amount of JDBC connection threads based on the models of that verify suite to accomondate the concurrency in order to complete the verifying of that suite. Usually if JDBC connection threads pool is at default value or has relatively high value (ex. a typical Production level) it will not be a problem. However if customer has a small max JDBC connection pool (e.g. 15 max JDBC connection threads), once the models amount exceeds the max JDBC connection pool max size, it will cause the verification process to hang and portal will be unresponsive.

Resolution

To resolve this issue, there are two options:

1. Increase the Tomcat JDBC Connection Pool max value to a more appropriate value to accommodate the concurrent verifying threads.
2. In the portal-ext.properties file, add the property verify.process.concurrency.threshold= and set the value higher than the numbers of model in some of the VerifyProcess suite to have the verify process run with single thread.