Issue
- The server is down with the below exception java.sql.SQLRecoverableException. What will be the possible root cause which triggers this exception?
2021-10-12 10:25:54,807 SEVERE [Common.CommonUtils] (default task-48) null: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:673)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:445)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:464)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:594)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:229)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
... 48 more
Environment
- Liferay DXP 7.0, Database, Third-party tools
Resolution
- The above snippet of the stack trace seems to be caused when the database listener is not listening to be connected with the Liferay.
-
Let me explain the example scenario with the higher possibility of this case,
- If this error may be thrown when one of the nodes is down, probably the reason why Liferay is failing after the failover of the database is that Liferay builds up a connection pool towards the database server and maintains it while the system is up and running.
- If the database cluster works in the background as a single robust database server, it should not cause any issues.
- Although if the connections are not replicated, when one of the nodes goes away, the connections from the connection pool get unusable and in the logs, you can see that they are closed.
- This means that the system remains without a working database connection and there is nothing Liferay can do about it without a restart.
- Furthermore, that in case the DB services have been restarted, Liferay needs to be restarted to re-establish the DB connections as the application server needs to enable locks on certain entries which get released at the time of DB service restart.
java.sql.SQLRecoverableException: Closed Connection
-
There might be different possibilities like below for the same:
1) Database Services are not running or not active or not having sufficienct connection at that time
2) Network Outage
3) May be any other components [firewall, network devices, Vm, load balancer] involved which might force the network connections to close after a period of the inactivity and hence forth.
Additional Information
- To cross verify it, please connect with your network expert, the DBA team to make sure the connections are active, having enough connections by monitoring it to see if there is any problem with the database when you have observed the unresponsiveness. This helps you to see what would be the right parameters or configuration that need to be set based on your environment architecture. please note that the Support Team is limited in providing specific guidance due to our support policy on Architectural Assistance.
-
Unofficial threads for a similar case that may give insight to you. Please choose to follow it at your sole discretion as these are not documented by Liferay: