Hi,

We are facing a problem in application, where we sudden spike in database connection pool and it is reaching maximum which caused all the running agents go into exception. We observed this behavior even when there was the minimal load and just after 90 mints of web node restart. We checked with all our teams(web , database) but we dont see any issue.

Please see the below log information on the log.

<Sep 10, 2017 3:00:39 PM CEST> <Error> <HTTP> <BEA-101215> <Malformed request "null". Request parsing failed, Code: -1>
2017-09-10 15:00:57,141 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:58,087 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:58,250 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:58,250 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:58,279 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:58,613 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:59,177 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:59,725 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:00:59,987 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,122 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,122 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,122 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,415 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,416 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,422 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,475 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,475 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,476 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:00,615 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:01,206 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:01,207 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:01,799 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:02,662 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:02,705 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:02,816 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:03,127 [fault (self-tuning)'] [ STANDARD] [ ] [ ] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:05,191 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again
2017-09-10 15:01:06,974 [fault (self-tuning)'] [ STANDARD] [ ] [ OPERCMTAdmin:01.17] (.access.DatabaseConnectionImpl) ERROR - Couldn't
obtain a connection. Refresh the DataSource, and try again

2017-09-10 15:01:13,129 [15624.IO.thread-in-1] [ STANDARD] [ ] [ ] ( nio.tcp.TcpIpConnection) INFO - [xx.xx.xx.xx]:5701 [1ac07b8356846befc505fb795b115624] [3.4.1] Connection [xx.xx.xx.xx/xx.xx.xx.xx:43335] lost. Reason: java.io.EOFException[Remote socket closed!]
2017-09-10 15:01:13,717 [fault (self-tuning)'] [ STANDARD] [ ] [ ] ( internal.access.DatabaseImpl) ERROR - There wa
s a problem with the database when getting a list:
:
com.pega.pegarules.pub.database.ConnectionException: Database-General Problem encountered when getting connection for database pegarules 0
weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool PegaRULES to allocate to applications, please increase the size of the pool and retry..
DatabaseException caused by prior exception: weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool PegaRULES to allocate to applications, please increase the size of the pool and retry..
| SQL Code: 0 | SQL State: null

DatabaseException caused by prior exception: java.sql.SQLException: Cannot obtain connection: driverURL = jdbc:weblogic:pool:PegaRULES, props = {EmulateTwoPhaseCommit=false, connectionPoolID=PegaRULES, jdbcTxDataSource=true, LoggingLastResource=false, dataSourceName=PegaRULES}
| SQL Code: 0 | SQL State: null

From: (B7B4A7F2111AB6D90E25CC97915CCA9F7)
Caused by SQL Problems.
Problem #1, SQLState null, Error code 0: weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool PegaRULES to allocate to applications, please increase the size of the pool and retry..
Problem #2, SQLState null, Error code 0: java.sql.SQLException: Cannot obtain connection: driverURL = jdbc:weblogic:pool:PegaRULES, props = {EmulateTwoPhaseCommit=false, connectionPoolID=PegaRULES, jdbcTxDataSource=true, LoggingLastResource=false, dataSourceName=PegaRULES}
at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:256)
at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnectionFromDataSourceIdentity(ConnectionManagerImpl.java:1087)
at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:1216)
at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:1156)
at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:347)
at com.pega.pegarules.data.internal.access.ThreadConnectionStoreImpl.getConnection(ThreadConnectionStoreImpl.java:426)
at com.pega.pegarules.data.internal.access.RDBPageResultPackager.getConnection(RDBPageResultPackager.java:924)
at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:6279)
at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6932)
at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6885)
at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6865)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.doesTopAppHashNeedToBeInsertedIntoFlattenedHierarchy(AppCentricRulesetIndexWriter.java:562)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:501)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:613)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:634)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.writeApplicationInfoToDatabase(AppCentricRulesetIndexWriter.java:193)
at com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.updateDBRSLAndAppStack(ManagedApplicationContext.java:860)
at com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.updateToParent(ManagedApplicationContext.java:430)
at com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.getChangeCount(ManagedApplicationContext.java:390)
at com.pega.pegarules.session.internal.authorization.context.AbstractApplicationContext.getAppInfo(AbstractApplicationContext.java:45)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:629)
at com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.writeApplicationInfoToDatabase(AppCentricRulesetIndexWriter.java:193)
at com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.updateDBRSLAndAppStack(ManagedApplicationContext.java:860)
at com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.<init>(ManagedApplicationContext.java:107)
at com.pega.pegarules.session.internal.authorization.context.custom.CustomApplicationContext.<init>(CustomApplicationContext.java:75)
at com.pega.pegarules.session.internal.authorization.context.ContextUtils.ensureCustomizable(ContextUtils.java:867)
at com.pega.pegarules.session.internal.authorization.Authorization.alterRuleSetList(Authorization.java:1048)
at com.pega.pegarules.session.internal.authorization.Authorization.alterRuleSetList(Authorization.java:1041)
at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:839)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1266)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1015)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:908)
at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:684)
at com.pega.pegarules.session.internal.async.Agent.queueBatchActivityForAgent(Agent.java:1526)
at com.pega.pegarules.session.internal.async.Agent.queueBatchActivityForAgent(Agent.java:1299)

***Edited by Moderator Marissa to update categories***