Question
Estes Express Lines Inc
IN
Last activity: 21 Jan 2020 14:05 EST
Timeout on the monitor Service package requestor pool
We are getting the below timeout exception in the logs in the prga rules log.
2020-01-16 14:57:10,544 [ default task-10] [ STANDARD] [ ] [ CityDisp:02.01.01] (ProcessItems._baseclass.Action) ERROR pegaprdappr13.estes.us.dom|127.0.0.1|Rest|offlinehttp|DataSync|pzsynchronizationservice|A6QTZOYVESRJX9VQLFCYSM3TO40UZX3W5 - Exception while committing the page to databasecom.pega.pegarules.pub.database.LockGoneException: Database-LockFailure-LockLost ESTES-OPS-CITYDISP-WORK S-11223096
From: (unknown)
2020-01-16 14:58:46,582 [ PegaRULES-Batch-1] [ STANDARD] [ ] [ PegaAESRemote:07.31] ( internal.mgmt.Executable) ERROR - Execution denied for class: PegaAESRemote-Interface-ElasticSearchStatus (activity: PushElasticSearchStatus)
2020-01-16 15:09:09,041 [ default task-12] [ STANDARD] [ ] [ ] ( internal.mgmt.RequestorPool) ERROR pegaprdappr13.estes.us.dom|127.0.0.1 - Timed out borrowing service requestor from requestor pool for service package: monitor
2020-01-16 15:09:09,042 [ default task-12] [ STANDARD] [ ] [ ] ( services.http.HTTPService) ERROR pegaprdappr13.estes.us.dom|127.0.0.1 - Resource not found
com.pega.pegarules.pub.PRException: Timed out borrowing service requestor from requestor pool for service package: monitor
at com.pega.pegarules.session.internal.mgmt.RequestorPool.borrowRequestor(RequestorPool.java:492) ~[prprivate.jar:?]
at com.pega.pegarules.session.internal.mgmt.RequestorPoolManager.borrowRequestor(RequestorPoolManager.java:139) ~[prprivate.jar:?]
We are getting the below timeout exception in the logs in the prga rules log.
2020-01-16 14:57:10,544 [ default task-10] [ STANDARD] [ ] [ CityDisp:02.01.01] (ProcessItems._baseclass.Action) ERROR pegaprdappr13.estes.us.dom|127.0.0.1|Rest|offlinehttp|DataSync|pzsynchronizationservice|A6QTZOYVESRJX9VQLFCYSM3TO40UZX3W5 - Exception while committing the page to databasecom.pega.pegarules.pub.database.LockGoneException: Database-LockFailure-LockLost ESTES-OPS-CITYDISP-WORK S-11223096
From: (unknown)
2020-01-16 14:58:46,582 [ PegaRULES-Batch-1] [ STANDARD] [ ] [ PegaAESRemote:07.31] ( internal.mgmt.Executable) ERROR - Execution denied for class: PegaAESRemote-Interface-ElasticSearchStatus (activity: PushElasticSearchStatus)
2020-01-16 15:09:09,041 [ default task-12] [ STANDARD] [ ] [ ] ( internal.mgmt.RequestorPool) ERROR pegaprdappr13.estes.us.dom|127.0.0.1 - Timed out borrowing service requestor from requestor pool for service package: monitor
2020-01-16 15:09:09,042 [ default task-12] [ STANDARD] [ ] [ ] ( services.http.HTTPService) ERROR pegaprdappr13.estes.us.dom|127.0.0.1 - Resource not found
com.pega.pegarules.pub.PRException: Timed out borrowing service requestor from requestor pool for service package: monitor
at com.pega.pegarules.session.internal.mgmt.RequestorPool.borrowRequestor(RequestorPool.java:492) ~[prprivate.jar:?]
at com.pega.pegarules.session.internal.mgmt.RequestorPoolManager.borrowRequestor(RequestorPoolManager.java:139) ~[prprivate.jar:?]
at com.pega.pegarules.integration.engine.internal.services.http.HTTPService.invoke(HTTPService.java:270) ~[printegrint.jar:?]
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:331) ~[prprivate.jar:?]
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:274) ~[prprivate.jar:?]
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at com.pega.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:751) ~[prcommons-pr54-1.0.4.jar:?]
at com.pega.pegarules.session.internal.mgmt.RequestorPool.borrowRequestor(RequestorPool.java:479) ~[prprivate.jar:?]
Attaching the requestor pool configuration screen print.
***Edited by Moderator Marissa to update platform capability tags****
Pegasystems Inc.
US
As per the screenshot the requestor pool config on monitor service package, is reaching the configured Most Active value (50). The above error is expected when the requestor pool is full for the incoming requests and couldn't allocate a new thread within the configured amount of time (in sec).
Consider opening the Monitor service package requestor pool tab and provide higher values based on the usage.
Estes
US
Is there a way to identify why the requestor pool size has exceeded the defined MaxActive value? for example, if the Maximum active count is 50, then 50 concurrent requestors is still a pretty large number.
Is there a way to get the insight about the which services are hogging the requestors?
Pegasystems Inc.
US
Hi Anand,
It depends on the Service REST/SOAP performances which the service package is holding.
In the above scenario, it is monitor service package holds only one Service REST method. Look for PEGA0011 alerts if the service REST is underperforming. If yes, try to fine-tune it based on the PAL data.
Eg:- Say If Service REST takes long time to execute, then each thread from the requestor pool will be still busy and if there are new incoming requests for any service in the same service package (requestor pool), then it needs to wait until the pool gets freed up.
Pretty much, start looking at how many potential incoming requests you can get to this Service every min and how long each request takes to complete. Some clients just wait for a certain seconds and send retry requests if any to Service REST for a response.
Hope this helps