Posted: 5 Apr 2017 3:33 EDT Last activity: 6 Jun 2017 11:21 EDT
Timed out borrowing service requestor from requestor pool for service package: monitor
We have multiple nodes running and the requests to the application are made using IAC. In one node ,In SMa the requestor count is growing high.Stale requestors are still present in System Management Application (SMA). In other node the below exception is comming in the logs and the system is non responsive.
Logs are getting filled with the below message and the system is becomming non-responsive.
2017-04-05 16:59:31,226 [ http-nio-80-exec-59] [ ] [ ] [ ] ( internal.mgmt.RequestorPool) ERROR username|ipaddress - Timed out borrowing service requestor from requestor pool for service package: monitor
Could any one please assist.
***Updated by moderator: Lochan to remove proprietary information***
1) Time out borrowing service requestor error ( the above link suggests to maximize the max requestors count to 100 . does it mean i need to add DATA-ADMIN-SERVICEPACKAGE MONITOR service package max requestors to 100 ? does this configuration need to be done manually in all environments ?
2) The second issue is the requestors are not getting killed and the requestors count is high when viewd in sma.
For the first issue check the number of requests for the service package from the SMA and if it is more than 100, you will have to update the requestor pool settings manually on all the nodes (please keep a note that by increasing the requestor pool setting will have an impact on the memory footprint)
For the second issue, could you please share the Pegarules logs and application server log files to verify the pattern and to make sure that passivation daemon is getting triggered or not.
Appologies for late reply. do to some other project deadlines i did not got chance to look in to this issue. Below is the scenaios i am facing the issue
I am facing this issue when the sever is restarting or some time when more requestors are ideal and stil the requestor is not killed.
on The monitor Service package i am seeing the default values as
Maximum idle requestors :10
Maximum active requestors :10
Maximum wait :10
Do you mean can i increase the Max active requestors to 100 or below 100 ? Also i was curious to know why was i am getting the "Timed out borrowing service requestor from requestor pool" error when i am restarting the server. Does all the requestors will not be killed when we restart the server ?
In regards to the requestors not being killed when you restart the server, you may still have requests coming into your service even while you are restarting the server. If that is the case, you may be experiencing the same scenario where you fill up the 10 max active requestors and then the 11th and subsequent requests start timing out as a requestor doesn't become available within the 10 second timeout. Again, you need to understand the load of the requests coming in to set the pool appropriately.
And this is a little abstract / high-level: but it might provide you with some ideas on how large to make your Requestor Pools (you will need an idea of what your expected throughput is for this) : https://en.wikipedia.org/wiki/Queueing_theory
Each node can have 10 active requestors at a time. If you have 3 nodes and you are configured to 10 max active requestors in your pool, spread across the 3 nodes you can handle 30 active requestors (10 for each node).