Timed out borrowing service requestor from requestor pool for service package: monitor
We have multiple nodes running and the requests to the application are made using IAC. In one node ,In SMa the requestor count is growing high.Stale requestors are still present in System Management Application (SMA). In other node the below exception is comming in the logs and the system is non responsive.
Logs are getting filled with the below message and the system is becomming non-responsive.
2017-04-05 16:59:31,226 [ http-nio-80-exec-59] [ ] [ ] [ ] ( internal.mgmt.RequestorPool) ERROR username|ipaddress - Timed out borrowing service requestor from requestor pool for service package: monitor
Could any one please assist.
***Updated by moderator: Lochan to remove proprietary information***
1) Time out borrowing service requestor error ( the above link suggests to maximize the max requestors count to 100 . does it mean i need to add DATA-ADMIN-SERVICEPACKAGE MONITOR service package max requestors to 100 ? does this configuration need to be done manually in all environments ?
2) The second issue is the requestors are not getting killed and the requestors count is high when viewd in sma.
For the first issue check the number of requests for the service package from the SMA and if it is more than 100, you will have to update the requestor pool settings manually on all the nodes (please keep a note that by increasing the requestor pool setting will have an impact on the memory footprint)
For the second issue, could you please share the Pegarules logs and application server log files to verify the pattern and to make sure that passivation daemon is getting triggered or not.
Appologies for late reply. do to some other project deadlines i did not got chance to look in to this issue. Below is the scenaios i am facing the issue
I am facing this issue when the sever is restarting or some time when more requestors are ideal and stil the requestor is not killed.
on The monitor Service package i am seeing the default values as
Maximum idle requestors :10
Maximum active requestors :10
Maximum wait :10
Do you mean can i increase the Max active requestors to 100 or below 100 ? Also i was curious to know why was i am getting the "Timed out borrowing service requestor from requestor pool" error when i am restarting the server. Does all the requestors will not be killed when we restart the server ?
The maximum 10 requestors indicated the max number of requests that it can handle at any time. If you have an 11th request, it will wait until one of the first 10 requestors finishes or until it hits the 10 second maximum wait time. When you see the "Timed out borrowing service requestor from requestor pool", that means the request trying to get processed has been waiting the maximum wait time and has timed out. In regards to the 100 or below question, you need to determine how big your pool size should be based on the number of anticipated requests at any one time. Once you determine that, you will want to set you max requestors appropriately (for example: 50 or 100) keeping in mind the memory increased that was pointed out above. You will also need to factor in how long the requests take so that your timeout/maximum active requestors are set appropriately to accommodate the expected load. You can review more information at the following links: https://pdn.pega.com/sites/pdn.pega.com/files/help_v722/procomhelpmain… and https://pdn.pega.com/sites/pdn.pega.com/files/help_v722/procomhelpmain…
In regards to the requestors not being killed when you restart the server, you may still have requests coming into your service even while you are restarting the server. If that is the case, you may be experiencing the same scenario where you fill up the 10 max active requestors and then the 11th and subsequent requests start timing out as a requestor doesn't become available within the 10 second timeout. Again, you need to understand the load of the requests coming in to set the pool appropriately.
And this is a little abstract / high-level: but it might provide you with some ideas on how large to make your Requestor Pools (you will need an idea of what your expected throughput is for this) : https://en.wikipedia.org/wiki/Queueing_theory
Each node can have 10 active requestors at a time. If you have 3 nodes and you are configured to 10 max active requestors in your pool, spread across the 3 nodes you can handle 30 active requestors (10 for each node).