The request buffer for thread pool "WorkManager.PRPCWorkManager" has reached its capacity.
We are getting below error in SystemOut.log.
[10/20/15 18:31:26:293 CEST] 0000002e ThreadPool I WSVR0629I: The request buffer for thread pool "WorkManager.PRPCWorkManager" has reached its capacity.
After few hours when there was no activity happening we get below message in logs.
[10/20/15 17:36:14:199 CEST] 0000002f ThreadMonitor W WSVR0606W: Thread "WorkManager.PRPCWorkManager : 0" (0000002f) was previously reported to be hung but has completed. It was active for approximately 7607430 milliseconds. There is/are 10 thread(s) in total in the server that still may be hung.
Our application is heavily agent based. We are calling multiple JMS based services and the listener hands over the response to an agent for processing. We are observing that for few agents the 'Last Run Finish' time is never updated. After deleting all agent entries from pr_Sys_Queues table it works fine for some time but it again start giving the error.
From SMA we can see that the agents are using the Thread "WorkManager.PRPCWorkManager. In WAS the poolsize in 5. We have more than 20 agents in application. This configuration works fine in DEV and ST environment. But in one environment it is giving above issue.
Increasing the WAS Thread pool size for WorkManager to 15 solved our issue. all agents use this Thread pool. With increasing Thread pool size the memory consumption increases, so count should not be too high otherwise out of memory error comes and application crashes.