Potential Load Balancing Issue with Incoming Requests
We have been facing a sudden Requester Spike issue in Pega application where one JVM will be affected with huge number of requesters in a short time period and CPU spike high in production.
We are thinking it is issue with load balancer distribution across all JVM's as we have many pega applications running across all 4 nodes with load balancer.
1.Is there any Configuration/setting that can be checked in Load Balancer side or Web Server side for more information?
2.Is there anything that we configure in Pega for requestors passing through load balancer and to Webserver JVMS?
3.Security Integration in the web container is a default setting as it pretty much guarantees that a user HTTP Session is mapped to the correct user when accessing secured pages to protect the session from unauthorized access.
What is relation between the “Requestor’s count and a user HTTP session because we are using that as the meter to compare the load distribution.
Please help us on solving this issue.Thank you.
***Edited by Moderator Marissa to update General to Product***
Can you share the details of the surging requestors? I would go to Admin Studio first. Are these browser requestors or something else? Also if you can share the topology of your env, that would help as well. I assume your production env has been running for a while. If that is the case, are there any recent changes in the environment?
Its Browser requestors and you see below from log message requestor count increasing from 132 to 526 and to 1042 within 10minutes and this requestor count is spiking suddenly on 1 JVM and CPU spikes high due to this requestor count.
On any regular day, the requestors count is 100-150 requestors.
Server details: We have 4 web servers and 4 application servers in cluster environment.
Each 4 app servers(node 1,2,3,4) has 3 JVMs each(node 1-1,node 1-2,node 1-3...) in cluster env.
2020-06-02 09:14:14,780 [.PRPCWorkManager : 3] [ STANDARD] [ ] [ PegaAPPli:02.01.02] ( internal.async.Agent) INFO - System date: Tue Jun 02 09:14:14 EDT 2020 Total memory: 8,544,059,392 Free memory: 4,011,364,232 Requestor Count: 132 Shared Pages memory usage: 0% Current number of threads: 8 2020-06-02 09:24:14,783 [.PRPCWorkManager : 0] [ STANDARD] [ ] [ PegaAPPli:01.01.01] ( internal.async.Agent) INFO - System date: Tue Jun 02 09:24:14 EDT 2020 Total memory: 8,544,059,392 Free memory: 2,912,995,160 Requestor Count:526 Shared Pages memory usage: 0% Current number of threads: 13
Application is in production env from 4 years and Pega version is PRPC 7.4V