Pega0019 alerts on production , while using Mashup
Hi,
We are observing a high count of Pega 0019 critical alert ( long requestor running) in the production environment. We are using Pega 7.2.2 and have embedded the customer 360 inside the web page as a mashup. On viewing the alert logs we see that the issue is caused by the following 2 activities:
pzGetURLHashes and UpdateOperatorID. The actual step is in the updateoperatorid but is called from the pzGetURLHashes. I would like to know what causes this activity to fire in the first place ? Why is it consuming so much time? We can also see LASTINPUT as "existing Requestor retrieved". Here is the snapshot of the alert for one user.
| GENERATEDDATETIME | VERSION | CATEGORY | MSGID | KPIVALUE | KPITHRESHOLD | KPIOVERTHRESHOLD | SEVERITY | SERVERID | REQUESTORID | USERID | LOGGERNAME | STACK | LASTINPUT | FIRSTACTIVITY | LASTSTEP |
| 2018-08-31 08:50:54.417 | 8 | Long Requestor Time | PEGA0019 | 695235 | 600000 | 95235 | CRITICAL | 6ea4116b336bd3006b570728845552c5 | HAC78D26D4C767E7255449C58F6886C77 | pXYZ | com.pega.pegarules.monitor.internal.context.ManagementDaemonImpl | NA | existing requestor retrieved | Rule-Obj-Activity:pzGetURLHashes | DATA-ADMIN-OPERATOR-ID UPDATEOPERATORID #20131010T173711.855 GMT Step: 1 Circum: 0 |
| 2018-08-31 09:00:49.925 | 8 | Long Requestor Time | PEGA0019 | 1290745 | 600000 | 690745 | CRITICAL | 6ea4116b336bd3006b570728845552c5 | HAC78D26D4C767E7255449C58F6886C77 | pXYZ | com.pega.pegarules.monitor.internal.context.ManagementDaemonImpl | NA | existing requestor retrieved | Rule-Obj-Activity:pzGetURLHashes | DATA-ADMIN-OPERATOR-ID UPDATEOPERATORID #20131010T173711.855 GMT Step: 1 Circum: 0 |
| 2018-08-31 09:10:44.996 | 8 | Long Requestor Time | PEGA0019 | 1885815 | 600000 | 1285815 | CRITICAL | 6ea4116b336bd3006b570728845552c5 | HAC78D26D4C767E7255449C58F6886C77 | pXYZ | com.pega.pegarules.monitor.internal.context.ManagementDaemonImpl | NA | existing requestor retrieved | Rule-Obj-Activity:pzGetURLHashes | DATA-ADMIN-OPERATOR-ID UPDATEOPERATORID #20131010T173711.855 GMT Step: 1 Circum: 0 |
We have a requestor timeout set to 15 mins, and have HA enabled in Production. we have requestor passivation persistance to Database. There are in total 8 nodes which are behind a load balancer (f5) which has a timeout of 1 hr.
Any direction where we should triage for this issue?
***Edited by Moderator Marissa to update SR Details***
Hi Domenico,
Thanks for your response. I have the Alert logs which I have added the snapshot from, and while looking at it in more details around the time frames above, I do see Pega0001 and Pega0029 alerts. AS I mentioned we are using Mashup ( customer interaction portal) in a webpage and it loads the customer 360 view with each search interaction. But these pega 0001 and Pega0029 entries are not for the same user I provided above, but they are on the same server. On the Pega logs side, I can see thread dumps. Unfortunately I cannot share the complete log file due to compliance issue, I have attached a snapshot of the logs and alert file during specific timeframe mentioned above.
There looks to be some hazelcast exceptions in the thread dump.
Hi AnoojitD,
Thanks for your reply.
Unfortunately the screenshot attached doesn't provide enough details to determine the issue root cause.
I appreciate that sharing the full logs publicly is not viable in your case. As such I would suggest you to raise a Support Request on My Support Portal (https://msp.pega.com/msp/MSPServlet) providing the following elements:
A clear description of the issue (Context, Application Ruleset versions, ...)
Environment Details (Operating System, Application Server, JVM ans DB details)
PegaRULES and PegaRULES-ALERT for the period when the issue was reproduced.
Note: Please mention this Community Question while raising the request.
Thank you,
Domenico
Hi AnoojitD,
Please let us know the SR# once created. We will tag it with this discussion and update once the resolution is achieved.
Regards,
Upon reviewing the associated SR, I see that HFIX-36906 was provided.
Did this resolve for you?
Can you please provide the PegaRULES and PegaRULES-ALERT logs for the same period?
We need to identify the interactions that are causing the PEGA0019 and identify where the time is being spent.
Unless there is a batch requestor you should expect a PEGA0001 or a PEGA0011 for respectively, browser or application requestors, at every interaction end.
Analyzing the sequence of alerts associated to each interaction should help to identify the issue root cause.
Usually database bottlenecks or remote service calls are responsible for similar behaviors but it's hard to tell without reviewing the logs.