Posted: 19 Aug 2019 6:52 EDT Last activity: 24 Mar 2020 10:05 EDT
Ask the Expert - Background Processing with Prithanka Chatterjee
Join Prithanka (@chatp) in this Ask the Expert session (26-30 August) on Background Processing.
About Prithanka: Prithanka is a Senior Product Manager with Pega’s Core Platform team, with specific focus on Core-Engine. Prithanka has over 16 years of experience in delivering best-in-class Platforms and Frameworks for distributed enterprise applications. His work in Pega includes ensuring the availability of highly resilient and scalable background processing capabilities.
Message from Prithanka: Hello all, I am passionate about enabling customers to get the best out of Pega platform, and that includes getting the best out of the background processing capabilities built into the platform. In this segment of our interaction, I would love to answer your questions regarding Background Processing in general and more specifically about Standard Agents, Advanced Agents, Queue Processors, and Job Schedulers.
Kafka is a stream-processing software platform which is used for building real-time data pipelines and streaming applications. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. Among a host of other benefits, I would primarily like to draw your attention towards the following few. Kafka is highly recommended as a streaming solution because:
It provides high throughput
It has very low latency
It is fault tolerant
It supports both vertical and horizontal scalability
It is distributed in nature
It supports both streaming and messaging capabilities
It provides high concurrency
It provides high durability
Kafka is highly configurable and has been configured in a way to best serve the Pega deployment. I would encourage you to work with the default settings as currently available, which have been tested to perform the best. In case you need any support for an advanced setting, I would request you to reach out to support.
The current defaults, as a few example, are as follows:
We want to build a rest service that can handle hundreds, if not thousands of requests for every few seconds. From pdn, it is my understanding that this can be done using asynchronous processing (and Service Request Processor). Could you please shed some light on this and if there are any downsides to this approach. Can we associate the Queue ID with the workobject using OOTB rules?
Background processing using Advanced/Standard agents or Queue Processors and Job Schedulers do not support the kind of functionality you need out of the box. However, you can choose to build it if required. But since you are already looking to use Service Request Processor for building the rest service, let me help you and tag the Expert for the same.
@nvkap : Request you to kindly answer this question.
Is there a configuration we can set or Rule to use so we can make sure the Queue Processors are automatically restarted every time PEGA crashes or is also restarted?. Specifically we're interested in automatically restarting the following queue processors:
Reason for this is that after the last restart we had (one of the nodes went down), we noticed the queue processors did not automatically started and had to manually be started.
Queue Processors are built to be resilient and survive crashes and restarts. As long as the Queue Processor is not Disabled on the Queue Processor Rule-Form or is not explicitly stopped from the Queue Processor Landing Page, in the Admin studio, the Queue Processor is supposed to start automatically on node crashes or generic restarts.
I request you to check for these two configurations, one on the rule-form and the other in the Landing-Page, to see that it is configured correctly. And if the problem persists, I would request you to log a support ticket.
I was trying to replace an existing advanced agent with Job Scheduler. I was able to successfully create the job scheduler rule and followed the configuration guidelines as per the rule help guide. I'm uploading the configuration snapshot.
Every Job Scheduler and Queue Processor in 8.2.x gets rule resolved against the context specified in ASYNCPROCESSOR Requestor Type. And unless a Job Scheduler is resolvable against the specified context, it will not show up in Admin Studio Landing Page. Please see the snippet below from Pega Help for more details:
AsyncProcessor requestor type
You use AsyncProcessor requestor type to resolve Job Scheduler and Queue Processor rules.
Each Pega Platform operator logs in using unique ID that is associated with a specific access group. This access group provides a context to resolve rules. Because Job Scheduler and Queue Processor rules in the background, no user-based context is available to resolve these rules. The AsyncProcessor requestor type provides the context to resolve Job Scheduler and Queue Processor rules.
Requestor type definition
The AsyncProcessor requestor type defines a list of rulesets that create a context for Job Scheduler and Queue Processor rules resolution.
At system startup, unique queue processors and job schedulers are found by using the context defined in the AsyncProcessor requestor type. When the system is running, and a new queue processor is added, or an existing one is overridden in a different ruleset, the context is updated to include the new ruleset to resolve the right rule.
The dafult access group is an Application-based Access group. This definition includes your custom access group that corresponds to your custom rulesets. When you use the Application-based access group, only job schedulers and queue processors that belong to this access group run. The default access group is PRPC:AsyncProcessor.
I request you to explore and assess the Access Group that is currently specified in the ASYNCPROCESSOR Requestor Type, and ensure that your Job Scheduler is resolvable against that.
Just to clarify the access group context in the Job scheduler or queue processor ruleform only determines the context for the activity rule mentioned in the rule but it doesn't have any significance for the rule to show up in the Admin studio.
For it show up in the Admin Studio we need to check the AsyncProcessor requestor type instance to include the application context for the job schedulers or queue processors created in the corresponding application context right?
Posted: 1 year ago
Updated: 1 year ago
Posted: 29 Aug 2019 9:12 EDT Updated: 29 Aug 2019 13:37 EDT
Firstly , thanks for the question and this is a really interesting one.
Agents always had a rule (Rule-Agent-Queue) and a data (Data-Agent-Queue) instance working together in combination to determine the current configuration. This, though was useful in some cases, it was also a cause for substantial pain for most of our users. For example:
This made it difficult to ascertain the state of a certain Agent just by looking at the rule.
Sometimes during migration or upgrades the data instances would be lost and that would cause a change in behavior.
This would also make Agents behave differently than other Pega rules.
So, in an effort to standardize the behavior, Job Schedulers are created to contain all configurations within the rule itself. Any change in the behavior of the rule has to be made and contained within the rule itself. Though this takes away a little bit of the flexibility, this offers better resilience and management capabilities. As for your particular case Mahi, I would advice you to resave the Job scheduler rule after you make changes (possibly in a production ruleset). And hopefully you are not required to make such changes frequently in Production systems (which again is not advisable).
If this does not solve your use case, please feel free to get in touch with me by sending me a private message here on Pega Community and we can discuss further, and explore your specific use case.