Pega 7.1.9 Processing many records over multiple nodes
Good Evening, We have a 10 node setup and need to allow our xml parsing to occur for a million records. Because this occurs in sequence this obviously limits the speed in which we can accomplish this task. I am trying to find out if we can setup up a Master Agent which would control the flow to child agents where we could have multiple child agents per node. Ideally lets say the Master Agent picks up 100,000 (1/10) of these records. Could the Master agent distribute that load to 5 child agents on node 1 through node 10 for a total of 50 child agent getting 2000 records a piece. Once the child agents on Node 1 as an example finishes processing the 10,000 records across its 5 child nodes could the Master Agent have ready the next 100,000 and give that node its next 10,000 (2000 per child) to process? Apologies for my ignorance but I am trying to understand the capabilities better here. Thank you very much!
You are on right track here. You can make the master agent as an advanced agent, this just queues the records for child agents to pick them. All the child agents would be standard agents which process the relevant queue records present in DB. This way you can configure the master agent to only queue a particular number of records based on the number of child agents that you have without creating a bottleneck on the child agents.
The interval times of the master agent and child agents would be the key in this. If the master agents start too fast the child agents can't handle the load, it would cause a bottleneck. If the master agents start too slow the child agents might be idle and you'll be wasting the resources.