We have designed a Queue Processor to do a large scale data migration from a CMIS IBM Filenet server to AWS S3. Although the process is working just fine, we have a performance issue as we need to move around 8 TB of data in Production. We have a staging environment which has close to 2 TB data, and the end to end process is taking over 20 hours, this won't be acceptable for Production. We looked at the various performance enhancement suggestions on Queue Processor like here : https://community.pega.com/knowledgebase/articles/system-administration… and https://community.pega.com/knowledgebase/articles/decision-management-o… but the information seems to be conflicting each other and confusing, so here are our major questions:
1. Can we scale out to n no. of nodes to improve performance? right now we have 3 nodes where we are running the QP from, with 5 threads in each node. It seems the maximum partition size each QP can support is 20. If that's case, if i add 2 more nodes, then can i achieve ( 5 * 5) = 25 threads parallel performance or would it still do a max of 20 threads?
2. What is the difference between partition size and thread count? at one place there are used interchangeably in the articles above. Where at another place, it mentions partition size = no. of nodes * no. of thread per QP.
3. Can i increase the partition count beyond the default size of 20? So if i add more nodes should i have to keep increasing the partition size every time?
***Edited by Moderator Marissa to update Support Case Details***
We have a call scheduled with Pega to talk about the concerns of scaling and load balancing for Queue Processors on 09/23. Once this discussion has happened and we have received satisfactory explanations from Pega on this topic, i will close out this question.