Abir Mondal (AbirM288)
Maximus Federal
Application Architect
Maximus Federal
AbirM288 Member since 2019 4 posts
Maximus Federal
Posted: September 15, 2020
Last activity: May 11, 2021
Posted: 15 Sep 2020 14:36 EDT
Last activity: 11 May 2021 12:29 EDT

Queue Processor : How to improve performance?

We have designed a Queue Processor to do a large scale data migration from a CMIS IBM Filenet server to AWS S3. Although the process is working just fine, we have a performance issue as we need to move around 8 TB of data in Production. We have a staging environment which has close to 2 TB data, and the end to end process is taking over 20 hours, this won't be acceptable for Production. We looked at the various performance enhancement suggestions on Queue Processor like here : and but the information seems to be conflicting each other and confusing, so here are our major questions: 1. Can we scale out to n no. of nodes to improve performance? right now we have 3 nodes where we are running the QP from, with 5 threads in each node. It seems the maximum partition size each QP can support is 20. If that's case, if i add 2 more nodes, then can i achieve ( 5 * 5) = 25 threads parallel performance or would it still do a max of 20 threads? 2. What is the difference between partition size and thread count? at one place there are used interchangeably in the articles above. Where at another place, it mentions partition size = no. of nodes * no. of thread per QP. 3. Can i increase the partition count beyond the default size of 20? So if i add more nodes should i have to keep increasing the partition size every time?

***Edited by Moderator Marissa to update Support Case Details***
Pega Platform 8.3.1 Cloud Services Client-managed Cloud Government Lead System Architect Support Case Exists