Queued items to Queue Processors are not stored in Pega DB Queue table. Items information is directly pushed to Kafka and while it is processing it will retrieve the information from Kafka message Queue.
The maximum number of Kafka partitions is 20 for one Pega Platform cluster, which means that the maximum number of threads that can process messages across all nodes in a cluster is 20.
I assume you are worried about performance when those tables are getting big. I checked the indices on pr_sys_msg_qp_brokenitmes, they are there including the column pzqueueprocessorname (which has an associated queue class). In theory we should be fine. Now the question is: are you already experiencing latency in your environments or is this a hypothetical question?
It has many dedicated queues(3 different queues, with 5 queue processes for each, we evenly distributed among 5), currently in production (there are about 300-500 ready to process queues, any given point of time on working days).
We did not find any issue as they get process, very quickly. (few seconds delay is fine for the application, near real time)
Now we are planning to upgrade to 8.3. After upgrade, will all the message of 3 different dedicated queues store in same Kofka Queue??
Will it give any performance impact?? or do we need to do some changes to create Kofka dedicated Queues ??
Above are the questions which flashed into our minds when we learned about new Queue Processer implementation in 8.3.