Rochwyn Rodrigues (RochwynR)
Maantic Inc
Rochwyn Rodrigues
Maantic Inc
RochwynR Member since 2014 5 posts
Maantic Inc
Posted: March 11, 2021
Last activity: March 12, 2021
Posted: 11 Mar 2021 16:37 EST
Last activity: 12 Mar 2021 3:24 EST

Queue message to KAFKA Partition ID

Use case: We are using queue processors to subscribe for CTI events. Each message does a long poll REST call with a time out of 30 secs. We have 17 agents, default KAFKA partition is 20 and 21 threads(7 threads for 3 background nodes). So partition to thread is 1:1. If an event is received in the 30 seconds the thread continues processing the event else it timeouts and requeues for another 30 seconds. This continues till the agent logs out of CTI.

Issue faced:

When using default round robin or message key based queuing we see partition to message is not 1:1. 

Example: If operator 1 and Operator 2 are queuing messages and are both allocated partition 1, now when operator 1 is long polling 30 secs, operator 2 message in partition 1 is not processed till 1 receives and event or timeouts(introducing a lag). So when a new CTI event occurs(new call/agent state change), that event gets delayed.

Possible workaround:

What we are looking for is having an internal mapping table where each operator will be assigned a partition. When we queue a message, we want to pass partition id(instead of round robin allocation or message key allocation) so that each operator message to partition is 1:1. Apache KAFKA supports a producer to queue to a particular partition but somehow we do not see that option exposed in pega. 

Any insight on this will be appreciated?


***Edited by Moderator: Pooja Gadige to add platform capability tags***
Pega Platform 8.4.4 Decision Management System Administration Conversational Channels Technology Services Lead System Architect