Question
Plexus Tech
ES
Last activity: 29 Aug 2019 7:07 EDT
Partition keys in Stream Data Set rules
Hi,
I created a Data Set to get messages from a kafka queue that has 2 partitions.
How do I assign value to the property specified as Partition Key?
Attached captures with the Data Set definition.
Thanks.
Pegasystems Inc.
IN
Hi Miguel Calderon,
What is the pega version that you are using?
From pega 7.4 ,by default number of partitions for kafka and stream dataset are 20(no configuration is needed). If you want to change this default number of partitions, go to kafka folder (in application server folder) and open connect-distributed.properties file and change the value of num.partitions.
By configuring partitioning you can ensure that related records are sent to the same partition. If no partition keys are set, the Kafka data set randomly assigns records to partitions.
Please go through below link for more information related to kafka data set creation.
Hi Miguel Calderon,
What is the pega version that you are using?
From pega 7.4 ,by default number of partitions for kafka and stream dataset are 20(no configuration is needed). If you want to change this default number of partitions, go to kafka folder (in application server folder) and open connect-distributed.properties file and change the value of num.partitions.
By configuring partitioning you can ensure that related records are sent to the same partition. If no partition keys are set, the Kafka data set randomly assigns records to partitions.
Please go through below link for more information related to kafka data set creation.
Generally to set the value of the property which is used as partition key for the dataset, you need to send the value in request itself.
Let's say i am using Age property as my partition key for stream data set, I need to send Age property value along with the required properties while sending the request to stream data set.
Hope it helps.Let me know if you have any queries.
Plexus Tech
ES