Siva Jyothi PamiReddy (Sivajyothi)

Sivajyothi Member since 2015 12 posts
Posted: March 24, 2020
Last activity: March 25, 2020
Posted: 24 Mar 2020 12:26 EDT
Last activity: 25 Mar 2020 18:10 EDT

Kafka Stream DataFlow consuming Old Data at a Particular time everyday

Hi ,

We are observing an unusual behavior of Kafka stream data set every day at particular time(between 23:00 - 00:00) where it is reading all the records of kafka topic again.

We are connecting to exterenalkafka system and the Kafka configirations are done as below at Pega:

-  configured Kafka Instance to connect to external Kafka server

- configured Kafka stream data set 

- configured Real time data flow with Kafka stream data set as source and with 'only read new records' option

Example scenario: 

- Kafka Topic got 100 records in a particular day and pega reads them as and when they are available in Kafka topic.

- At end of the day(between 23:00 - 00:00) , again reading all those 100 records.


Pega Version : 8.1.7

Server : WAS Liberty.

Please provide any inputs .Thanks in advance.




Pega Marketing 8.1 Enterprise Application Development Dev/Designer Studio Data Integration Decision Management Conversational Channels Other Industry
Moderation Team has archived post, This thread is closed to future replies. Content and links will no longer be updated. If you have the same/similar Question, please write a new Question.