Close popover
Siva Jyothi PamiReddy (Sivajyothi)
HSBC Software Solution

HSBC Software Solution
Sivajyothi Member since 2015 11 posts
HSBC Software Solution
Posted: March 24, 2020
Last activity: March 25, 2020

Kafka Stream DataFlow consuming Old Data at a Particular time everyday

Hi ,

We are observing an unusual behavior of Kafka stream data set every day at particular time(between 23:00 - 00:00) where it is reading all the records of kafka topic again.

We are connecting to exterenalkafka system and the Kafka configirations are done as below at Pega:

-  configured Kafka Instance to connect to external Kafka server

- configured Kafka stream data set 

- configured Real time data flow with Kafka stream data set as source and with 'only read new records' option

Example scenario: 

- Kafka Topic got 100 records in a particular day and pega reads them as and when they are available in Kafka topic.

- At end of the day(between 23:00 - 00:00) , again reading all those 100 records.


Pega Version : 8.1.7

Server : WAS Liberty.

Please provide any inputs .Thanks in advance.




Pega Marketing 8.1 Low-Code App Development Dev/Designer Studio Data Integration Decision Management Conversational Channels Other Industry