Question 1 Replies 63 Views × Close popover Siva Jyothi PamiReddy (Sivajyothi) DHL DHL IN View Profile Send Message Sivajyothi Member since 2015 12 posts DHL Posted: March 24, 2020 Last activity: March 25, 2020 Posted: 24 Mar 2020 12:26 EDT Last activity: 25 Mar 2020 18:10 EDT Closed Kafka Stream DataFlow consuming Old Data at a Particular time everyday Hi , We are observing an unusual behavior of Kafka stream data set every day at particular time(between 23:00 - 00:00) where it is reading all the records of kafka topic again. We are connecting to exterenalkafka system and the Kafka configirations are done as below at Pega: - configured Kafka Instance to connect to external Kafka server - configured Kafka stream data set - configured Real time data flow with Kafka stream data set as source and with 'only read new records' option Example scenario: - Kafka Topic got 100 records in a particular day and pega reads them as and when they are available in Kafka topic. - At end of the day(between 23:00 - 00:00) , again reading all those 100 records. Pega Version : 8.1.7 Server : WAS Liberty. Please provide any inputs .Thanks in advance. Pega Marketing 8.1 Enterprise Application Development Dev/Designer Studio Data Integration Decision Management Conversational Channels Other Industry Share × Close popover Facebook Twitter Linkedin Email Copy Link Copied! Moderation Team has archived post, learn more × Close popover This thread is closed to future replies. Content and links will no longer be updated. If you have the same/similar Question, please write a new Question. Posted: 1 year ago Posted: 25 Mar 2020 18:10 EDT × Close popover Kevin Zheng (KevinZheng_GCS) PEGA Director, Software Solutions Engineering Pegasystems Inc. US View Profile Send Message KevinZheng_GCS PEGA replied to Sivajyothi This is very interesting. How did you know that the same 100 records were consumed again at 23:00? By turning on org.apache.kafka log? Can you attach the logs if possible?