Question1Replies41Views×Close popoverSiva Jyothi PamiReddy (Sivajyothi) HSBC Software Solution HSBC Software Solution IN View Profile Sivajyothi Member since 2015 11 posts HSBC Software Solution Posted: March 24, 2020Last activity: March 25, 2020Posted: 24 Mar 2020 12:26 EDTLast activity: 25 Mar 2020 18:10 EDT Kafka Stream DataFlow consuming Old Data at a Particular time everydayHi ,We are observing an unusual behavior of Kafka stream data set every day at particular time(between 23:00 - 00:00) where it is reading all the records of kafka topic again.We are connecting to exterenalkafka system and the Kafka configirations are done as below at Pega:- configured Kafka Instance to connect to external Kafka server- configured Kafka stream data set - configured Real time data flow with Kafka stream data set as source and with 'only read new records' optionExample scenario: - Kafka Topic got 100 records in a particular day and pega reads them as and when they are available in Kafka topic.- At end of the day(between 23:00 - 00:00) , again reading all those 100 records. Pega Version : 8.1.7Server : WAS Liberty.Please provide any inputs .Thanks in advance. Pega Marketing 8.1 Low-Code App Development Dev/Designer Studio Data Integration Decision Management Conversational Channels Other Industry ×Close popoverFacebookTwitterLinkedinEmail Copy Link Copied! Posted: 9 months agoPosted: 25 Mar 2020 18:10 EDT×Close popoverKevin Zheng (KevinZheng_GCS) PEGA Director, Software Solutions Engineering Pegasystems Inc. US View ProfileKevinZheng_GCS PEGA replied to SivajyothiThis is very interesting. How did you know that the same 100 records were consumed again at 23:00? By turning on org.apache.kafka log? Can you attach the logs if possible?