Pega Kafka integration - messages are lost when dataflow is not running
Hi,
Currently our application is on Pega 7.3.1. We are trying to subscribe to a kafka topic from our application. I have created a kafka dataset and dataflow to receive messages. However, I see some issues out there. It works fine when dataflow is running and a new message is published to a topic. However, consider dataflow is stopped and a new message is published to the topic, Now when we restart the dataflow the message is not received. This causes message loss.
I have tried the option read from beginning but no luck.
Please help!!
Posted: 4 months ago
From 7.4 onwards the Data Flow has a setting "Read existing and new records" which ensure that you read records added to the topic when data flow is not running
https://community.pega.com/knowledgebase/articles/decision-management/84/creating-data-flow
For 7.3.1,it was a part of the Kafka Dataset rule itself. The "Read from Beginning" in the Kafka Dataset rule should ensure that you process all the records before the data flow run was started. If its not working as expected, I suggest you create a support request for the same.