Posted: 30 Jun 2020 5:18 EDT Last activity: 30 Jun 2020 10:57 EDT
Pega Kafka integration - messages are lost when dataflow is not running
Currently our application is on Pega 7.3.1. We are trying to subscribe to a kafka topic from our application. I have created a kafka dataset and dataflow to receive messages. However, I see some issues out there. It works fine when dataflow is running and a new message is published to a topic. However, consider dataflow is stopped and a new message is published to the topic, Now when we restart the dataflow the message is not received. This causes message loss.
I have tried the option read from beginning but no luck.
For 7.3.1,it was a part of the Kafka Dataset rule itself. The "Read from Beginning" in the Kafka Dataset rule should ensure that you process all the records before the data flow run was started. If its not working as expected, I suggest you create a support request for the same.