Question

5
Replies
294
Views
ollae Member since 2013 3 posts
PEGA
Posted: 4 months ago
Last activity: 4 months 2 weeks ago

Problem on Kafka partitions associated to pyFTSIncremental indexer

Hello,

I am new to Kafka and i am trying why the Search does not work.

I have a VM with Pega 8.4. The problem is not on the Queue Processor pyFTSIncrementalIndexer  is more on the data flow called pyFTSIncrementalIndexer that is accessible from the System > Search landing page and appears to be blocked and shows problems on Kafka partitions:

Problem on PyFTSIncremental Indexer data flow

Opening the details i see the following exception:

Caused by: com.pega.fnx.stream.spi.StreamServiceException: Unknown stream PYFTSINCREMENTALINDEXER
	at com.pega.fnx.stream.spi.impl.processor.StreamServiceExceptions.translateKafkaException(StreamServiceExceptions.java:56)
	at com.pega.fnx.stream.spi.impl.processor.StreamServiceExceptions.translate(StreamServiceExceptions.java:40)
	at com.pega.fnx.stream.spi.impl.processor.GetStreamPartitionsRequestProcessor.getDetails(GetStreamPartitionsRequestProcessor.java:98)
	at com.pega.fnx.stream.spi.impl.processor.GetStreamPartitionsRequestProcessor.lambda$execute$0(GetStreamPartitionsRequestProcessor.java:56)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

When i starting Pega there are no issues. When instead i am looking on the PegaDATAFLOW.log and filter by pyFTSIncrementalIndexer i see a sequence of messages like these: first pyFTSIncrementalIndexer try to pick the partition after the partition is no more available.

2020-04-23 18:13:35,896 (.DataFlowDiagnosticsFileLogger) INFO  - {"type":".RunStatusTransitionMessage","senderNodeId":"feeeef21337df7f3148f6c70310a8c96","timestamp":1587680015895,"messageId":"2f555b4e-a793-44ff-b78f-43c12d1ec2ef","runId":"pyFTSIncrementalIndexer","previousStatus":"PENDING_ASSIGNMENT","newStatus":"PENDING_PICKUP","originator":"AssignPartitionsTask","reason":"Assigned partitions for stream run"}
2020-04-23 18:13:59,906 (.DataFlowDiagnosticsFileLogger) INFO  - {"type":".RunStatusTransitionMessage","senderNodeId":"feeeef21337df7f3148f6c70310a8c96","timestamp":1587680039904,"messageId":"bdcc3c59-c995-4700-bc02-ead9026326e5","runId":"pyFTSIncrementalIndexer","previousStatus":"PENDING_PICKUP","newStatus":"PENDING_ASSIGNMENT","originator":"CheckAssignmentsTask","reason":"No partition assignments found for run"}
2020-04-23 18:14:07,892 (.DataFlowDiagnosticsFileLogger) INFO  - {"type":".RunStatusTransitionMessage","senderNodeId":"feeeef21337df7f3148f6c70310a8c96","timestamp":1587680047891,"messageId":"523f416e-0291-41db-acd6-0d947c06de4e","runId":"pyFTSIncrementalIndexer","previousStatus":"PENDING_ASSIGNMENT","newStatus":"PENDING_PICKUP","originator":"AssignPartitionsTask","reason":"Assigned partitions for stream run"}

I have tried to restart the Pega Server after decommission the Stream Node but nothing change. No errors appears on the logs except the following messages on repetition, like the QueueProcessor pyFTSIncrementalIndexer try to start but due of the problem on partitions it can't.

2020-04-23 17:20:28,120 [OBSCHEDULER_THREAD_4] [  STANDARD] [                    ] [                    ] ( QueueProcessorDataFlowHandler) ERROR   - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
2020-04-23 17:20:28,120 [OBSCHEDULER_THREAD_4] [  STANDARD] [                    ] [                    ] (ndexer.DistributedIndexerAgent) WARN    - Started Queue processor: pyFTSIncrementalIndexer
2020-04-23 17:29:48,679 [ttp-nio-8080-exec-10] [dminStudio] [                    ] [   GoGoRoad:01.01.01] ( QueueProcessorDataFlowHandler) ERROR localhost|127.0.0.1 admin@gogoroad - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
2020-04-23 17:31:13,081 [OBSCHEDULER_THREAD_4] [  STANDARD] [                    ] [                    ] ( QueueProcessorDataFlowHandler) ERROR   - Queue processor: pyFTSIncrementalIndexer couldn't come to a stable state. Status: Initializing
2020-04-23 17:31:13,082 [OBSCHEDULER_THREAD_4] [  STANDARD] [                    ] [                    ] (ndexer.DistributedIndexerAgent) WARN    - Started Queue processor: pyFTSIncrementalIndexer

How can i unblock the situation, and how to make the partition setup correct for pyFTSIncrementalIndexer ?

Thank you.

Regards,

Eliseo

***Edited by Moderator: Pallavi to change content type from Discussion to Question***  

Pega Platform 8.4 Senior System Architect
Share this page LinkedIn