Disable Kafka and DSM
Our 8.4 app logs keep on filling with messages like below.
Is Kafka, DSM necessary. We are not using these features...so how to disable them. So far I have decommissioned the node from Decision: Services > DDS, ADM, RTDB and Stream tabs.
prconfig/initservices/dnode/default is also false.
Anything else need to be done?
2020-04-24 22:10:15,591 [tionTask-Executor:42] [ STANDARD] [ ] [ ] (icepulse.RunInitializationTask) ERROR - Failed to prepare partitions for run [Aggregates_pxInteractionsStream] java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-19.0.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:142) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:106) ~[d-node.jar:?] at com.pega.dsm.dnode.api.dataflow.DataFlow.getDataPartitions(DataFlow.java:801) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator.createFor(PartitionCreator.java:80) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator.access$000(PartitionCreator.java:45) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator$1.run(PartitionCreator.java:62) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator$1.run(PartitionCreator.java:59) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper$2.run(ServiceHelper.java:286) ~[d-node.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1397) ~[prprivate-session.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1140) ~[prprivate-session.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:947) ~[prprivate-session.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:279) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:145) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator.createFor(PartitionCreator.java:59) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.RunInitializationTask.preparePartitions(RunInitializationTask.java:173) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.RunInitializationTask.asyncExecution(RunInitializationTask.java:123) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.AsyncAssignedTask$1.run(AsyncAssignedTask.java:69) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.AsyncAssignedTask$1.run(AsyncAssignedTask.java:58) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:296) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:145) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.AsyncAssignedTask.lambda$tryToExecuteInLock$0(AsyncAssignedTask.java:58) ~[d-node.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) ~[guava-19.0.jar:?] at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) ~[guava-19.0.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) ~[guava-19.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_151] at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124) ~[d-node.jar:?] Caused by: java.util.concurrent.ExecutionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:1.8.0_151] at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:1.8.0_151] at com.pega.dsm.dnode.impl.dataset.stream.StreamPartitioningFeature$1.emit(StreamPartitioningFeature.java:134) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:117) ~[d-node.jar:?] ... 31 more Caused by: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getBootstrapServers(KafkaSettingsProvider.java:157) ~[?:?] at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.defaultAdminClientProperties(KafkaSettingsProvider.java:93) ~[?:?] at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getDefaultAdminClientProperties(KafkaSettingsProvider.java:75) ~[?:?] at com.pega.fnx.stream.spi.impl.kafka.KafkaServerImpl.getAdminClient(KafkaServerImpl.java:60) ~[?:?] at com.pega.fnx.stream.spi.impl.processor.create.CreateStreamRequestProcessor.createTopic(CreateStreamRequestProcessor.java:64) ~[?:?] at com.pega.fnx.stream.spi.impl.processor.create.CreateStreamRequestProcessor.lambda$execute$0(CreateStreamRequestProcessor.java:56) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_151] at java.lang.Thread.run(Unknown Source) [?:1.8.0_151]
***Edited by Moderator Marissa to update Platform Capability tags****
Thanks Kevin.
Could you then please advise what these errors are and steps to resolve it. As of now our logs fill up very quickly because of them and our kafka-data folder is using 5.17gb alone...server team is raising questions over this.
Attached are logs and kakfa-data folder screenshot.
Hi Shakes,
We are also facing the same issue within our environment. Were you able to resolve this issue? Could you please let me know the steps to fix this?
This issue will come when you enable the DSM node for Dataflow/batch and Dataflow/realtime services. If you are not using decisioning services then you can make the below setting
1. Set below DSS value to false
interactionHistory/AllowPxInteractionsStreamAggregation
2. Open DataSet named as "Aggregation:pxInteractionsStream" and set the right Access group
3. Stop the Dataflow named as Aggregates_pxInteractionsStream
Is this fixed? I am also facing the same problem.
Sorry guys we have not been able or I should say not tried to disable Kafka. We just left it as is. But since then, have gathered some more info and how kafka is an integral part of the platform now from Pega perspective. Basically you need it at least for you Queue Processors to function. As a test you can start your cluster without a stream node and you'll see your QPs complaining and fillings logs with error that it needs a stream node and kafka. See below for a bit more info.
What happens when I queue messages with no stream node running?
When a stream node is not available and you queue a message, the system saves the message in a database and pushes it to Kafka when the stream node is running. The system does not process the messages in a queue until at least one stream mode is available.
Embedded Kafka (which is used by Pega stream node) is now a fundamental component in Pega platform, you will need that. At the minimal, specifying these node types in JVM argument: -DNodeType=Search,Stream,BackgroundProcessing,WebUser.