Disable Kafka and DSM
Our 8.4 app logs keep on filling with messages like below.
Is Kafka, DSM necessary. We are not using these features...so how to disable them. So far I have decommissioned the node from Decision: Services > DDS, ADM, RTDB and Stream tabs.
prconfig/initservices/dnode/default is also false.
Anything else need to be done?
2020-04-24 22:10:15,591 [tionTask-Executor:42] [ STANDARD] [ ] [ ] (icepulse.RunInitializationTask) ERROR - Failed to prepare partitions for run [Aggregates_pxInteractionsStream] java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-19.0.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:142) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:106) ~[d-node.jar:?] at com.pega.dsm.dnode.api.dataflow.DataFlow.getDataPartitions(DataFlow.java:801) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator.createFor(PartitionCreator.java:80) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator.access$000(PartitionCreator.java:45) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator$1.run(PartitionCreator.java:62) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator$1.run(PartitionCreator.java:59) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper$2.run(ServiceHelper.java:286) ~[d-node.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1397) ~[prprivate-session.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1140) ~[prprivate-session.jar:?] at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:947) ~[prprivate-session.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:279) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:145) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.PartitionCreator.createFor(PartitionCreator.java:59) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.RunInitializationTask.preparePartitions(RunInitializationTask.java:173) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.RunInitializationTask.asyncExecution(RunInitializationTask.java:123) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.AsyncAssignedTask$1.run(AsyncAssignedTask.java:69) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.AsyncAssignedTask$1.run(AsyncAssignedTask.java:58) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:296) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:145) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.dataflow.service.servicepulse.AsyncAssignedTask.lambda$tryToExecuteInLock$0(AsyncAssignedTask.java:58) ~[d-node.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) ~[guava-19.0.jar:?] at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) ~[guava-19.0.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) ~[guava-19.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_151] at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56) ~[d-node.jar:?] at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124) ~[d-node.jar:?] Caused by: java.util.concurrent.ExecutionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:1.8.0_151] at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:1.8.0_151] at com.pega.dsm.dnode.impl.dataset.stream.StreamPartitioningFeature$1.emit(StreamPartitioningFeature.java:134) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?] at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:117) ~[d-node.jar:?] ... 31 more Caused by: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point. at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getBootstrapServers(KafkaSettingsProvider.java:157) ~[?:?] at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.defaultAdminClientProperties(KafkaSettingsProvider.java:93) ~[?:?] at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getDefaultAdminClientProperties(KafkaSettingsProvider.java:75) ~[?:?] at com.pega.fnx.stream.spi.impl.kafka.KafkaServerImpl.getAdminClient(KafkaServerImpl.java:60) ~[?:?] at com.pega.fnx.stream.spi.impl.processor.create.CreateStreamRequestProcessor.createTopic(CreateStreamRequestProcessor.java:64) ~[?:?] at com.pega.fnx.stream.spi.impl.processor.create.CreateStreamRequestProcessor.lambda$execute$0(CreateStreamRequestProcessor.java:56) ~[?:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) ~[?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_151] at java.lang.Thread.run(Unknown Source) [?:1.8.0_151]
***Edited by Moderator Marissa to update Platform Capability tags****
Thanks Kevin.
Could you then please advise what these errors are and steps to resolve it. As of now our logs fill up very quickly because of them and our kafka-data folder is using 5.17gb alone...server team is raising questions over this.
Attached are logs and kakfa-data folder screenshot.
Hi Shakes,
We are also facing the same issue within our environment. Were you able to resolve this issue? Could you please let me know the steps to fix this?
This issue will come when you enable the DSM node for Dataflow/batch and Dataflow/realtime services. If you are not using decisioning services then you can make the below setting
1. Set below DSS value to false
interactionHistory/AllowPxInteractionsStreamAggregation
2. Open DataSet named as "Aggregation:pxInteractionsStream" and set the right Access group
3. Stop the Dataflow named as Aggregates_pxInteractionsStream
Is this fixed? I am also facing the same problem.
Embedded Kafka (which is used by Pega stream node) is now a fundamental component in Pega platform, you will need that. At the minimal, specifying these node types in JVM argument: -DNodeType=Search,Stream,BackgroundProcessing,WebUser.