Question
Joining_failed for Kafka node
Hi,
On our Pega 7.4 development environment(Linux/tomcat based) we can start all decioning processes/nodes fine except for the stream service(kafka). The exception is below. I have allready tried to set the logging levels on both Pega side as well as the the Kafka Log4j.Properties but not much information coming out into the log files. The Zookeeper and Kafka server start just fine from the Linux command line but just not from within Pega it seems.
com.pega.dsm.dnode.api.StreamServiceException: Unable to start Kafka broker.
at com.pega.dsm.kafka.Kafka$1.process(Kafka.java:149)
at com.pega.dsm.dnode.api.prpc.service.managedprocess.ManagedProcess.start(ManagedProcess.java:117)
at com.pega.dsm.kafka.Kafka.bootstrap(Kafka.java:161)
at com.pega.dsm.dnode.api.server.StreamServerService$StreamServiceStartOperation$3.emit(StreamServerService.java:400)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:338)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:40)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:102)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:91)
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.doActualServerStart(StartOperation.java:139)
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.access$400(StartOperation.java:22)
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation$2.execute(StartOperation.java:120)
at com.pega.dsm.dnode.util.OperationWithLock$LockingOperation.couldAcquireLock(OperationWithLock.java:131)
at com.pega.dsm.dnode.util.OperationWithLock$LockingOperation.performLockOperation(OperationWithLock.java:121)
at com.pega.dsm.dnode.util.OperationWithLock$LockingOperation.access$100(OperationWithLock.java:69)
at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:66)
at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:62)
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLockInternal(ServiceHelper.java:205)
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLock(ServiceHelper.java:169)
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.initializeServerMode(StartOperation.java:117)
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.access$200(StartOperation.java:22)
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation$1.emit(StartOperation.java:83)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:338)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:40)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:102)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.toList(DataObservableImpl.java:81)
at com.pega.dsm.dnode.impl.prpc.service.RemoteServiceOperation$ExecuteOperationMessage.call(RemoteServiceOperation.java:112)
at com.pega.dsm.dnode.impl.prpc.service.RemoteServiceOperation$ExecuteOperationMessage.call(RemoteServiceOperation.java:72)
at com.pega.dsm.dnode.api.prpc.service.CallableMessageWithContext$1.run(CallableMessageWithContext.java:33)
at com.pega.dsm.dnode.api.prpc.service.CallableMessageWithContext$1.run(CallableMessageWithContext.java:30)
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:52)
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper$3.run(ServiceHelper.java:219)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1357)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1105)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:912)
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:212)
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:104)
at com.pega.dsm.dnode.api.prpc.service.CallableMessageWithContext.call(CallableMessageWithContext.java:30)
at com.pega.dsm.dnode.api.prpc.service.AbstractDsmService.handleMessageInternal(AbstractDsmService.java:257)
at com.pega.dsm.dnode.impl.prpc.service.ServiceDeploymentImpl$PostMessageJobData.handleMessage(ServiceDeploymentImpl.java:500)
at com.pega.dsm.dnode.impl.prpc.service.ServiceDeploymentImpl$PostMessageJob.execute(ServiceDeploymentImpl.java:470)
at com.pega.dsm.dnode.impl.prpc.service.ServiceDeploymentImpl$PostMessageJob.execute(ServiceDeploymentImpl.java:467)
at com.pega.pegarules.cluster.internal.PRPCTask.call(PRPCTask.java:130)
at com.pega.pegarules.cluster.internal.PRPCTask.call(PRPCTask.java:53)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:241)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:228)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
regards,
Leo de Blauaw
Hello,
Could you please check the table pegadata.pr_sys_statusnode. If you will see any additional records (web servers), please remove and restart the entire environment.
Query:
select pxinsname,pycleanshutdown,pysystemname,pysysnodeid,pysystemname,pyrunstate from pegadata.pr_sys_statusnodes;