Question

4
Replies
505
Views
Close popover
Sreedhar Muduganti (SreedharM3618)
Transport for NSW

Transport for NSW
AU
SreedharM3618 Member since 2019 7 posts
Transport for NSW
Posted: August 11, 2020
Last activity: February 12, 2021

Pega 8.4 kafka stability issues: CharlatanExceptions in logs

We recently upgraded to 8.4.1 from 7.2.2 and after that we have been constantly seeing lot of issues related to kafka on our Dev and SIT servers. 

Recently we have done multi node setup in our test environments (2 separate servers) with default node classification  (i.e Web, BackgroundProcessing, Search and Stream)

1) we started seeing strange issues like the below log which gets logged every 10 seconds.

We tried truncating PR_SYS_STATUSNODES and started the nodes and things looked fine for couple of days before surfacing again.

Stream status shows normal for both the nodes.

2) One more strange behaviour is that Admin studio is showing '0 nodes as running' though both the nodes are running fine and we're able to log into the applications on both the nodes.

The communication b/w both the nodes also seem to be working - I telnet'ed connecting to different ports used by pega from one node to other and the connection is established okay (unless I overlooked something).

Has anyone faced this and any help is really appreciated?

 

=================================

2020-08-11 22:35:55,215 [  New I/O worker #65] [  STANDARD] [                    ] [                    ] (ion.service.SessionServiceImpl) ERROR   - Failed to accept in coming connection for the session '-1375665877' com.pega.charlatan.utils.CharlatanException$SessionExpiredException: KeeperErrorCode = Session expired         at com.pega.charlatan.session.service.SessionServiceImpl.handleConnectRequest(SessionServiceImpl.java:129) ~[charlatan-server.jar:?]         at com.pega.charlatan.session.service.SessionServiceImpl.processRequest(SessionServiceImpl.java:84) ~[charlatan-server.jar:?]         at com.pega.charlatan.server.CharlatanNettyConnection.receiveMessage(CharlatanNettyConnection.java:78) ~[charlatan-server.jar:?]         at com.pega.charlatan.server.CharlatanNettyServer$CharlatanChannelHandler.processMessage(CharlatanNettyServer.java:213) ~[charlatan-server.jar:?]         at com.pega.charlatan.server.CharlatanNettyServer$CharlatanChannelHandler.messageReceived(CharlatanNettyServer.java:207) ~[charlatan-server.jar:?]         at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.10.6.Final.jar:?]         at com.pega.charlatan.server.CharlatanNettyServer$CharlatanChannelHandler.handleUpstream(CharlatanNettyServer.java:200) ~[charlatan-server.jar:?]         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]         at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_77]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_77]         at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59) ~[d-node.jar:?]         at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56) ~[d-node.jar:?]         at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?]         at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124) ~[d-node.jar:?]

===============================

 

***Edited by Moderator Marissa to change type from General to Upgrade, update Platform Capability tags****

***Edited by Moderator: Pooja to update product version***
Pega Platform 8.4.1 System Administration Upgrades Government Upgrade