Question

2
Replies
431
Views
Faisal Member since 2018 3 posts
Paypal Inc
Posted: 1 year ago
Last activity: 1 year 1 month ago
Closed

Hazelcast Issues Pega 8

We are observing a lot of issues related to hazelcast in our environment.

Sometime hazelcast engine goes down still the prsysstatus node tables gets updated. PegaCluster log stops getting updated.

2019-08-17 04:25:01,920 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Shutting down node engine...

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (zelcast.instance.NodeExtension) INFO - [10.73.198.36]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Destroying node NodeExtension.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Hazelcast Shutdown is completed in 106401 ms.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is SHUTDOWN

2019-08-17 04:25:05,321 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is MERGE_FAILED

We get Pega0090 alerts.

A lot of times there are communication failures between members.

Reason: Exception in Connection[id=602, /yyyyy:5707->/yyyyy:55601, endpoint=[yyyyyy]:5706, alive=true, type=MEMBER], thread=hz._hzInstance_1_a948064bf1085f5cb78ac1448bc1f331.IO.thread-in-1][STACK][java.io.IOException: Connection reset by peer<CR> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)<CR> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)<CR> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)<CR> at sun.nio.ch.IOUtil.read(IOUtil.java:197)<CR> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)<CR> at com.hazelcast.nio.ssl.SSLChannel.read(SSLChannel.java:189)<CR> at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:129)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKey(NioThread.java:383)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKeys(NioThread.java:368)<CR> at com.hazelcast.internal.networking.nio.NioThread.selectLoopWithFix(NioThread.java:289)<CR> at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:224)

Sub clusters getting formed

2019-08-17 04:23:54,941 [448bc1f331.migration] [ ] [ ] (artition.impl.MigrationManager) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Partition balance is ok, no need to re-partition cluster data...
2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (nternal.cluster.ClusterService) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10]
Members {size:1, ver:201} [
Member [yyyyy]:5711 - 54c532a9-36c3-4968-91c7-ae591b16648c this
]
2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (cluster.impl.MembershipManager) INFO - [xxxxxx]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Mastership is claimed with: MembersView{version=201, members=[MemberInfo{address=[xxxxxxx]:5711, uuid=54c532a9-36c3-4968-91c7-ae591b16648c, liteMember=false, memberListJoinVersion=200}]}
Anyone has a cluster size of around 60?

***Edited by Moderator Marissa to update platform capability tags; update SR Details****

Pega Platform System Administration SR Exists
Moderation Team has archived post
Share this page LinkedIn