Question

2
Replies
559
Views
Close popover
MD FAISAL KHAN (Faisal)
Paypal Inc

Paypal Inc
IN
Faisal Member since 2018 3 posts
Paypal Inc
Posted: August 19, 2019
Last activity: August 22, 2019
Closed

Hazelcast Issues Pega 8

We are observing a lot of issues related to hazelcast in our environment.

Sometime hazelcast engine goes down still the prsysstatus node tables gets updated. PegaCluster log stops getting updated.

2019-08-17 04:25:01,920 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Shutting down node engine...

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (zelcast.instance.NodeExtension) INFO - [10.73.198.36]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Destroying node NodeExtension.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Hazelcast Shutdown is completed in 106401 ms.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is SHUTDOWN

2019-08-17 04:25:05,321 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is MERGE_FAILED

We get Pega0090 alerts.

A lot of times there are communication failures between members.

Reason: Exception in Connection[id=602, /yyyyy:5707->/yyyyy:55601, endpoint=[yyyyyy]:5706, alive=true, type=MEMBER], thread=hz._hzInstance_1_a948064bf1085f5cb78ac1448bc1f331.IO.thread-in-1][STACK][java.io.IOException: Connection reset by peer<CR> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)<CR> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)<CR> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)<CR> at sun.nio.ch.IOUtil.read(IOUtil.java:197)<CR> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)<CR> at com.hazelcast.nio.ssl.SSLChannel.read(SSLChannel.java:189)<CR> at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:129)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKey(NioThread.java:383)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKeys(NioThread.java:368)<CR> at com.hazelcast.internal.networking.nio.NioThread.selectLoopWithFix(NioThread.java:289)<CR> at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:224)

Sub clusters getting formed

2019-08-17 04:23:54,941 [448bc1f331.migration] [ ] [ ] (artition.impl.MigrationManager) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Partition balance is ok, no need to re-partition cluster data...
2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (nternal.cluster.ClusterService) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10]
Members {size:1, ver:201} [
Member [yyyyy]:5711 - 54c532a9-36c3-4968-91c7-ae591b16648c this
]
2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (cluster.impl.MembershipManager) INFO - [xxxxxx]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Mastership is claimed with: MembersView{version=201, members=[MemberInfo{address=[xxxxxxx]:5711, uuid=54c532a9-36c3-4968-91c7-ae591b16648c, liteMember=false, memberListJoinVersion=200}]}
Anyone has a cluster size of around 60?

***Edited by Moderator Marissa to update platform capability tags; update SR Details****

Pega Platform System Administration Support Case Exists
Moderation Team has archived post,
Close popover This thread is closed to future replies. Content and links will no longer be updated. If you have the same/similar Question, please write a new Question.