Question
Rule cache not in sync across nodes
We are facing an issue in production where the newly deployed rule is getting picked up for one operator and not for the other. After some time, both operators get to use the same rule. We suspect that this is because of rule cache syncing not happening properly.
I went through https://pdn.pega.com/community/product-support/question/rule-cache-not-sync-multi-node-environment and understood that hazelcast settings in prconfig.xml can cause such problems. Was trying to follow the instructions on this post but I see that hazelcast related settings are not at all mentioned in our prconfig.xml.
Can someone help me what settings should be inserted to config file which will resolve our problem?
Thanks Basavaraj. Please validate my statement below -
Even if hazelcast is default clustering topology for Pega 7.3, we need to add these settings in prconfig so that hazelcast functions as expected.
Hi Kunal,
Enclosed are the settings for the prconfig for hazelcast;
1. ("cluster/hazelcast/ports", "5701-5800") - specify range of the hazelcast ports and make sure these are available not blocked
2. ("cluster/hazelcast/multicast/enabled", false) - by default set to false and do not change this value
3. ("identification/cluster/name") - the full text search cluster name, either user specified or auto generated
4. ("cluster/hazelcast/interface") - refer https://pdn.pega.com/support-articles/hazelcast-nodes-do-not-auto-detect-across-dmz
5. ("cluster/hazelcast/members") - refer https://pdn.pega.com/support-articles/hazelcast-nodes-do-not-auto-detect-across-dmz
Example: <env name="cluster/hazelcast/members" value="10.88.29.85,10.88.29.99" /> -- list of values, ip addresses
<env name="cluster/hazelcast/members" value="10.88.29.85-99" /> -- range of values (.85-.99 inclusive)
6. ("cluster/hazelcast/heartbeat", "OFF") - by default set to false
7. ("alerts/cluster/clockdeltathreshold", 10): refer https://pdn.pega.com/performance/understanding-the-pega0055-alert-clock-drift-exceeded-the-configured-time-threshold
8. ("identification/cluster/protocol" , "hazelcast") - default clustering topology set to hazelcast
9. ("identification/cluster/logging" , "") -- yes, an empty string. Used for debug purpose
This is used along with below in prlogging.xml for debug purposes if the hazelcast is failing to forma cluster
<appender name="HAZELCAST" class="com.pega.pegarules.priv.util.DailySizeRollingFileAppenderPega">
<param name="FileNamePattern" value="'@{web.tmpdir}/PegaHAZELCAST-'yyyy-MMM-dd'.log'"/>
<param name="MaxFileSize" value="50MB" />
<layout class="com.pega.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d [%20.20t] [%20.20X{tenantid}] [%20.20X{app}] (%30.30c{3}) %-5p %X{stack} %X{userid} - %m%n"/>
</layout>
</appender>
<category name="com.hazelcast">
<priority value="info"/>
<appender-ref ref="HAZELCAST"/>
</category>
In your scenario, settings 1,2, 4 and 5 need to be added.
Example: Say if you have two nodes;
NODE 1:
<env name="cluster/hazelcast/ports" value="5701-5800"/> <!-- available ports that have access across the firewall for use by our cluster -->
<env name="cluster/hazelcast/interface" value="xx.x.xxx.xx"/> <!-- The IP address that I will listen to -->
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="xx.x.xxx.*"/> <!-- optional list of IPs for other members in our cluster - can be a range or comma separated -->
NODE 2:
<env name="cluster/hazelcast/ports" value="5701-5800"/> <!-- available ports that have access across the firewall for use by our cluster -->
<env name="cluster/hazelcast/interface" value="xx.x.xxx.xx"/> <!-- The IP address that I will listen to -->
<env name="cluster/hazelcast/multicast/enabled" value="false"/>
<env name="cluster/hazelcast/members" value="xx.x.xxx.*"/> <!-- optional list of IPs for other members in our cluster - can be a range or comma separated -->
You have to make sure that the above nodes are started serially(one by one) and not paralell to make sure that the hazelcast cluster is formed appropriately. You will also notice it in the PegaRULES logs the cluster details with hazelcast members with their IP address and ports
---sample from the PRPC logfile----
Members [2] {
Member [XX.XX.XXX.XXX]:5702
Member [XX.XX.XXX.XXX]:5701 this
}
In PRPC v7.3.1 there is a support for both hazelcast and ignite clustering topology. You can refer to this PDN https://pdn.pega.com/community/product-support/question/clustering-topology-apache-ignite-vs-hazlecast for more information.
Let me know if above information helps.
Regards,
Basavaraj Kadakol