Posted: 31 Aug 2016 21:31 EDT Last activity: 3 Jul 2017 8:10 EDT
Server startup taking 1 hour with 30 minutes spent during "Starts Initializing Search Infrastructure" step
Hi All - I recently synced up one environment(single node) by copying data from other environment(multinode). now server startup is taking 1 hour (on high level 30 minutes: during Starts Initializing Search Infrastructure and 6 minutes: during checking absolete triggers ). we are using v719, oracle 12c & weblogic1036. Earlier before restore it use to take just at max 10 minutes to come up i.e JVM settings didnt change. I deleted all system nodes records from pr_data_admin and also I dont have any node specified under System--> Settings-->search screen for elastic search as os now in the target environment.
Can someone help me resolve this issue or let me know the area to ivestigate.
If you have cloned the environment and the other environment was using full text search, then you should truncate pr_sys_statusnodes table before starting the weblogic nodes on the new setup. Otherwise, the current cluster will try to connect to the other cluster's ElasticSearch leading to index corruption and start up time overhead in the current cluster.
This is important for both the clustering technologies used in the Pega platform (Hazelcast and ElasticSearch).
Hi Rajeev - Thank you for your response! I tried truncating pr_sys_statusnodes and also deleting extract marker file and now it took around 1:20 hrs(20 minutes more b/c of deletion of extract marker), meaning it didnt help.
we are on v7.1.9 and I dont see any thread dumps in the log
I took a look at the pr_sys_statusnodes entry and found that the pyClusterAddress is "10.19.203.153:50032" and pyIndexerAddress is "10.21.44.160:9301". It does look like for the same node entry there are two different IP addresses. Does this machine have two network cards? If yes, which IP address should we be using? If the pyClusterAddress is correct, then the pyIndexerAddress needs to be fixed. This can be done by overriding the prconfig entry. You can find the details here - https://pdn.pega.com/support-articles/search-returns-empty-results.
Hi baigh - Today I do see some thread dumps in the logs(when trying to save as a rule) which wait on search threads, not sure if it is related to the issue with server startup which take more than 30 minutes on "Starts Initializing Search Infrastructure" step during startup: