Issue after starting multiple Pega nodes in paralell
Our Production and UAT Pega 7.2 environment setup has a cluster with six nodes running on JBoss EAP 6.4 (on RHEL Linux), connecting to a DB2 database and using a double schema configuration (DATA and RULES) running on AIX.
We have noticed that when starting more than one node at a time we experience different kind of issues always resulting in a production incident.
During the startup process we don't see any error in the log files, application becomes available and we are able to login and navigate without any issue, but after users start navigating through the application, they face diverse errors (some of them shown below).
However if we start the nodes one-by-one, and let them complete the startup process, we don't face any issues.
We have decided to use this approach as a best practice: start nodes one-by-one.
Should we need to perform a specific configuration in the application server or database layer to avoid issues when starting two or more nodes in parallel?
Does this kind of issue could be related to tuning needed during the start process?
Best regards, and thanks in advance!
P.D.: These are some of the errors we face after starting up the nodes in parallel:
ERROR - Section 'FlowActionHTML_Simple' execution error on page 'pyWorkPage' of class 'APPName-Work-Party'.
com.pega.pegarules.pub.generator.UnresolvedAssemblyError: in RULE-OBJ-WHEN WORK- PZSHOWACTIONASBUTTONS #20130919T005132.921 GMT: Invalid expresson for label
** Error loading RULE-DECLARE-PAGES D_GETCUSTOMERPROFILE #20170906T153212.229 GMT [CustomerType:"xxx",CustomerID:"xxxx"] ,
Reason : ** Error occurred trying to parse the expression and augmenting the code: com.pega.pegarules.pub.PRRuntimeException: caught DatabaseException while trying to save conclusion Property!PEPNUMBER: Exception saving ClipboardPage instance via e-tier bean: code:
SQLState: java.lang.RuntimeException: Problem during method invocation (saveInOwnTransaction)
Message: DatabaseException caused by prior exception: javax.ejb.EJBException: java.lang.RuntimeException:
Problem during method invocation (saveInOwnTransaction) DatabaseException caused by prior exception:
java.lang.RuntimeException: Problem during method invocation (saveInOwnTransaction) DatabaseException caused by prior exception:
Problem encountered when getting connection for database pegarules 0
javax.resource.ResourceException: IJ000460: Error checking for a transaction DatabaseException caused by prior exception:
java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction | SQL Code: 0 | SQL State: null