Server not coming up after completion of upgrade scripts
The server dis not come up after deploying the prweb. The error message from the catalina logs are below. Please note we are directly giving our DB credentials in the prbootstrap.properties file
Configuration:
- upgrade from Pega 6.2 SP1 to Pega 7.1.9
- Java 1.7, Tomcat 7.0 and Oracle 11g
Errors found:
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrapDataSource
19830421: Loading bootstrap properties from /prbootstrap.properties
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.SettingReaderJNDI
19830421: Could not find java:comp/env/prbootstrap/ in the local JNDI context, skipping prconfig setting lookup
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.SettingReaderJNDI
19830421: Could not find prbootstrap in the local JNDI context, skipping prconfig setting lookup
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.SettingReaderJNDI
19830421: Could not find java:comp/env/prconfig/ in the local JNDI context, skipping prconfig setting lookup
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.SettingReaderJNDI
19830421: Could not find prconfig in the local JNDI context, skipping prconfig setting lookup
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrapDataSource
19830421: Bootstrap datatables schema: null
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrapDataSource
19830421: Archive version [0]
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrap
19830421: Engine schema defined in allclasses: null
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrap
19830421: Engine classes schema: null
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrap
19830421: Assembled classes schema: null
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrap
19830421: prbootstrap.properties merged with prbootstrap entries in Data-Admin-System-Settings
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRBootstrap
19830421: Using latest detected engine version:(EngineCodeSetVersion= 06-02-10). To specify a different version, use the setting com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode in prbootstrap.properties.
May 05, 2016 9:10:47 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader
19830421: Will load phase 2 bootstrap from Pega-EngineCode:06-02-10
May 05, 2016 9:10:47 AM com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega
SEVERE: Problem during method invocation (contextInitialized)
java.lang.NoClassDefFoundError: com/pega/pegarules/internal/bootstrap/BaseLogger
at com.pega.pegarules.internal.bootstrap.phase2.PRBootstrapImpl.<clinit>(PRBootstrapImpl.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.checkForStartup(PRBootstrap.java:685)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:408)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:272)
at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:83)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4779)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:962)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:536)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1471)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:301)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at org.apache.catalina.manager.ManagerServlet.check(ManagerServlet.java:1436)
at org.apache.catalina.manager.ManagerServlet.deploy(ManagerServlet.java:856)
at org.apache.catalina.manager.ManagerServlet.doGet(ManagerServlet.java:353)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)
at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1770)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.pega.pegarules.internal.bootstrap.BaseLogger
at com.pega.pegarules.internal.bootstrap.PRMiniLoader.loadClass(PRMiniLoader.java:388)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 51 more
**Moderation Team has archived post**
This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.
Updated: 10 May 2016 9:24 EDT
Cognizant
IN
To address this issue we set the following property in the prbootstarp.properties as shown below.
com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode=07-10-19
However this resulted in a new error in Catalina ...
To address this issue we set the following property in the prbootstarp.properties as shown below.
com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode=07-10-19
However this resulted in a new error in Catalina ...
May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader
SEVERE:
May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader
SEVERE: Subsequent exceptions relating to class not being found are all related to this root cause...
May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader
SEVERE: Check to make sure the database is correctly configured and that the codeset version specified was correct and is present in that database.
May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader
SEVERE: =========================================================================================
May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader
SEVERE:
May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRBootstrap
19830421:
============ PegaRULES Bootstrap Configuration ============
Date: Fri May 06 09:28:25 EDT 2016
Java Version: Oracle Corporation 1.7.0_11
Configuration: file:/local/test/deployment/ABC_TEST_tomcat/webapps/prweb/WEB-INF/classes/prbootstrap.properties
Pega-EngineCode: =07-10-19
Database: Oracle Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning and Automatic Storage Management options
Driver: Oracle JDBC driver 11.2.0.3.0
(!) Ignored JNDI Options
(!) Unable to use given database configuration
(!) Unable to locate classes on filesystem
Cognizant
IN
There is a PDN article carrying a similar error message for 7.1.7 upgrade ...
https://community.pega.com/support/support-articles/upgrade-717-dataonly-fails-finding-codeset-version-required
However we are not sure if its solution applies to our scenario in 7.1.9 since the following properties in the setupdatabase.properties were never modified and left untouched as shown below.
# codeset and version for loading hotfixes
import.codeset.name=Customer
import.codeset.version=06-01-01
Updated: 11 May 2016 4:24 EDT
Cognizant
IN
we commented out the following property in the bootstrap.properties
#com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode=07-10-19
Then we un-commented the following two properties and supplied schema names in the bootstrap.properties
com.pega.pegarules.bootstrap.allclasses.schema=APP_RULES
com.pega.pegarules.bootstrap.datatables.schema=APP_DATA
After this we found more progress in the Catalina and PegaRules.log
As such we are stuck with the error below.
Do we need to specify rules/data schema names again; either in the PRPC at prconfig.xml (or) in the webserver at context.xml ?
2016-05-10 12:21:25,489 [.svr.us.abc.net] [ STANDARD] [ ] [ ] (cluster.internal.PRClusterImpl) ERROR - Error while retrieving the run state of the nodes
com.pega.pegarules.pub.database.DatabaseException: Database-General Problem encountered executing a stored procedure 6550 65000 ORA-06550: line 1, column 7:
PLS-00201: identifier 'APP_DATA.SPPR_SYS_UPDATESCACHE_GETTIME' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
Updated: 11 May 2016 4:19 EDT
Capgemini India
IN
We are not using any datasource/jndi. The entire DB configuration is hardcoded in the prconfig.xml. Is there a way to specify the defaultschema and the dataschema in the prconfig.xml
This procedure SPPR_SYS_UPDATESCACHE_GETTIME exists in the rules schema, but not in the data schema. Any help/suggestion will be really helpful.
AIG
US
Please try with contenxt.xml jndi setup method, looks like DB connection issue, engine is not started
revert boottrap property changes to original status
add DB connection like below to context.xml
delete content inside Pega temp directory, start the server
<Resource name="jdbc/PegaRULES"
auth="Container"
type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@1x.xx.244.237:1521:ORCLDB"
username="PegaAdminbb"
password="ctsxxxxx"
maxActive="100"
maxIdle="30"
maxWait="10000"
/>
<Environment name="prconfig/database/databases/PegaRULES/defaultSchema" value="PegaRulesbb" type="java.lang.String" />
<Environment name="prconfig/database/databases/PegaDATA/defaultSchema" value="PegaDatabb" type="java.lang.String" />
<Environment name="url/initialization/explicittempdir" value="C:\TMP\Pega_Temp" type="java.lang.String" />
AIG
US
In happy connection, you should see following logs -
In happy connection, you should see following logs -
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Deployed in web tier only
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Servlet container: Tomcat
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Web server: Apache Tomcat/8.0.11
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Web server environment: Tomcat
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Web-tier default Engine bean name: EngineBMT
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Web to Enterprise tier delegate type: pojo
2016-05-11 14:04:24,307 [ocalhost-startStop-1] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Now starting the PegaRULES Engine...
2016-05-11 14:04:24,478 [ocalhost-startStop-1] [ ] [ ] [ ] ( internal.mgmt.PRNodeImpl) INFO - Using codeset version and timestamp map: {pega-enginecode=07-10-23(this can vary as per Pega 7 ML version)@2016-05-09 11:07:00.0, customer=06-01-01@1970-01-01 05:30:00.0}
2016-05-11 14:04:25,149 [ CTSCcc] [ ] [ ] [ ] (rnal.mgmt.InitialConfiguration) INFO - Generated source : file://runtime:/PRGenJava/
2016-05-11 14:04:25,149 [ CTSCcc] [ ] [ ] [ ] (rnal.mgmt.InitialConfiguration) INFO - Generated classes: file://runtime:/PRGenClasses/
Capgemini India
IN
Thanks Bhushan for the suggestion.
However, our existing configuration in v6 had everything in the prconfig without having any jndi/datasource defined. The client wants to go with the same configuration in v7 also for now. As I said earlier we have every DB information (including credentials) mentioned in the prconfig.xml. We are using keyring for the passwords But there is no mention of the separate rule and data schema there. Is there a way to mention the rule schema and data schema in the prconfig like below
<env value="APP_DATA" name="database/databases/PegaDATA/defaultSchema"/>
<env value="APP_RULES" name="database/databases/PegaRULES/defaultSchema"/>
Cognizant
IN
As such only our prbootstrap.properties has the schema details; we are still working on specifying the same schema details in either the context.xml or prconfig.xml
Upgrade guide for 7.1.9 is demanding a change in the context.xml; there is no reference to prconfig.xml
Database connections have to be handled at the application server level so the guide is leaning towards context.xml and that is fine.
Meanwhile please verify the warning we found in the PegaRULES.log
Its a clear indication that specifying schema details only in prbootstrap.properties is not enough and it needs to be specified at least one more time in context.xml or prconfig.xml.
2016-05-10 11:03:22,037 [9.27.75-16000-exec-3] [ ] [ ] [ ] ( etier.impl.EngineStartup) INFO - Now starting the PegaRULES Engine...
2016-05-10 11:03:22,127 [9.27.75-16000-exec-3] [ ] [ ] [ ] ( internal.mgmt.PRNodeImpl) INFO - Using codeset version and timestamp map: {pega-enginecode=07-10-19@2016-04-29 04:41:00.0, customer=06-01-01@2013-01-09 12:34:00.0}
2016-05-10 11:03:22,295 [9.27.75-15000-exec-3] [ ] [ ] [ ] (l.access.ConnectionManagerImpl) WARN - The database pegadata is only defined in the bootstrap configuration; create a Data-Admin-DB-Name definition
Updated: 12 May 2016 7:37 EDT
Cognizant
IN
We did specify the default data and rules schema in our prConfig.xml
Then we attempted a restart on our tomcats; we got the following errors in the PegaRULES.log
ERROR TWO seems to hold the clue. Please verify both the errors and help us.
ERROR ONE
2016-05-11 12:11:18,761 [EPAGEDEFINITIONCACHE] [ STANDARD] [ ] [ ] ( external.async.IStartupTask) INFO - load Declarative Page Definition Cache ..done
2016-05-11 12:11:26,932 [123.abc.xy.org.net] [ STANDARD] [ ] [ ] ( internal.mgmt.Executable) ERROR - Exception
java.lang.NullPointerException
at com.pega.pegarules.generation.internal.PRGenProviderImpl.get(PRGenProviderImpl.java:474)
ERROR TWO REFERENCE # (https://community.pega.com/support/support-articles/unable-start-pega-after-upgrade-716-719)
We did specify the default data and rules schema in our prConfig.xml
Then we attempted a restart on our tomcats; we got the following errors in the PegaRULES.log
ERROR TWO seems to hold the clue. Please verify both the errors and help us.
ERROR ONE
2016-05-11 12:11:18,761 [EPAGEDEFINITIONCACHE] [ STANDARD] [ ] [ ] ( external.async.IStartupTask) INFO - load Declarative Page Definition Cache ..done
2016-05-11 12:11:26,932 [123.abc.xy.org.net] [ STANDARD] [ ] [ ] ( internal.mgmt.Executable) ERROR - Exception
java.lang.NullPointerException
at com.pega.pegarules.generation.internal.PRGenProviderImpl.get(PRGenProviderImpl.java:474)
ERROR TWO REFERENCE # (https://community.pega.com/support/support-articles/unable-start-pega-after-upgrade-716-719)
2016-05-11 12:11:34,548 [123.abc.xy.org.net] [ STANDARD] [ ] [ ] ( etier.impl.EngineStartup) ERROR - PegaRULES initialization failed. Server: 123.abc.xy.org.net
com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed
at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:396
...
...
Caused by: com.pega.pegarules.pub.runtime.IndeterminateConditionalException: Executable.evaluateWhen-Exception Encountered a :java.lang.NullPointerException
at com.pega.pegarules.session.internal.mgmt.Executable.evaluateWhen(Executable.java:4265)
Cognizant
IN
After we performed the dataonly upgrade in QA; we tired to bring up the tomcat.
We got IndeterminateConditionalException: Executable.evaluateWhen-Exception again.
Having a custom access group for the DATA-ADMIN-REQUESTOR PEGA!BATCH is the rootcause.
Before we start an upgrade; we have to verify this requestor in the target environment.
If this was missed and the dataonly upgrade is already done then try to replace the concerned row in
pr_data_admin using a DB-LINK (or) replace only the PZPVSTREAM column from a good DB source.
Consult with your DBA and they would know how to do it.
Select * from PegaDATA.pr_data_admin
where
pzInsKey = 'DATA-ADMIN-REQUESTOR PEGA!BATCH'
Cognizant
IN
We addressed the ERROR 2 above based on the solution provided in the reference after which we made some progress in the PegaRULES log. Now we got another error...
Pega engine is not finding the OOTB stored procedures in the upgraded data schema; we verified this manually and they were present in both rules and data schema.
We found a similar error in article SA-3506; please verify the error below and help us with your suggestions.
ERROR REFERENCE # (https://community.pega.com/support/support-articles/rulesync-does-not-happen-multiple-nodes-after-pega7-upgrade)
We addressed the ERROR 2 above based on the solution provided in the reference after which we made some progress in the PegaRULES log. Now we got another error...
Pega engine is not finding the OOTB stored procedures in the upgraded data schema; we verified this manually and they were present in both rules and data schema.
We found a similar error in article SA-3506; please verify the error below and help us with your suggestions.
ERROR REFERENCE # (https://community.pega.com/support/support-articles/rulesync-does-not-happen-multiple-nodes-after-pega7-upgrade)
2016-05-13 07:12:29,499 [EPAGEDEFINITIONCACHE] [ STANDARD] [ ] [ ] ( external.async.IStartupTask) INFO - load Declarative Page Definition Cache ..done
2016-05-13 07:12:31,445 [ LOAD_INSTANCEFILTER] [ STANDARD] [ ] [ ] (al.access.InstanceExistsFilter) INFO - Added 139328 entries to the instance exists filter in 7.155000 seconds with a false positive rate of 0.001
2016-05-13 07:12:31,445 [ LOAD_INSTANCEFILTER] [ STANDARD] [ ] [ ] ( external.async.IStartupTask) INFO - load instance filter ..done
2016-05-13 07:12:31,939 [.abc.xy.org...] [ STANDARD] [ ] [ ] (internal.PRClusterProviderImpl) INFO - Using system pulse type: cluster
2016-05-13 07:12:32,310 [.abc.xy.org...] [ STANDARD] [ ] [ ] (cluster.internal.PRClusterImpl) ERROR - Error while retrieving the run state of the nodes
com.pega.pegarules.pub.database.DatabaseException: Database-General Problem encountered executing a stored procedure 6550 65000 ORA-06550: line 1, column 7:
PLS-00201: identifier 'PEGA_DATA.SPPR_SYS_UPDATESCACHE_GETTIME' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
From: (B3451A9B6F42E23A2459C370FF286CCC:(PREnvironment))
SQL: {call PEGA_DATA.sppr_sys_updatescache_gettime (?,?,?)}
Capgemini India
IN
The latest issue occurred as the DB user used by the application did not have proper grants to access the stored procedure SPPR_SYS_UPDATESCACHE_GETTIME. This happened as the said SP was deleted and then recreated again by the upgrade scripts and as a results the existing grants got removed. Note; we are using a different user (not the schema user) to connect to the DB from Pega. We have overcome the issue by giving proper grants and privilege on the existing DB objects in the Data schema to the user. The server is up and running now.
Thanks all for the support.
Cognizant
IN
We did not follow the resolution found in the article SA-3506 ( we did not move around store procs between rules and data shcema) instead whatever has been posted above by Ranesh was followed for data_schema.SPPR_SYS_UPDATESCACHE_GETTIME and the tomcat came up.
We were able to login and view the Pega7 Designer Studio now.
Also, post upgrade only SPPR_REBUILD_INDEXES should have been present in rules_schema however in our case we found other OOTB store procs too. These extra store procs such as SPPC_DATA_UNIQUEID / SPPR_PURGE_TABLE / SPPR_POPULATE_INDEX_DATA ... were manually dropped from rules_schema.
Cognizant
IN
After we performed the dataonly upgrade in QA; we restarted the tomcat.
We got this error and the tomcat did not come up...
com.pega.pegarules.pub.context.InvalidConfigurationException: No such Rule-Application instance: CUSTOM 01.01.01
Workarounds could be found in these articles and they helped us resolve this issue.
https://community.pega.com/support/support-articles/initializationfailederror-log-file-during-prpc-start
https://community.pega.com/support/support-articles/cannot-start-pega-instance