To address this issue we set the following property in the prbootstarp.properties as shown below.

com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode=07-10-19

However this resulted in a new error in Catalina ...

May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader

SEVERE:

May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader

SEVERE: Subsequent exceptions relating to class not being found are all related to this root cause...

May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader

SEVERE: Check to make sure the database is correctly configured and that the codeset version specified was correct and is present in that database.

May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader

SEVERE: =========================================================================================

May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRMiniLoader

SEVERE:

May 06, 2016 9:28:25 AM com.pega.pegarules.internal.bootstrap.PRBootstrap

19830421:

============ PegaRULES Bootstrap Configuration ============

           Date: Fri May 06 09:28:25 EDT 2016

   Java Version: Oracle Corporation 1.7.0_11

  Configuration: file:/local/test/deployment/ABC_TEST_tomcat/webapps/prweb/WEB-INF/classes/prbootstrap.properties

Pega-EngineCode: =07-10-19

       Database: Oracle Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning and Automatic Storage Management options

         Driver: Oracle JDBC driver 11.2.0.3.0

(!) Ignored JNDI Options

(!) Unable to use given database configuration

(!) Unable to locate classes on filesystem

There is a PDN article carrying a similar error message for 7.1.7 upgrade ...
https://pdn.pega.com/support-articles/upgrade-717-dataonly-fails-finding-codeset-version-required

However we are not sure if its solution applies to our scenario in 7.1.9 since the following properties in the setupdatabase.properties were never modified and left untouched as shown below.

# codeset and version for loading hotfixes

import.codeset.name=Customer

import.codeset.version=06-01-01 

we commented out the following property in the bootstrap.properties

#com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode=07-10-19

Then we un-commented the following two properties and supplied schema names in the bootstrap.properties

com.pega.pegarules.bootstrap.allclasses.schema=APP_RULES

com.pega.pegarules.bootstrap.datatables.schema=APP_DATA

After this we found more progress in the Catalina and PegaRules.log

As such we are stuck with the error below. 

Do we need to specify rules/data schema names again; either in the PRPC at prconfig.xml (or) in the webserver at context.xml ?

2016-05-10 12:21:25,489 [.svr.us.abc.net] [  STANDARD] [                    ] [                    ] (cluster.internal.PRClusterImpl) ERROR   - Error while retrieving the run state of the nodes

com.pega.pegarules.pub.database.DatabaseException: Database-General  Problem encountered executing a stored procedure   6550 65000 ORA-06550: line 1, column 7:

PLS-00201: identifier 'APP_DATA.SPPR_SYS_UPDATESCACHE_GETTIME' must be declared

ORA-06550: line 1, column 7:

PL/SQL: Statement ignored

We are not using any datasource/jndi. The entire DB configuration is hardcoded in the prconfig.xml. Is there a way to specify the defaultschema and the dataschema in the prconfig.xml

This procedure SPPR_SYS_UPDATESCACHE_GETTIME exists in the rules schema, but not in the data schema. Any help/suggestion will be really helpful.

Please try with contenxt.xml jndi setup method, looks like DB connection issue, engine is not started

  revert boottrap property changes to original status

  add DB connection like below to context.xml

  delete content inside Pega temp directory, start the server

<Resource name="jdbc/PegaRULES"

auth="Container"

type="javax.sql.DataSource"

driverClassName="oracle.jdbc.OracleDriver"

url="jdbc:oracle:thin:@1x.xx.244.237:1521:ORCLDB"

username="PegaAdminbb"

password="ctsxxxxx"

maxActive="100"

maxIdle="30"

maxWait="10000"

/>

<Environment name="prconfig/database/databases/PegaRULES/defaultSchema" value="PegaRulesbb" type="java.lang.String" />

<Environment name="prconfig/database/databases/PegaDATA/defaultSchema" value="PegaDatabb" type="java.lang.String" />

<Environment name="url/initialization/explicittempdir" value="C:\TMP\Pega_Temp" type="java.lang.String" />

In happy connection, you should see following logs -

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - Deployed in web tier only

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -                        Servlet container: Tomcat

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -                               Web server: Apache Tomcat/8.0.11

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -                   Web server environment: Tomcat

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -        Web-tier default Engine bean name: EngineBMT

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -     Web to Enterprise tier delegate type: pojo

2016-05-11 14:04:24,307 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - Now starting the PegaRULES Engine...

2016-05-11 14:04:24,478 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      internal.mgmt.PRNodeImpl) INFO    - Using codeset version and timestamp map: {pega-enginecode=07-10-23(this can vary as per Pega 7 ML version)@2016-05-09 11:07:00.0, customer=06-01-01@1970-01-01 05:30:00.0}

2016-05-11 14:04:25,149 [     CTSCcc] [          ] [                    ] [                    ] (rnal.mgmt.InitialConfiguration) INFO    - Generated source : file://runtime:/PRGenJava/

2016-05-11 14:04:25,149 [     CTSCcc] [          ] [                    ] [                    ] (rnal.mgmt.InitialConfiguration) INFO    - Generated classes: file://runtime:/PRGenClasses/

Thanks Bhushan for the suggestion.

However, our existing configuration in v6 had everything in the prconfig without having any jndi/datasource defined. The client wants to go with the same configuration in v7 also for now. As I said earlier we have every DB information (including credentials) mentioned in the prconfig.xml. We are using keyring for the passwords But there is no mention of the separate rule and data schema there. Is there a way to mention the rule schema and data schema in the prconfig like below

<env value="APP_DATA" name="database/databases/PegaDATA/defaultSchema"/>

<env value="APP_RULES" name="database/databases/PegaRULES/defaultSchema"/>

As such only our prbootstrap.properties has the schema details; we are still working on specifying the same schema details in either the context.xml or prconfig.xml

Upgrade guide for 7.1.9 is demanding a change in the context.xml; there is no reference to prconfig.xml

Database connections have to be handled at the application server level so the guide is leaning towards context.xml and that is fine.

Meanwhile please verify the warning we found in the PegaRULES.log

Its a clear indication that specifying schema details only in prbootstrap.properties is not enough and it needs to be specified at least one more time in context.xml or prconfig.xml. 

2016-05-10 11:03:22,037 [9.27.75-16000-exec-3] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - Now starting the PegaRULES Engine...

2016-05-10 11:03:22,127 [9.27.75-16000-exec-3] [          ] [                    ] [                    ] (      internal.mgmt.PRNodeImpl) INFO    - Using codeset version and timestamp map: {pega-enginecode=07-10-19@2016-04-29 04:41:00.0, customer=06-01-01@2013-01-09 12:34:00.0}

2016-05-10 11:03:22,295 [9.27.75-15000-exec-3] [          ] [                    ] [                    ] (l.access.ConnectionManagerImpl) WARN    - The database pegadata is only defined in the bootstrap configuration; create a Data-Admin-DB-Name definition

We did specify the default data and rules schema in our prConfig.xml

Then we attempted a restart on our tomcats; we got the following errors in the PegaRULES.log

ERROR TWO seems to hold the clue. Please verify both the errors and help us.

ERROR ONE

2016-05-11 12:11:18,761 [EPAGEDEFINITIONCACHE] [  STANDARD] [                    ] [                    ] (   external.async.IStartupTask) INFO    - load Declarative Page Definition Cache  ..done

2016-05-11 12:11:26,932 [123.abc.xy.org.net] [  STANDARD] [                    ] [                    ] (      internal.mgmt.Executable) ERROR   - Exception

java.lang.NullPointerException

 at com.pega.pegarules.generation.internal.PRGenProviderImpl.get(PRGenProviderImpl.java:474)

ERROR TWO    REFERENCE # (https://pdn.pega.com/support-articles/unable-start-pega-after-upgrade-716-719)

2016-05-11 12:11:34,548 [123.abc.xy.org.net] [  STANDARD] [                    ] [                    ] (      etier.impl.EngineStartup) ERROR   - PegaRULES initialization failed. Server: 123.abc.xy.org.net

com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed

at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:396

...

...

Caused by: com.pega.pegarules.pub.runtime.IndeterminateConditionalException: Executable.evaluateWhen-Exception Encountered a :java.lang.NullPointerException

at com.pega.pegarules.session.internal.mgmt.Executable.evaluateWhen(Executable.java:4265)

We addressed the ERROR 2 above based on the solution provided in the reference after which we made some progress in the PegaRULES log. Now we got another error...

Pega engine is not finding the OOTB stored procedures in the upgraded data schema; we verified this manually and they were present in both rules and data schema.

We found a similar error in article SA-3506; please verify the error below and help us with your suggestions.

ERROR REFERENCE # (https://pdn.pega.com/support-articles/rulesync-does-not-happen-multiple-nodes-after-pega7-upgrade)

2016-05-13 07:12:29,499 [EPAGEDEFINITIONCACHE] [  STANDARD] [                    ] [                    ] (   external.async.IStartupTask) INFO    - load Declarative Page Definition Cache  ..done

2016-05-13 07:12:31,445 [ LOAD_INSTANCEFILTER] [  STANDARD] [                    ] [                    ] (al.access.InstanceExistsFilter) INFO    - Added 139328 entries to the instance exists filter in 7.155000 seconds with a false positive rate of 0.001

2016-05-13 07:12:31,445 [ LOAD_INSTANCEFILTER] [  STANDARD] [                    ] [                    ] (   external.async.IStartupTask) INFO    - load instance filter  ..done

2016-05-13 07:12:31,939 [.abc.xy.org...] [  STANDARD] [                    ] [                    ] (internal.PRClusterProviderImpl) INFO    - Using system pulse type: cluster

2016-05-13 07:12:32,310 [.abc.xy.org...] [  STANDARD] [                    ] [                    ] (cluster.internal.PRClusterImpl) ERROR   - Error while retrieving the run state of the nodes

com.pega.pegarules.pub.database.DatabaseException: Database-General Problem encountered executing a stored procedure 6550 65000 ORA-06550: line 1, column 7:

PLS-00201: identifier 'PEGA_DATA.SPPR_SYS_UPDATESCACHE_GETTIME' must be declared

ORA-06550: line 1, column 7:

PL/SQL: Statement ignored

From: (B3451A9B6F42E23A2459C370FF286CCC:(PREnvironment)) 

SQL: {call PEGA_DATA.sppr_sys_updatescache_gettime (?,?,?)}

The latest issue occurred as the DB user used by the application did not have proper grants to access the stored procedure SPPR_SYS_UPDATESCACHE_GETTIME. This happened as the said SP was deleted and then recreated again by the upgrade scripts and as a results the existing grants got removed. Note; we are using a different user (not the schema user) to connect to the DB from Pega. We have overcome the issue by giving proper grants and privilege on the existing DB objects in the Data schema to the user. The server is up and running now.

Thanks all for the support.

We did not follow the resolution found in the article SA-3506 ( we did not move around store procs between rules and data shcema) instead whatever has been posted above by Ranesh was followed for data_schema.SPPR_SYS_UPDATESCACHE_GETTIME and the tomcat came up. 

We were able to login and view the Pega7 Designer Studio now.

Also, post upgrade only SPPR_REBUILD_INDEXES should have been present in rules_schema however in our case we found other OOTB store procs too. These extra store procs such as SPPC_DATA_UNIQUEID / SPPR_PURGE_TABLE / SPPR_POPULATE_INDEX_DATA ... were manually dropped from rules_schema.

After we performed the dataonly upgrade in QA; we restarted the tomcat.
We got this error and the tomcat did not come up...

com.pega.pegarules.pub.context.InvalidConfigurationException: No such Rule-Application instance: CUSTOM 01.01.01

Workarounds could be found in these articles and they helped us resolve this issue.

https://pdn.pega.com/support-articles/initializationfailederror-log-file-during-prpc-start
https://pdn.pega.com/support-articles/cannot-start-pega-instance