Question

1
Replies
197
Views
GerardoA Member since 2012 2 posts
Rulesware
Posted: 5 months ago
Last activity: 5 months 1 week ago

Pega 8.4 fresh install, Crazy amount of errors on catalina.out, eating RAM whats wrong?

Whats happening

Fresh install Pega 8.4(single Universal node, 2vms one postgres one tomcat), left untouched for couple of hours, afterwards it starts to go crazy on the catalina.out logs, also Postgres connections starts to drop, memory starts rising beyond Xmx, swap space is also used.

Have try several things in the connectors, DB, nothing seem to fix the issue

 

 Errors Warning, Stacktraces Catalina.out
*2020-04-09 23:09:55,117 [ow-Pulse-Executor:32] [  STANDARD] [                    ] [                    ] (ce.run.ThreadCompletionHandler) ERROR   - Failed to finalize data flow thread for run [pyBIBatchIndexProcessor]
*[ow-Pulse-Executor:32] [  STANDARD] [                    ] [                    ] (ce.run.ThreadCompletionHandler) ERROR   - Failed to finalize data flow thread for run [pyBIIncrementalIndexer] java.lang.NoClassDefFoundError: com/pega/dsm/dnode/impl/dataflow/service/run/ThreadCompletionHandler$1
*

AsyncLogger error handling event seq=306053, value='Logger=com.pega.dsm.dnode.impl.dataflow.service.run.ThreadCompletionHandler Level=ERROR Message=Failed to finalize data flow thread for run [pyBIIncrementalIndexer]': java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/pattern/NotANumber 09-Apr-2020 23:09:55.121 WARNING [charlatan-watcher:7] org.apache.tomcat.jdbc.pool.PooledConnection.clearWarnings Unable to clear Warnings, connection will be closed 

org.postgresql.util.PSQLException: This connection has been closed.

*[ow-Pulse-Executor:32] [  STANDARD] [                    ] [                    ] (.DataFlowServicePulseOperation) ERROR   - Error while exec uting pulse task: ThreadFinalizationTask java.lang.reflect.UndeclaredThrowableException: null
*Caused by: com.pega.pegarules.data.internal.rd.queryexec.ExecuteQueryException: An error occured on executing the query for the report definition - There was a problem getting a list: code: 0 SQLState: 08003 Message: This connection has been closed. DatabaseException caused by prior exception: org.postgresql.util.PSQLException: This connection has been closed.  | SQL Code: 0 | SQL State: 08003
*

Caused by: com.pega.pegarules.pub.database.ConnectionException: There was a problem getting a list: code: 0 SQLState: 08003 Message: This connection has been closed. DatabaseException caused by prior exception: org.postgresql.util.PSQLException: This connection has been closed.  | SQL Code: 0 | SQL State: 08003

        at com.pega.pegarules.data.internal.access.ExceptionInformation.createAppropriateExceptionDueToDBFailure(ExceptionInformation.java:378) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:363) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.Lister.convertSqlExceptionToDatabaseException(Lister.java:607) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.Lister.listWithResultPackager(Lister.java:427) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.Lister.list(Lister.java:190) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:3006) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:2997) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:98) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:75) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.queryexec.NativeSQLQueryExecutor.executeQuery(NativeSQLQueryExecutor.java:51) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.executeQuery(NativeSQLListExecutor.java:130) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.executeSql(NativeSQLListExecutor.java:112) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.execute(NativeSQLListExecutor.java:82) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.sqlapi.exec.NativeSQLListExecutor.execute(NativeSQLListExecutor.java:52) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.PRDataProviderImpl.executeNativeSQL(PRDataProviderImpl.java:1641) ~[prprivate-data.jar:?]         at com.pega.pegarules.data.internal.PRDataProviderForModulesImpl.executeNativeSQL(PRDataProviderForModulesImpl.java:137) ~[prprivate-data.jar:?]         at com.pega.platform.serviceregistry.internal.database.persistence.PlatformKeyValueStorePersistence.selectKey(PlatformKeyValueStorePersistence.java:147) ~[service-registry.jar:?]         at com.pega.platform.serviceregistry.internal.database.persistence.PlatformKeyValueStorePersistence.get(PlatformKeyValueStorePersistence.java:88) ~[service-

*WARNING [charlatan-watcher:7] org.apache.tomcat.jdbc.pool.PooledConnection.clearWarnings Unable to clear Warnings, connection will be closed.         org.postgresql.util.PSQLException: This connection has been closed.

SPECS:

Single node Pega 8.4 NodeType(Universal) on 2 GCP vms:  

 

Specs
 TomcatPostgres
Version9.0.3311
Specs2 CPU 14 RAM2CPU 7.5 RAM

 

Postgres
postgresql.confmax_connections = 150  shared_buffers = 1792MB  effective_cache_size = 5376MB  maintenance_work_mem = 448MB  checkpoint_completion_target = 0.9  wal_buffers = 16MB  default_statistics_target = 100  random_page_cost = 1.1  effective_io_concurrency = 200  work_mem = 6116kB  min_wal_size = 1GB  max_wal_size = 4GB  max_worker_processes = 2  max_parallel_workers_per_gather = 1  max_parallel_workers = 2  max_parallel_maintenance_workers = 1
Tomcat
Relevant JVM_OPTS

-Xms4G -Xmx8G  -XX:MaxMetaspaceSize=768m  -Djava.security.egd=file:/dev/./urandom

catalina.out

about 2G and growing
contex.xml <Resource name="jdbc/PegaRULES"                  auth="Container"                  type="javax.sql.DataSource"                  factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"                  driverClassName="org.postgresql.Driver"                  url="jdbc:postgresql://${db.ip}:5432/${db.name}"                  maxTotal="100"                 maxIdle="30"                 maxWaitMillis="10000"         />         <Resource name="jdbc/AdminPegaRULES"                  auth="Container"                  type="javax.sql.DataSource"                  factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"                  driverClassName="org.postgresql.Driver"                  url="jdbc:postgresql://${db.ip}:5432/${db.name}"                  maxTotal="100"                 maxIdle="30"                 maxWaitMillis="10000"         />
server.xml<Connector port="8080" protocol="HTTP/1.1"                connectionTimeout="20000"                />

Don't know what is wrong with this setup

Any ideas on how to fix this ? What would be the optimal parameters for a single node for a max of 5 users 

like JVM options, server and context tomcat conf files

Postgres optimal defaults, etc.

Thanks

 

 

 

 

 

 

 

 

Pega Platform 8.4 System Administration Financial Services System Administrator Senior System Architect
Share this page LinkedIn