Question
5
Replies
1243
Views
Anthem
Posted: 2 years ago
Last activity: 1 year 7 months ago
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Hi Team,
We are unable to deploy the code to one of our environments because of the below mentioned errors.
PFA for the complete log.
build 13-Jul-2018 10:57:06 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 10:57:06 [java] at com.hazelcast.logging.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:166) build 13-Jul-2018 10:57:06 [java] at com.hazelcast.logging.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:158) build 13-Jul-2018 10:57:06 [java] at com.hazelcast.logging.AbstractLogger.finest(AbstractLogger.java:30) build 13-Jul-2018 10:57:08 [java] at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$MonitorInvocationsTask.run0(InvocationMonitor.java:286) build 13-Jul-2018 10:57:08 [java] at com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor$FixedRateMonitorTask.run(InvocationMonitor.java:257) build 13-Jul-2018 10:57:08 [java] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) build 13-Jul-2018 10:57:08 [java] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) build 13-Jul-2018 10:57:08 [java] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) build 13-Jul-2018 10:57:08 [java] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) build 13-Jul-2018 10:58:32 [java] Exception in thread "hz._hzInstance_1_65b6da5135c3ecccb0f77ce104466df8.SlowOperationDetectorThread" java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 10:59:10 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 10:59:10 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 10:59:43 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:01:03 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:01:03 [java] AsyncLogger error handling event seq=1468, value='Logger=com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor Level=FATAL Message=[30.138.136.231]:5701 [65b6da5135c3ecccb0f77ce104466df8] [3.8] GC overhead limit exceeded': build 13-Jul-2018 11:01:08 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:02:00 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:02 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] 2018-07-13 10:58:28,069 [ VDAASW702048] [ STANDARD] [ ] [ ] (pboard.StorageStreamCommonImpl) INFO - [Encoding - Error during encode - build 13-Jul-2018 11:03:19 [java] 2018-07-13 10:58:28,069 [ VDAASW702048] [ STANDARD] [ ] [ ] (pboard.StorageStreamCommonImpl) INFO - HISTORY-RULE RULE-PORTALSKIN ANTMCSFW #20180605T210813.637 GMT!20180713T145527.074 GMT build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] at com.hazelcast.nio.Packet.readValue(Packet.java:281) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.nio.Packet.readFrom(Packet.java:274) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.nio.tcp.MemberReadHandler.onRead(MemberReadHandler.java:58) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:161) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206) build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] Exception in thread "hz._hzInstance_1_65b6da5135c3ecccb0f77ce104466df8.IOBalancerThread" java.lang.RuntimeException: Unable to create engine instance - class path be wrong, or contain invalid contents build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.pub.PegaRULES.getEngine(PegaRULES.java:60) build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.priv.alert.AlertBuilder.ensureNodeSet(AlertBuilder.java:266) build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.priv.alert.AlertBuilder.withoutRequestor(AlertBuilder.java:127) build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.priv.LogHelper.alert(LogHelper.java:1768) build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.priv.AlertEvent$1.log(AlertEvent.java:82) build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.priv.LogHelper.doAlertException(LogHelper.java:1342) build 13-Jul-2018 11:03:19 [java] at com.pega.pegarules.priv.LogHelper.fatal(LogHelper.java:1312) build 13-Jul-2018 11:03:19 [java] at com.pega.platform.cluster.internal.util.HazelCastLogHelper.severe(HazelCastLogHelper.java:230) build 13-Jul-2018 11:03:19 [java] at com.pega.platform.cluster.internal.util.HazelCastLogHelper.log(HazelCastLogHelper.java:196) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.logging.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:168) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.logging.AbstractLogger.severe(AbstractLogger.java:75) build 13-Jul-2018 11:03:19 [java] at com.hazelcast.internal.networking.nonblocking.iobalancer.IOBalancerThread.run(IOBalancerThread.java:60) build 13-Jul-2018 11:03:19 [java] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] Exception in thread "PegaRULES LicenseDaemon" java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] Exception in thread "PegaRULES UsageDaemon" java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded build 13-Jul-2018 11:03:19 [java] AsyncLogger error handling event seq=1469, value='Logger=com.hazelcast.internal.networking.nonblocking.iobalancer.IOBalancer Level=FATAL Message=[30.138.136.231]:5701 [65b6da5135c3ecccb0f77ce104466df8] [3.8] IOBalancer failed': build 13-Jul-2018 11:03:19 [java] java.lang.OutOfMemoryError: GC overhead limit exceeded
Posted: 2 years ago
Posted: 2 years ago
Hi Harish,
Strangely,I dont have MaxHeapSize argument in the JVM's argument list.
PFB for the list of arguments.
-Dcatalina.home=xxx
-Dcatalina.base=xxx
-Djava.endorsed.dirs=xxx
-Djava.io.tmpdir=xxx
-Djavax.net.ssl.trustStoreType=xxx
-Djavax.net.ssl.trustStore=xxx
-Djavax.net.ssl.trustStorePassword=xxx
-Djsse.enableSNIExtension=false
-XX:HeapDumpPath=xxx
-Xloggc:xxx
-Xms8192m
-Xmx8192m
-XX:+UseG1GC
-XX:MaxGCPauseMillis=350
-XX:InitiatingHeapOccupancyPercent=45
-XX:ReservedCodeCacheSize=512m
-XX:+PrintAdaptiveSizePolicy
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+DisableExplicitGC
-XX:MetaspaceSize=512m
-XX:MaxMetaspaceSize=2048m
-Djava.awt.headless=true
-verbose:gc
-XX:+HeapDumpOnOutOfMemoryError
-XX:+PrintHeapAtGC
-XX:+ParallelRefProcEnabled
PFA for the latest log.
Posted: 2 years ago
Hello Pradeep,
Xmx8192m denotes that you have 8192MB of maximum Java Heap Size. I believe that "prpcUtilsTarget_Compass_CI.properties" is setting "XX:MaxHeapSize" to "1280m" and is overriding your JVM argument list.
Try locating and removing this setting from the file to resolve this.
Thank you,
Pawan
Posted: 1 year ago
hi,
Please check the prpcUtil.xml file. Pega has set the max limit. We can modify the same to our need.
Extract from the file-
<ac:then>
<condition property="jvm.arg.mem.max.temp" value="${jvm.arg.mem.max}" else="-Xmx1280m">
<isset property="jvm.arg.mem.max"/>
</condition>
<condition property="jvm.arg.mem.maxpermsize.temp" value="${jvm.arg.mem.maxpermsize}" else="-XX:MaxPermSize=512m">
<isset property="jvm.arg.mem.maxpermsize"/>
</condition>
<property name="jvm.args.temp" value="${jvm.arg.mem.max.temp} ${jvm.arg.mem.maxpermsize.temp}"/>
</ac:then>
<ac:else>
<condition property="jvm.arg.mem.max.temp" value="${jvm.arg.mem.max}" else="-XX:MaxHeapSize=1280m">
<isset property="jvm.arg.mem.max"/>
</condition>
<property name="jvm.args.temp" value="${jvm.arg.mem.max.temp}"/>
Thanks
Posted: 1 year ago
You could limit the GC time by including the JVM argument of -XX:+UseGCOverheadLimit.
In addition, I would also include a GC policy option in the JVM argument.
Reference:
https://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html
Hope it helps
Simon
Hi Pradeep,
It looks like the deployment of the archive zip file failed on the target Pega DB due to java.lang.OutOfMemoryError: GC overhead limit exceeded
This message means that for some reason the garbage collector is taking an excessive amount of time (by default 98% of all CPU time of the process) and recovers very little memory in each run (by default 2% of the heap).
This error is to notify server admins (to take corrective actions) to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.
For deployment using PRPCUtils embedded server initialization will be done followed by rap import. It looks like the max heap is specified as 1 gig for this process.
Try increasing it to higher heap size and retry the deployment. Let me know how it goes.
Hope this helps!