Wildfly 10.1消耗所有核心

我们最近将我们的银行应用程序从java 1.6升级到1.8,将jboss 4.x升级到wildfly 10.1。

我们观察到java消耗了机器上可用的所有核心(10)。

有人可以告诉是什么原因,通常使用jboss 4.x最大CPU利用率高达4核心。

我是否需要为垃圾收集配置任何东西?

下面是(添加进程消耗高CPU)的结果

ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep 3630

在此处输入图像描述

下面是每个消耗高CPU的LWP的hex

在此处输入图像描述

现在我检查了LWP 7914的线程转储

“default task-7”#182 prio = 5 os_prio = 0 tid = 0x00007f5c24033800 nid = 0x1c1a runnable [0x00007f5bb85e5000] java.lang.Thread.State:RUNNABLE at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:240)at io.undertow.server.HttpServerExchange $ WriteDispatchChannel.write(HttpServerExchange.java:2028)at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl .java:563)在i.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:216)的java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)java.io.BufferedOutputStream.write(BufferedOutputStream。 java:95) – 在javax.servlet.http.HttpServlet.service(HttpServlet)的com.eko.app.offlineKyc.servlet.KycPictureServlet.doGet(KycPictureServlet.java:58)上锁定(一个java.io.BufferedOutputStream) .java:687)在javax.servlet.http.HttpServl et.service(HttpServlet.java:790)at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:129) at com.eko.framework.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:48)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter (FilterHandler.java:131)位于io.undertow.servlet的io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)的com.eko.framework.URLSessionFilter.doFilter(URLSessionFilter.java:38)。 handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)atg.mifos.framework.security.util.FileNameFilter.doFilter(FileNameFilter.java:59)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter。 java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterHan dler.handleRequest(FilterHandler.java:84)at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)在oro.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.security .SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43 )at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)at io.undertow.security.handlers.AbstractC onfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java: 59)在io.undertow.security.handlers.handlers上的io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) .NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at org.wildfly.extension.undertow.security.jacc.JACCContex 在Io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)的io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)上的tIdHandler.handleRequest(JACCContextIdHandler.java:61) .undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)at io.undertow.servlet.handlers.ServletInitialHandler.access $ 100(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 2.call (ServletInitialHandler.java:138)io.undertow.servlet.handlers.ServletInitialHandler $ 2.call(ServletInitialHandler.java:135)at io.undertow.servlet.core.ServletRequestContextThreadSetupAction $ 1.call(ServletRequestContextThreadSetupAction.java:48)at io。 undero.servlet.core.ContextClassLoaderSetupAction $ 1.call(ContextClassLoaderSetupAction.java:43)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionW 说唱者$ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44 )ato.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)at io.undertow.servlet.handlers.ServletInitialHandler .access $ 000(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 1.handleRequest(ServletInitialHandler.java:104)at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)at io。在java.util.concurrent.ThreadPoolExecutor的java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)上的undertow.server.HttpServerExchange $ 1.run(HttpServerExchange.java:805)$ Worker.run(ThreadPoolExecutor.java:617 )在java.lang.Thr ead.run(Thread.java:745)

对于LWP 7249

“default task-27”#231 prio = 5 os_prio = 0 tid = 0x00007f5c2401c000 nid = 0x1c51 runnable [0x00007f5ca9a63000] java.lang.Thread.State:RUNNABLE at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:158) at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:179)位于org.xnio.channels.Channels.writeBlocking(Channels.java)的io.undertow.server.HttpServerExchange $ WriteDispatchChannel.write(HttpServerExchange.java:1993) :152)ato.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:283)at io.undertow.io.BlockingSenderImpl.writeBuffer(BlockingSenderImpl.java:190)at io.undertow.io.BlockingSenderImpl.send( BlockingSenderImpl.java:71)ato.undertow.server.handlers.resource.CachedResource.serve(CachedResource.java:187)at io.undertow.servlet.handlers.DefaultServlet.serveFileBlocking(DefaultServlet.java:332)at io.undertow javax.servlet.http.Htt中的.servlet.handlers.DefaultServlet.doGet(DefaultServlet.java:180) 位于io.undertow的io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)的javax.servlet.http.HttpServlet.service(HttpServlet.java:790)中的pServlet.service(HttpServlet.java:687)位于com.eko.framework.MifosNGFilter.doNormal(MifosNGFilter.java:283)的.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:129)at com.eko.framework.MifosNGFilter.doFilter(MifosNGFilter.java:137) )at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at com.eko.framework.CrossScriptingFilter的io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)。 doFilter(CrossScriptingFilter.java:48)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at com .eko.framework.URLSessionFilter.doFilter(URLSessionFilter.java:38)位于io.undertow的io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)。 servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)atg.mifos.framework.security.util.FileNameFilter.doFilter(FileNameFilter.java:59)at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61)ato.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)at io.undertow位于org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest的io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)中的.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) (SecurityContextAssociationHandler.java:78)io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)at io .undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest (AuthenticationConstraintHandler.java:53)at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)at io。在io.undertow.security.handle.Auhertow.servlet.handler.security.CallAuthenticatedSessionHandler的io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)上的了underow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)。在io.undertow.security上的handleRequest(CachedAuthenticatedSessionHandler.java:77) .handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) )ato.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)at io.undertow.server.handler.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.server。 Handler.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)at io.undertow.servlet.handlers.ServletInitialHandler.access $ 100(ServletInitialHandler.java:81 )io.undertow.servlet.handlers.ServletInitialHandler $ 2.call(ServletInitialHandler.java:138)at io.udertow.servlet.handlers.ServletInitialHandler $ 2.call(ServletInitialHandler.java:135)at io.u ndertow.servlet.core.ServletRequestContextThreadSetupAction $ 1.call(ServletRequestContextThreadSetupAction.java:48)at io.undertow.servlet.core.ContextClassLoaderSetupAction $ 1.call(ContextClassLoaderSetupAction.java:43)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call (LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io。 undero.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( ServletInitialHandler.java:272)at io.undertow.servlet.handlers.ServletInitialHandler.access $ 000(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 1.ha ndleRequest(ServletInitialHandler.java:104)位于io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)的io.undertow.server.HttpServerExchange $ 1.run(HttpServerExchange.java:805),位于java.util.concurrent。 ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)

下面是完成线程转储线程转储的链接

服务器不稳定,所以我们不得不恢复到jdk 1.6和jboss 4.我想知道jboss 4一切正常,现在我无法确定问题出在哪里,使用wildfly设置,或者我的应用程序或GC设置。

线程转储分析的链接线程转储分析结果

代码KycPictureServlet

@Override public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { ServletOutputStream out = null; FileInputStream fin = null; BufferedInputStream bin = null; BufferedOutputStream bout = null; try { String filename = String.valueOf(request.getParameter("fileName")); String folderPathId = String.valueOf(request.getParameter("folderPathId")); String kycDocPath = ""; System.out.println("Folder Path : " + folderPathId + " \t" + filename); if (folderPathId != null && !folderPathId.equals("null")) { kycDocPath = new StoragePathService().getStoragePathByPathID(Integer.parseInt(folderPathId)).getFolderPath(); } else { kycDocPath = new StoragePathService().getStoragePathByPathID(EkoDBConstants.KYC_FILE_UPLOAD_PATH).getFolderPath(); } String uploadFolderPath = kycDocPath; final String filetofetch = uploadFolderPath + filename; if (filetofetch.toLowerCase().endsWith("pdf")) { response.setContentType("application/pdf"); // response.setHeader("Content-Disposition", "inline; // filename=\"" + filetofetch + "\""); } else if (filetofetch.toLowerCase().endsWith("tif") || filetofetch.endsWith("tiff")) { response.setContentType("image/tiff"); } else { response.setContentType("image/jpeg"); } File uploadFolder = new File(filetofetch); if(uploadFolder != null && uploadFolder.exists()){ out = response.getOutputStream(); fin = new FileInputStream(filetofetch); bin = new BufferedInputStream(fin); bout = new BufferedOutputStream(out); int ch = 0; ; while ((ch = bin.read()) != -1) { bout.write(ch); } } } catch (Exception e) { e.printStackTrace(); } finally { if (bin != null) { bin.close(); } if (fin != null) { fin.close(); } if (bout != null) { bout.close(); } if (out != null) { out.close(); } } } 

以前没有任何finally块,所以如果exception流没有关闭,第二件事我添加了额外的检查以检查文件是否存在然后只打开流。

进行更改后,不再有线程卡在该servlet上。

我已经采取了最新的转储,发现一些奇怪的进程消耗cpu而没有结束

 "C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ff8dc231000 nid=0x11b6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ff8dc22e800 nid=0x11b5 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff8dc22d000 nid=0x11b4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff8dc22a000 nid=0x11b3 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE 

我还配置了从OLD Jboss 4复制的GC设置

在线程转储中,我发现了一些消耗CPU但没有结束的LWP

 "VM Thread" os_prio=0 tid=0x00007ff8dc1e9000 nid=0x11B411af runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff8dc01e800 nid=0x11a6 runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ff8dc020800 nid=0x11a7 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ff8dc022000 nid=0x11a8 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ff8dc024000 nid=0x11a9 runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ff8dc026000 nid=0x11aa runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ff8dc027800 nid=0x11ab runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ff8dc029800 nid=0x11ac runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ff8dc02b800 nid=0x11ad runnable "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007ff8dc02d000 nid=0x11ae runnable 

内存和GC设置

 JAVA_OPTS="-Xms4G -Xmx12G -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -XX:-UseConcMarkSweepGC -XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/data/eko/GC.log -verbose:gc -XX:NewRatio=2" 

可悲的是,我的魔法水晶球被打破了,所以我们必须用甩尾来弄脏手。
首先,我们需要弄清楚,什么线程导致Cpu负载,因为app服务器不是单线程的。 假设您使用的是linux,您可以使用标准ps来获取每个进程线程的cpu使用情况的详细信息:

ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep [PID_OF_YOUR_WILDFLY_PROCESS] > unixthread.txt

这里我们感兴趣的是列nlwp lwp和pcpu。 找到具有最高CPU负载的线程。
接下来我们需要java线程转储:在高CPU负载期间获取线程转储:

jstack -l [PID_OF_YOUR_WILDFLY_PROCESS] > jstack.out

现在对于一些线程搜索:说你的unixthread识别出一个以nwlp 9999作为CPU占用的线程,将数字转换为hex-0x270F并在你的jstack.out文件中搜索这个值。 它应该与java线程匹配,stacktrace应该可以帮助您查明问题。

如果它与GC相关,则继续堆转储以确保它不是您的应用程序中的泄漏。
如果您进行分析并仍然认为这是jvm或gc设置的问题,您应该与redhat伙伴联系。 我相信,如果你打算在RedHat堆栈上运行一个银行应用程序,那么你肯定应该支付带有付费支持的EAP7,而不是支持Upstream OS项目的hazzard,而不支持最终的错误或问题。