<< Back to previous view

[WSIT-1260] memory leak from ws security Created: 21/Sep/09  Updated: 28/Feb/12  Resolved: 04/Oct/10

Status: Resolved
Project: wsit
Component/s: security
Affects Version/s: 2.0
Fix Version/s: 2.1

Type: Bug Priority: Minor
Reporter: geturnerlmco Assignee: kumarjayanti
Resolution: Fixed Votes: 0
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: All
Platform: All

File Attachments: Zip Archive MetroLeakTest.zip     Text File thread-dump-filtered.txt    
Issuezilla Id: 1,260
Tags: metro2_0-waived
Participants: geturnerlmco, geturnerlmco, kumarjayanti, kumarjayanti, Nithya Ramakrishnan and wap


This issue was discovered while performing leak testing using JProbe 8.1. The
leak is from security classes that are allocated from using a WebServiceRef in
an EJB or servlet. The leak can be partially cleanup up by calling the close
method of the Closeable interface of the port type, but this is only a
documented feature of the Reliable Messaging handling. This has been discussed
with Vbkumar.Jayanti@Sun.COM. I have created a test case that uses a servlet
to trigger the leak (https://localhost:8181/MetroLeakTest/test?LEAK=no, or
LEAK=yes) will either call the close method, or not. The servlet class must be
updated with the correct constants for the local server.

Comment by geturnerlmco [ 21/Sep/09 02:22 PM ]

Created an attachment (id=1022)
EAR source to show wss leak

Comment by kumarjayanti [ 24/Sep/09 01:37 AM ]


Comment by kumarjayanti [ 02/Nov/09 02:00 AM ]

While i am working on the fix, as indicated in the bug report :

calling ((com.sun.xml.ws.Closeable)proxy).close() once you are done with the
Proxy is a workaround for this problem.

Comment by kumarjayanti [ 02/Nov/09 11:03 PM ]

The possible solutions for this bug will be discussed on 5th Nov, thursday in
Tango Tech meeting. I have been asked in the meanwhile to update this bug in
view of Nov 3rd HCF. The priority will be brought back to original post Nov 3rd.

Comment by kumarjayanti [ 04/Oct/10 12:54 AM ]

cleaned up the code to not use a tube reference as the key to the hashmap. Made use of weakreferences at
a few other places.

Comment by kumarjayanti [ 04/Oct/10 12:57 AM ]

cleaned up the code to not use a tube reference as the key to the hashmap. Made use of weakreferences at
a few other places.

Comment by wap [ 07/Dec/11 01:20 PM ]

I think, this issue is not fixed. Neither in version 2.1 nor in 2.1.1 the memory is cleaned up correctly without the workaround invoking close() explicitly. The hashtable in the IssuedTokenManager remains growing until the OOME occurs.

The workaround has a another issue, which comes up, if close fails de to violation of the security policy, e.g. clock skew to big, an exception occurs, before the token manager could be cleaned. In this case, the memory continues growing, and I even saw some input streams remaining open and a blocking thread.

I think, close() has to handle such situations and clean up, as I do not see any chance to recover from that situation. Even better, as the OP suggests, the usage of close() should be entirely unnecessary.

Comment by wap [ 07/Dec/11 01:23 PM ]

A stuck thread that waits forever on an input stream.

Comment by wap [ 24/Feb/12 09:53 AM ]

The leak is fixed for me in Metro 2.2-b13. However, the workaround with invocation of Closable.close() throws an NPE now. But this is only a very minor issues, as the client should not invoke it at all.

Comment by Nithya Ramakrishnan [ 24/Feb/12 10:29 AM ]

Could you please provide the stack trace ? Also would the previously attached test case help to reproduce the error?


Comment by geturnerlmco [ 27/Feb/12 04:11 PM ]

There was no stack trace, it was a memory leak. All of the information to reproduce the problem is in the original filing. Thanks.

Comment by kumarjayanti [ 28/Feb/12 09:38 AM ]

You mentioned the workaround with invocation of Closable.close() throws an NPE now, so we are asking what is the stack trace. Otherwise we understand the issue is fixed ?.

[JERSEY-1628] Payloads are not logged in POST methods Created: 17/Dec/12  Updated: 06/Nov/13  Resolved: 06/Nov/13

Status: Resolved
Project: jersey
Component/s: core
Affects Version/s: 1.12
Fix Version/s: 1.18

Type: Bug Priority: Major
Reporter: geturnerlmco Assignee: Michal Gajdos
Resolution: Cannot Reproduce Votes: 0
Remaining Estimate: 0 minutes
Time Spent: 1 hour
Original Estimate: 3 hours


Participants: geturnerlmco and Michal Gajdos


The HTTP header information is logged but not the payload. In our case, we are using XML, and the XML is not logged.

Comment by geturnerlmco [ 17/Dec/12 07:57 PM ]

If forgot to add that a LoggingFilter is configured in the web.xml

Comment by Michal Gajdos [ 06/Nov/13 12:29 PM ]

Cannot reproduce with Jersey 1.18-SNAPSHOT:

Nov 06, 2013 1:26:27 PM com.sun.jersey.api.container.filter.LoggingFilter filter
INFO: 3 * Server in-bound request
3 > POST http://localhost:9998/jaxb/XmlRootElement
3 > user-agent: curl/7.21.4 (x86_64-apple-darwin10.7.3) libcurl/7.21.4 OpenSSL/0.9.7l zlib/1.2.5
3 > host: localhost:9998
3 > accept: */*
3 > content-type: application/xml
3 > content-length: 72
3 >
<jaxbXmlRootElement><value>xml root element</value></jaxbXmlRootElement>

Nov 06, 2013 1:26:27 PM com.sun.jersey.api.container.filter.LoggingFilter$Adapter finish
INFO: 3 * Server out-bound response
3 < 200
3 < Content-Type: application/xml
3 <
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><jaxbXmlRootElement><value>xml root element</value></jaxbXmlRootElement>
Comment by Michal Gajdos [ 06/Nov/13 12:30 PM ]

Resolving as Cannot Reproduce. Feel free to re-open at any time with your test-case (if you're still experiencing the issue).

[GLASSFISH-16937] Having REST services in separate WARs in a single EAR prints classloading warnings Created: 01/Jul/11  Updated: 17/Mar/14

Status: Open
Project: glassfish
Component/s: web_container
Affects Version/s: 3.1
Fix Version/s: 4.0.1

Type: Bug Priority: Minor
Reporter: mmuller Assignee: Daniel
Resolution: Unresolved Votes: 10
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Windows Server 2008

File Attachments: File ear-1.0.ear     GZip Archive jaxrs_multimodule_test.tar.gz    
Tags: jax-rs glassfish-3-1 classloader
Participants: andrey.v.markelov, Dan.Daneasa, Daniel, geturnerlmco, hpgisler, mmuller, nabizamani, obfischer, pettymt, phealy, Shing Wai Chan and yonatan


Warnings show up when an EAR is deployed, containing more than one WAR archove with REST web services. Looks like GlassFish/Jersey tries to load all the REST service classes for each of the WARs being deployed, and then shows classloader warnings:

WEB9052: Unable to load class <classname>, reason: java.lang.ClassNotFoundException: <classname>

I did a minimal project which produces an EAR with this structure:


`- application.xml
  • war-1-1.0.war
  • classes/
  `- test/war1/Service1.class
`- web.xml
`- war-2-1.0.war
  • classes/
`- test/war2/Service2.class
`- web.xml

(See the attached maven project).

test.war1.Service1 and test.war2.Service2 are POJOs with class-level @Path annotation and method-level @GET or @POST annotations.
The application.xml DD is generated by maven-ear-plugin and only contains the two webmodules and their contextroots.
Both web.xml only contain a display name and the Jersey ServletContainer loaded on startup.

When deploying to GlassFish 3.1 on Windows Server 2008, the log contains the following entries:

[#|2011-07-01T10:45:59.159+0200|WARNING|glassfish3.1|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=183;_ThreadName=Thread-1;|WEB9052: Unable to load class test.war2.Service2, reason: java.lang.ClassNotFoundException: test.war2.Service2|#]

[#|2011-07-01T10:45:59.187+0200|INFO|glassfish3.1|com.sun.jersey.api.core.WebAppResourceConfig|_ThreadID=23;_ThreadName=Thread-1;|Scanning for root resource and provider classes in the Web app resource paths:

[#|2011-07-01T10:45:59.187+0200|INFO|glassfish3.1|com.sun.jersey.api.core.ScanningResourceConfig|_ThreadID=23;_ThreadName=Thread-1;|Root resource classes found:
class test.war1.Service1|#]

[#|2011-07-01T10:45:59.187+0200|INFO|glassfish3.1|com.sun.jersey.api.core.ScanningResourceConfig|_ThreadID=23;_ThreadName=Thread-1;|No provider classes found.|#]

[#|2011-07-01T10:45:59.187+0200|INFO|glassfish3.1|com.sun.jersey.server.impl.application.WebApplicationImpl|_ThreadID=23;_ThreadName=Thread-1;|Initiating Jersey application, version 'Jersey: 1.5 01/14/2011 12:36 PM'|#]

[#|2011-07-01T10:45:59.354+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=183;_ThreadName=Thread-1;|WEB0671: Loading application ear-1.0#war-1-1.0.war at [/war1]|#]

[#|2011-07-01T10:45:59.368+0200|WARNING|glassfish3.1|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=183;_ThreadName=Thread-1;|WEB9052: Unable to load class test.war1.Service1, reason: java.lang.ClassNotFoundException: test.war1.Service1|#]

[#|2011-07-01T10:45:59.382+0200|INFO|glassfish3.1|com.sun.jersey.api.core.WebAppResourceConfig|_ThreadID=23;_ThreadName=Thread-1;|Scanning for root resource and provider classes in the Web app resource paths:

[#|2011-07-01T10:45:59.382+0200|INFO|glassfish3.1|com.sun.jersey.api.core.ScanningResourceConfig|_ThreadID=23;_ThreadName=Thread-1;|Root resource classes found:
class test.war2.Service2|#]

[#|2011-07-01T10:45:59.382+0200|INFO|glassfish3.1|com.sun.jersey.api.core.ScanningResourceConfig|_ThreadID=23;_ThreadName=Thread-1;|No provider classes found.|#]

[#|2011-07-01T10:45:59.382+0200|INFO|glassfish3.1|com.sun.jersey.server.impl.application.WebApplicationImpl|_ThreadID=23;_ThreadName=Thread-1;|Initiating Jersey application, version 'Jersey: 1.5 01/14/2011 12:36 PM'|#]

[#|2011-07-01T10:45:59.521+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=183;_ThreadName=Thread-1;|WEB0671: Loading application ear-1.0#war-2-1.0.war at [/war2]|#]

[#|2011-07-01T10:45:59.535+0200|INFO|glassfish3.1|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=183;_ThreadName=Thread-1;|ear-1.0 was successfully deployed in 543 milliseconds.|#]

So this looks like Glassfish tries to load both services for each WAR module. When loading war-1, it cannot load test.war2.Service2, and while loading war-2, it cannot load test.war1.Service1...
Although, the application is correctly deployed and all the services do work !!

I guess this is a tiny bug, and could be fixed within a handful of lines. I could try to fix it, if someone could point me to the right module producing this warning.

Comment by mmuller [ 01/Jul/11 09:02 AM ]

Oops, looks like Jira didn't like my folder-tree syntax.

So, the EAR structure is

Comment by Shing Wai Chan [ 01/Jul/11 10:08 PM ]

The calling stack is as follows:
--> ServletContainerInitializerUtil#getInitializerList
--> ServletContainerInitializerUtil#checkAgainstInterestList

and the warning is coming from calling the checkAgainstInterestList method

The issue is from argument of calling the last method as follows:
1. Types classInfo contains the classes information for the ear
which is from WebModule#getTypes
Types.class.getName(), Types.class);
2. WebappClassLoader cl, for war1, say, then it cannot load classes for war2

Even though the returned value of the method is correct, there is unnecessary call for loading other classes.

Comment by geturnerlmco [ 13/Oct/11 10:32 PM ]

This happens if there are multiple WARs, period. I have an application with 5 WARs, two are SOAP service endpoints and 3 are REST. If I remove 2 of the 3 REST WARS, I still get the WARNINGS. If I remove the 2 SOAP WARs, the WARNINGS are gone. I have searched every forum for an answer to this annoying problem, but this is the first one that is the closest to my problem. If anyone has a workaround other than destroying my EAR structure (which doesn't work well for fielding, ha ha) I would really appreciate knowing it.

Comment by geturnerlmco [ 14/Oct/11 02:45 AM ]

One additional comment that I thought would be helpful. I commented out all of the additional WARs from my application.xml but still built the EAR normally. The WARNINGS occured, just because the modules existed in the EAR directory structure, but the module definitions were commented out! Is the class scanner looking for ANY .class file in the EAR directory, whether it was defined to be part of the application or not???

Comment by Dan.Daneasa [ 23/Oct/11 10:32 AM ]

I have more comments on this.
I have 2 wars deployed in the same ear. One is a Jersey simple west app, the other one is an empty war.
I can see the ClassNotFoundExceptions.

There may be more to this problem. If i have an ear with 2 wars, one is a simple Jersey rest app, the other a simple Mojarra faces app. I see that the ClassNotFoundException is still present, also there are Mojarra-impl ClassNotFoundException warnings.
The same happens if i try to integrate MyFaces instead of Mojarra in the faces webapp.
Mojarra/Myfaces tries to load in the Jersey app as well, and vice-versa, Jersey tries to load in the Mojarra/MyFaces app.

If I have only one of the wars in the ear in any of the above combinations then the warnings are not present.

Comment by hpgisler [ 14/Nov/11 12:46 PM ]

Just for the record: Same problem here.

Two WAR's in an EAR
1. WAR JSF frontend
2. WAR REST frontend

If packaging only one of them into EAR, no problem; if packaging both inside EAR, Problem.

Comment by pettymt [ 10/Apr/12 03:55 AM ]

Confirmed on v3.1.1 (build 12)

30 WARs(each containing a REST service) in an EAR is a lot of warnings.

Comment by phealy [ 27/Mar/13 02:47 AM ]

I am getting a warning for every PimesFaces class every time I deploy (because I also have a Jersey app in my EAR). This does not reflect well on Glassfish.

Comment by nabizamani [ 13/Apr/13 12:30 PM ]

I have the same issue with GlassFish (build 5) when deploying an EAR containing 2 wars and 1 ejb module (one of the wars contains restful web services).

WARNING: WEB9052: Unable to load class com.demo.service.exception.RestExceptionCatcher, reason: java.lang.ClassNotFoundException: com.demo.service.exception.RestExceptionCatcher
WARNING: WEB9052: Unable to load class com.demo.service.Order, reason: java.lang.ClassNotFoundException: com.demo.service.Order

RestExceptionCatcher is a @Provider and implements ExceptionMapper<Throwable>.
Order is a very simple REST service.

These warnings are very frustrating and I really want to get rid of them! But that would means I cannot use my EAR

Comment by yonatan [ 01/May/13 07:51 AM ]

If everything works properly, until the issue is resolved you can raise the logging level of that specific logger by adding javax.enterprise.system.container.web.org.glassfish.web.loader.level=SEVERE to the DOMAIN_HOME/config/logging.properties file.

Comment by andrey.v.markelov [ 07/May/13 06:20 AM ]

I have got the same trouble. Are you going to fix that?

Comment by Shing Wai Chan [ 25/Jun/13 06:29 PM ]

The given test case may need to update for GlassFish 4.

Comment by Daniel [ 25/Jun/13 10:54 PM ]

The web.xml for GF4 should use org.glassfish.jersey.servlet.ServletContainer API instead.

Comment by obfischer [ 17/Mar/14 10:02 PM ]

Is there a chance to get this fixed in the near future? Our monitoring reports all log messages with a log level above INFO. It is very annoying to warnings for a non-existent problem.

Generated at Fri Apr 18 22:55:49 UTC 2014 using JIRA 4.0.2#472.