[GLASSFISH-18360] Update tool configuration breaks default network behaviour Created: 14/Feb/12  Updated: 02/Nov/13

Status: Open
Project: glassfish
Component/s: update_center
Affects Version/s: 3.1.1_b12
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: thehpi Assignee: Snjezana Sevo-Zenzerovic
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux debian squeeze (6.0.4). Running my web applications in the same jvm as the DAS


Tags: 3_1_2-exclude, 3_1_2-next

 Description   

The update tool creates at some point a .updatetool directory with a init.cfg file.
The init.cfg file contains (after default installation and executing latest updates)

[main]
date: 1328646278464
optin.update.notification: true
image_list: /home/glassfish_prod/glassfish312/bin/..
[network]
proxy.use.system: true
proxy.required: false

The setting

proxy.use.system: true

Is used by the class

com.sun.pkg.client.SystemInfo

to set the following system property

java.net.useSystemProxies=true

This class is executed when glassfish starts up.

This setting overrides the setting in $

{JAVA_HOME}/jre/lib/net.properties

The actual problem I'm having is that the jvm (read glassfish) will crash when I
excecute several threads which call an external web service.

The crash dump shows (partial)

Stack: [0x00007fd589d79000,0x00007fd589e7a000], sp=0x00007fd589e71578, free space=993k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libdbus-1.so.3+0x28de0] dbus_malloc+0xa0

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j sun.net.spi.DefaultProxySelector.getSystemProxy(Ljava/lang/String;Ljava/lang/String;)Ljava/net/Proxy;+0
j sun.net.spi.DefaultProxySelector.access00(Lsun/net/spi/DefaultProxySelector;Ljava/lang/String;Ljava/lang/String;)Ljava/net/Proxy;+3
j sun.net.spi.DefaultProxySelector.run()Ljava/net/Proxy;+151
j sun.net.spi.DefaultProxySelector.run()Ljava/lang/Object;+1
v ~StubRoutines::call_stub
J java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
j sun.net.spi.DefaultProxySelector.select(Ljava/net/URI;)Ljava/util/List;+223
j sun.net.www.protocol.http.HttpURLConnection.plainConnect()V+314
j sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect()V+9
j sun.net.www.protocol.http.HttpURLConnection.getOutputStream()Ljava/io/OutputStream;+134
j sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream()Ljava/io/OutputStream;+4
j com.sun.xml.ws.mex.client.HttpPoster.post(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/io/InputStream;+60

The DefaultProxySelector (see top of stack) contains the following static block

static {
final String key = "java.net.useSystemProxies";
Boolean b = (Boolean) AccessController.doPrivileged(
new PrivilegedAction() {
public Object run() {
return NetProperties.getBoolean(key);
}});
if (b != null && b.booleanValue()) { java.security.AccessController.doPrivileged( new sun.security.action.LoadLibraryAction("net")); hasSystemProxies = init(); }

This will set the hasSystemProxies boolean only when the java.net.useSystemProxies system property is true.
The hasSystemProxies boolean is checked in the select method and only if it is true it will call the getSystemProxy
method which will crash the jvm (when multiple threads do so).

So actually, there probably is a concurrency bug/problem with the libdbus native library which causes the actual crash.

But: I expect the system proxy not to be checked (as specified by ${JAVA_HOME}

/jre/lib/net.properties)
Update tool is being initialized when glassfish starts up and the proxy.use.system=true setting will cause
the system property java.net.useSystemProxies to be set to true which effectively causes the problem.



 Comments   
Comment by Bobby Bissett [ 14/Feb/12 ]

Moving to 'update center' category. I know the categories are confusing, but 'upgrade tool' is for asupgrade which acts on a domain so that an older domain can be used with a newer GF installation.

Comment by Joe Di Pol [ 17/Feb/12 ]

Too late for 3.1.2. Tagging to revisit for next release.

Comment by kovica [ 19/Jul/12 ]

As a workaround you could make java use direct connection instead of trying system proxies.
Do this:

  • cd $GLASSFISH_INSTALL_DIR/bin
  • edit file pkg and search for line
    echo "proxy.use.system=true" >> "$BOOTSTRAPPROPS"
  • change it to:
    echo "proxy.use.system=false" >> "$BOOTSTRAPPROPS"

Do the same with updatetool file if you want to install the graphical GUI to update tool

Comment by thehpi [ 19/Jul/12 ]

The pkg file does not contain this line. The pkg.bat file however does.
I did find this line in the file $GLASSFISH_INSTALL_DIR/pkg/lib/pkg-bootstub.sh
So it looks like this is the file that needs the fix.

Comment by kovica [ 19/Jul/12 ]

Sorry, I don't point out that I'm using glassfish-3.1.2.zip as the installation file.

Comment by ljnelson [ 02/Nov/13 ]

Hello; this bug appears to be related to research done as part of https://java.net/jira/browse/GLASSFISH-12213. It affects GlassFish 3.1.2.2 as well.

Notes for posterity and non-GlassFish-team people:

The Java pkg client has a class called com.sun.pkg.client.SystemInfo. At some point GlassFish must cause this class to get loaded at startup.

When this class loads, it ends up having its loadProxyInfo() method called.

This method looks for either an init.cfg or a defaults.cfg file in various locations. One of those locations is (on my Mac, as an example) /Users/ljnelson/Library/Application Support/updatetool.

If it finds such a file, and if that file has a property setting of:

proxy.use.system = true

...then SystemInfo will call System.setProperty("java.net.useSystemProxies", "true").

This has a couple of side effects.

The first side effect is that it overrides whatever is present in $JAVA_HOME/jre/lib/net.properties in its java.net.useSystemProxies line. (Actually it's not clear to me this file is consulted under GlassFish, as sun.net.NetProperties constructs the path to that file by doing something basically identical to System.getProperty("java.home") + "/lib/net.properties", which at least on my Mac would not involve the jre subdirectory.) So the value of the java.net.useSystemProxies property is essentially always true when GlassFish is running.

Next, this has the effect of triggering a non-threadsafe native library load and execution by way of sun.net.spi.DefaultProxySelector, resulting in https://java.net/jira/browse/GLASSFISH-12213 (the potential of a JVM crash). Briefly, since DefaultProxySelector concludes that system proxies are to be used, it makes use of libgconf which turns out to be at the root of this JVM bug: http://bugs.sun.com/view_bug.do?bug_id=7188755 This JVM issue is fixed in a later build of Java 7.

The workaround for this bug (maybe?) and GLASSFISH-12213 is most likely to make sure that defaults.cfg or init.cfg contains the line:

proxy.use.system = false

,,,as that will avoid the System.setProperty() call in SystemInfo and hence the native library load.

Generated at Wed Apr 01 13:56:27 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.