glassfish
  1. glassfish
  2. GLASSFISH-18024

virtual network interfaces introduced by virtualization systems regress Glassfish 3.1.2 GMS auto selection of an appropriate network interface to use

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.2_b14
    • Fix Version/s: 4.0
    • Labels:
      None
    • Environment:

      OEL 5,
      JDK 1.6.0_24 64 bits

      Description

      The EJB automatic timer migration works in shutdown instance mode. That means after automatic timer is created, use asadmin stop-instance, the timer migration will happen. But it doesn't work in crashing mode when the instance containing timer was killed.

      The EJB timer app and logs are attached. The steps to reproduce the error are in EJB_Autotimer_FO/ant.output. The DAS log and instance logs are under EJB_Autotimer_FO/testListAutoTimer/logs/st-domain and st-cluster.

        Issue Links

          Activity

          mzh777 created issue -
          Hide
          mzh777 added a comment -

          Since the killing of instance103 is happened after the tests, attach more logs of stack trace during fail-over.

          Show
          mzh777 added a comment - Since the killing of instance103 is happened after the tests, attach more logs of stack trace during fail-over.
          mzh777 made changes -
          Field Original Value New Value
          Attachment issue-18024.zip [ 48564 ]
          Hide
          marina vatkina added a comment -

          Joe, can you take a look? See the logs issue-18024.zip

          The tx-log-dir is stored correctly on instance103:

          [#|2011-12-15T16:50:08.408-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.jta|_ThreadID=10;_ThreadName=Thread-2;|Storing GMS instance instance103 data TX_LOG_DIR : /net/asqe-logs.us.oracle.com/export1/hatxLogsMing/instance103/tx|#]

          But on instance104 it's not found:

          [#|2011-12-16T00:57:32.670+0000|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.jta|_ThreadID=34;_ThreadName=Thread-2;|[GMSCallBack] Recovering for instance: instance103 logdir: null|#]

          Show
          marina vatkina added a comment - Joe, can you take a look? See the logs issue-18024.zip The tx-log-dir is stored correctly on instance103: [#|2011-12-15T16:50:08.408-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.jta|_ThreadID=10;_ThreadName=Thread-2;|Storing GMS instance instance103 data TX_LOG_DIR : /net/asqe-logs.us.oracle.com/export1/hatxLogsMing/instance103/tx|#] But on instance104 it's not found: [#|2011-12-16T00:57:32.670+0000|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.jta|_ThreadID=34;_ThreadName=Thread-2;| [GMSCallBack] Recovering for instance: instance103 logdir: null|#]
          marina vatkina made changes -
          Assignee marina vatkina [ mvatkina ] Joe Fialli [ jfialli ]
          Hide
          Joe Fialli added a comment -

          There appears to be an inconsistency in machine configurations for the test.
          Some cluster members are defaulting to IPv6 addresses for GMS while others are defaulting to IPv4.
          instance103, instance106 and instance110 have a GMS_LISTENER for tcp at an IPv6 address.

          Potentially the workaround from http://java.net/jira/browse/GLASSFISH-17926 needs to be employed so the IPv6 addresses
          will not be in log. Since instance103 was involved in not getting info and it has an IPv6 address below,
          really need to workaround known issue described in GF-17926. (fix was integrated in BG workspace on 12/9,
          I am uncertain if that is gf 4.0 b14)

          From the server.log.
          [#|2011-12-16T00:48:45.055+0000|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=20;_ThreadName=Thread-2;|GMS1092: GMS View Change Received for group: st-cluster : Members in view for CLUSTER_STOP_EVENT(before change analysis) are :
          1: MemberId: instance101, MemberType: CORE, Address: 10.133.184.158:9160:228.9.217.5:29944:st-cluster:instance101
          2: MemberId: instance102, MemberType: CORE, Address: 10.133.184.159:9106:228.9.217.5:29944:st-cluster:instance102
          3: MemberId: instance103, MemberType: CORE, Address: fe80:0:0:0:fcff:ffff:feff:ffff%6:9099:228.9.217.5:29944:st-cluster:instance103
          4: MemberId: instance104, MemberType: CORE, Address: 10.133.184.158:9166:228.9.217.5:29944:st-cluster:instance104
          5: MemberId: instance105, MemberType: CORE, Address: 10.133.184.159:9168:228.9.217.5:29944:st-cluster:instance105
          6: MemberId: instance106, MemberType: CORE, Address: fe80:0:0:0:fcff:ffff:feff:ffff%6:9115:228.9.217.5:29944:st-cluster:instance106
          7: MemberId: instance107, MemberType: CORE, Address: 10.133.184.159:9158:228.9.217.5:29944:st-cluster:instance107
          8: MemberId: instance109, MemberType: CORE, Address: 10.133.184.159:9152:228.9.217.5:29944:st-cluster:instance109
          9: MemberId: instance110, MemberType: CORE, Address: fe80:0:0:0:fcff:ffff:feff:ffff%6:9095:228.9.217.5:29944:st-cluster:instance110
          10: MemberId: server, MemberType: SPECTATOR, Address: 10.133.184.158:9165:228.9.217.5:29944:st-cluster:server

          #]
          Show
          Joe Fialli added a comment - There appears to be an inconsistency in machine configurations for the test. Some cluster members are defaulting to IPv6 addresses for GMS while others are defaulting to IPv4. instance103, instance106 and instance110 have a GMS_LISTENER for tcp at an IPv6 address. Potentially the workaround from http://java.net/jira/browse/GLASSFISH-17926 needs to be employed so the IPv6 addresses will not be in log. Since instance103 was involved in not getting info and it has an IPv6 address below, really need to workaround known issue described in GF-17926. (fix was integrated in BG workspace on 12/9, I am uncertain if that is gf 4.0 b14) From the server.log. [#|2011-12-16T00:48:45.055+0000|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=20;_ThreadName=Thread-2;|GMS1092: GMS View Change Received for group: st-cluster : Members in view for CLUSTER_STOP_EVENT(before change analysis) are : 1: MemberId: instance101, MemberType: CORE, Address: 10.133.184.158:9160:228.9.217.5:29944:st-cluster:instance101 2: MemberId: instance102, MemberType: CORE, Address: 10.133.184.159:9106:228.9.217.5:29944:st-cluster:instance102 3: MemberId: instance103, MemberType: CORE, Address: fe80:0:0:0:fcff:ffff:feff:ffff%6:9099:228.9.217.5:29944:st-cluster:instance103 4: MemberId: instance104, MemberType: CORE, Address: 10.133.184.158:9166:228.9.217.5:29944:st-cluster:instance104 5: MemberId: instance105, MemberType: CORE, Address: 10.133.184.159:9168:228.9.217.5:29944:st-cluster:instance105 6: MemberId: instance106, MemberType: CORE, Address: fe80:0:0:0:fcff:ffff:feff:ffff%6:9115:228.9.217.5:29944:st-cluster:instance106 7: MemberId: instance107, MemberType: CORE, Address: 10.133.184.159:9158:228.9.217.5:29944:st-cluster:instance107 8: MemberId: instance109, MemberType: CORE, Address: 10.133.184.159:9152:228.9.217.5:29944:st-cluster:instance109 9: MemberId: instance110, MemberType: CORE, Address: fe80:0:0:0:fcff:ffff:feff:ffff%6:9095:228.9.217.5:29944:st-cluster:instance110 10: MemberId: server, MemberType: SPECTATOR, Address: 10.133.184.158:9165:228.9.217.5:29944:st-cluster:server #]
          Hide
          mzh777 added a comment -

          Network diagnostic util results on asqe-x2250-st3.us.oracle.com:

          1. java -classpath shoal-gms-impl.jar com.sun.enterprise.mgmt.transport.NetworkUtility
            Java property java.net.preferIPv6Addresses=false
            AllLocalAddresses() = [/fe80:0:0:0:fcff:ffff:feff:ffff%6, /fe80:0:0:0:223:8bff:fe64:7a56%7, /10.133.184.160, /fe80:0:0:0:200:ff:fe00:0%5, /192.168.122.1]
            interface name:vif0.0 isUp?:true
            Found first interface.vif0.0 isUp?:true
            Dec 16, 2011 10:30:23 AM com.sun.enterprise.mgmt.transport.NetworkUtility getFirstNetworkInterface
            INFO: getFirstNetworkInterface result: interface name:vif0.0 address:/fe80:0:0:0:fcff:ffff:feff:ffff%6
            getFirstNetworkInterface() = name:vif0.0 (vif0.0) index: 6 addresses:
            /fe80:0:0:0:fcff:ffff:feff:ffff%6;

          getFirstInetAddress(preferIPv6Addresses:false)=null
          getFirstInetAddress()=/fe80:0:0:0:fcff:ffff:feff:ffff%6
          getFirstInetAddress( true ) = /fe80:0:0:0:fcff:ffff:feff:ffff%6
          getFirstInetAddress( false ) = null
          getLocalHostAddress = asqe-x2250-st3/10.133.184.160
          getFirstNetworkInteface() = name:vif0.0 (vif0.0) index: 6 addresses:
          /fe80:0:0:0:fcff:ffff:feff:ffff%6;

          getNetworkInetAddress(firstNetworkInteface, true) = /fe80:0:0:0:fcff:ffff:feff:ffff%6
          getNetworkInetAddress(firstNetworkInteface, false) = null

          -------------------------------------------------------

          All Network Interfaces

          **************************************************
          Display name: vif0.0
          Name: vif0.0
          PreferIPv6Addresses: false
          InetAddress: /fe80:0:0:0:fcff:ffff:feff:ffff%6
          Up? true
          Loopback? false
          PointToPoint? false
          Supports multicast? false
          Virtual? false
          Hardware address: [-2, -1, -1, -1, -1, -1]
          MTU: 1500
          Network Inet Address (preferIPV6=false) null
          Network Inet Address (preferIPV6=true) /fe80:0:0:0:fcff:ffff:feff:ffff%6
          resolveBindInterfaceName(vif0.0)=fe80:0:0:0:fcff:ffff:feff:ffff%6

          **************************************************
          Display name: peth0
          Name: peth0
          PreferIPv6Addresses: false
          InetAddress: /fe80:0:0:0:fcff:ffff:feff:ffff%2
          Up? true
          Loopback? false
          PointToPoint? false
          Supports multicast? false
          Virtual? false
          Hardware address: [-2, -1, -1, -1, -1, -1]
          MTU: 1500
          Network Inet Address (preferIPV6=false) null
          Network Inet Address (preferIPV6=true) /fe80:0:0:0:fcff:ffff:feff:ffff%2
          resolveBindInterfaceName(peth0)=fe80:0:0:0:fcff:ffff:feff:ffff%2

          **************************************************
          Display name: eth0
          Name: eth0
          PreferIPv6Addresses: false
          InetAddress: /fe80:0:0:0:223:8bff:fe64:7a56%7
          InetAddress: /10.133.184.160
          Up? true
          Loopback? false
          PointToPoint? false
          Supports multicast? true
          Virtual? false
          Hardware address: [0, 35, -117, 100, 122, 86]
          MTU: 1500
          Network Inet Address (preferIPV6=false) /10.133.184.160
          Network Inet Address (preferIPV6=true) /fe80:0:0:0:223:8bff:fe64:7a56%7
          resolveBindInterfaceName(eth0)=127.0.0.1

          **************************************************
          Display name: virbr0
          Name: virbr0
          PreferIPv6Addresses: false
          InetAddress: /fe80:0:0:0:200:ff:fe00:0%5
          InetAddress: /192.168.122.1
          Up? true
          Loopback? false
          PointToPoint? false
          Supports multicast? true
          Virtual? false
          Hardware address: null
          MTU: 1500
          Network Inet Address (preferIPV6=false) /192.168.122.1
          Network Inet Address (preferIPV6=true) /fe80:0:0:0:200:ff:fe00:0%5
          resolveBindInterfaceName(virbr0)=192.168.122.1

          **************************************************
          Display name: lo
          Name: lo
          PreferIPv6Addresses: false
          InetAddress: /0:0:0:0:0:0:0:1%1
          InetAddress: /127.0.0.1
          Up? true
          Loopback? true
          PointToPoint? false
          Supports multicast? false
          Virtual? false
          Hardware address: null
          MTU: 16436
          Network Inet Address (preferIPV6=false) /127.0.0.1
          Network Inet Address (preferIPV6=true) /0:0:0:0:0:0:0:1%1
          resolveBindInterfaceName(lo)=127.0.0.1

          1. ifconfig -a
            eth0 Link encap:Ethernet HWaddr 00:23:8B:64:7A:56
            inet addr:10.133.184.160 Bcast:10.133.191.255 Mask:255.255.248.0
            inet6 addr: fe80::223:8bff:fe64:7a56/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:103208929 errors:0 dropped:0 overruns:0 frame:0
            TX packets:9896652 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:26781337018 (24.9 GiB) TX bytes:3843440786 (3.5 GiB)

          eth1 Link encap:Ethernet HWaddr 00:23:8B:64:7A:57
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
          Memory:dffa0000-dffc0000

          lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:525791 errors:0 dropped:0 overruns:0 frame:0
          TX packets:525791 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:43047668 (41.0 MiB) TX bytes:43047668 (41.0 MiB)

          peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
          RX packets:106850888 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9907515 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:27051284760 (25.1 GiB) TX bytes:3844287158 (3.5 GiB)
          Memory:dffe0000-e0000000

          sit0 Link encap:IPv6-in-IPv4
          NOARP MTU:1480 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          veth1 Link encap:Ethernet HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          veth2 Link encap:Ethernet HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          veth3 Link encap:Ethernet HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          veth4 Link encap:Ethernet HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
          RX packets:9896433 errors:0 dropped:0 overruns:0 frame:0
          TX packets:103208954 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3843147487 (3.5 GiB) TX bytes:26781338668 (24.9 GiB)

          vif0.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          vif0.2 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          vif0.3 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          vif0.4 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

          virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:2954 (2.8 KiB)

          xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
          RX packets:65446109 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:12163201681 (11.3 GiB) TX bytes:0 (0.0 b)

          Show
          mzh777 added a comment - Network diagnostic util results on asqe-x2250-st3.us.oracle.com: java -classpath shoal-gms-impl.jar com.sun.enterprise.mgmt.transport.NetworkUtility Java property java.net.preferIPv6Addresses=false AllLocalAddresses() = [/fe80:0:0:0:fcff:ffff:feff:ffff%6, /fe80:0:0:0:223:8bff:fe64:7a56%7, /10.133.184.160, /fe80:0:0:0:200:ff:fe00:0%5, /192.168.122.1] interface name:vif0.0 isUp?:true Found first interface.vif0.0 isUp?:true Dec 16, 2011 10:30:23 AM com.sun.enterprise.mgmt.transport.NetworkUtility getFirstNetworkInterface INFO: getFirstNetworkInterface result: interface name:vif0.0 address:/fe80:0:0:0:fcff:ffff:feff:ffff%6 getFirstNetworkInterface() = name:vif0.0 (vif0.0) index: 6 addresses: /fe80:0:0:0:fcff:ffff:feff:ffff%6; getFirstInetAddress(preferIPv6Addresses:false)=null getFirstInetAddress()=/fe80:0:0:0:fcff:ffff:feff:ffff%6 getFirstInetAddress( true ) = /fe80:0:0:0:fcff:ffff:feff:ffff%6 getFirstInetAddress( false ) = null getLocalHostAddress = asqe-x2250-st3/10.133.184.160 getFirstNetworkInteface() = name:vif0.0 (vif0.0) index: 6 addresses: /fe80:0:0:0:fcff:ffff:feff:ffff%6; getNetworkInetAddress(firstNetworkInteface, true) = /fe80:0:0:0:fcff:ffff:feff:ffff%6 getNetworkInetAddress(firstNetworkInteface, false) = null ------------------------------------------------------- All Network Interfaces ************************************************** Display name: vif0.0 Name: vif0.0 PreferIPv6Addresses: false InetAddress: /fe80:0:0:0:fcff:ffff:feff:ffff%6 Up? true Loopback? false PointToPoint? false Supports multicast? false Virtual? false Hardware address: [-2, -1, -1, -1, -1, -1] MTU: 1500 Network Inet Address (preferIPV6=false) null Network Inet Address (preferIPV6=true) /fe80:0:0:0:fcff:ffff:feff:ffff%6 resolveBindInterfaceName(vif0.0)=fe80:0:0:0:fcff:ffff:feff:ffff%6 ************************************************** Display name: peth0 Name: peth0 PreferIPv6Addresses: false InetAddress: /fe80:0:0:0:fcff:ffff:feff:ffff%2 Up? true Loopback? false PointToPoint? false Supports multicast? false Virtual? false Hardware address: [-2, -1, -1, -1, -1, -1] MTU: 1500 Network Inet Address (preferIPV6=false) null Network Inet Address (preferIPV6=true) /fe80:0:0:0:fcff:ffff:feff:ffff%2 resolveBindInterfaceName(peth0)=fe80:0:0:0:fcff:ffff:feff:ffff%2 ************************************************** Display name: eth0 Name: eth0 PreferIPv6Addresses: false InetAddress: /fe80:0:0:0:223:8bff:fe64:7a56%7 InetAddress: /10.133.184.160 Up? true Loopback? false PointToPoint? false Supports multicast? true Virtual? false Hardware address: [0, 35, -117, 100, 122, 86] MTU: 1500 Network Inet Address (preferIPV6=false) /10.133.184.160 Network Inet Address (preferIPV6=true) /fe80:0:0:0:223:8bff:fe64:7a56%7 resolveBindInterfaceName(eth0)=127.0.0.1 ************************************************** Display name: virbr0 Name: virbr0 PreferIPv6Addresses: false InetAddress: /fe80:0:0:0:200:ff:fe00:0%5 InetAddress: /192.168.122.1 Up? true Loopback? false PointToPoint? false Supports multicast? true Virtual? false Hardware address: null MTU: 1500 Network Inet Address (preferIPV6=false) /192.168.122.1 Network Inet Address (preferIPV6=true) /fe80:0:0:0:200:ff:fe00:0%5 resolveBindInterfaceName(virbr0)=192.168.122.1 ************************************************** Display name: lo Name: lo PreferIPv6Addresses: false InetAddress: /0:0:0:0:0:0:0:1%1 InetAddress: /127.0.0.1 Up? true Loopback? true PointToPoint? false Supports multicast? false Virtual? false Hardware address: null MTU: 16436 Network Inet Address (preferIPV6=false) /127.0.0.1 Network Inet Address (preferIPV6=true) /0:0:0:0:0:0:0:1%1 resolveBindInterfaceName(lo)=127.0.0.1 ifconfig -a eth0 Link encap:Ethernet HWaddr 00:23:8B:64:7A:56 inet addr:10.133.184.160 Bcast:10.133.191.255 Mask:255.255.248.0 inet6 addr: fe80::223:8bff:fe64:7a56/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:103208929 errors:0 dropped:0 overruns:0 frame:0 TX packets:9896652 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:26781337018 (24.9 GiB) TX bytes:3843440786 (3.5 GiB) eth1 Link encap:Ethernet HWaddr 00:23:8B:64:7A:57 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Memory:dffa0000-dffc0000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:525791 errors:0 dropped:0 overruns:0 frame:0 TX packets:525791 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:43047668 (41.0 MiB) TX bytes:43047668 (41.0 MiB) peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:106850888 errors:0 dropped:0 overruns:0 frame:0 TX packets:9907515 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:27051284760 (25.1 GiB) TX bytes:3844287158 (3.5 GiB) Memory:dffe0000-e0000000 sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) veth1 Link encap:Ethernet HWaddr 00:00:00:00:00:00 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) veth2 Link encap:Ethernet HWaddr 00:00:00:00:00:00 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) veth3 Link encap:Ethernet HWaddr 00:00:00:00:00:00 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) veth4 Link encap:Ethernet HWaddr 00:00:00:00:00:00 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:9896433 errors:0 dropped:0 overruns:0 frame:0 TX packets:103208954 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3843147487 (3.5 GiB) TX bytes:26781338668 (24.9 GiB) vif0.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) vif0.2 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) vif0.3 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) vif0.4 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:24 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:2954 (2.8 KiB) xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:65446109 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:12163201681 (11.3 GiB) TX bytes:0 (0.0 b)
          Hide
          Joe Fialli added a comment -

          Some of the machines in cluster are configured with Xen and some are not.
          See network interface "vif0.0" in http://wiki.xen.org/xenwiki/XenNetworking
          for more info on all the virtual network interfaces being introduced by Xen
          on some of the machines running the cluster (but not all of the machines.)

          The ones with Xen on them have network interface vif0.0 that has only an IPv6 address
          as first network interface. Other machines in cluster have their first network interface
          as a dual stack (Ipv4 and IPv6). There is an issue if not all machines have same
          homogenous networking configured AND one does not specify which network interface to
          use on a multi-home machine.

          Quickest possible Workaround:
          asadmin create-cluster --bindaddress eth0 <clustername>

          This will workaround that not all multihome machines in cluster are not configured in same manner.
          This states to use "eth0" network interface for all cluster members.

          Or remove/disable Xen network interfaces on machines if they should not be there.

          Show
          Joe Fialli added a comment - Some of the machines in cluster are configured with Xen and some are not. See network interface "vif0.0" in http://wiki.xen.org/xenwiki/XenNetworking for more info on all the virtual network interfaces being introduced by Xen on some of the machines running the cluster (but not all of the machines.) The ones with Xen on them have network interface vif0.0 that has only an IPv6 address as first network interface. Other machines in cluster have their first network interface as a dual stack (Ipv4 and IPv6). There is an issue if not all machines have same homogenous networking configured AND one does not specify which network interface to use on a multi-home machine. Quickest possible Workaround: asadmin create-cluster --bindaddress eth0 <clustername> This will workaround that not all multihome machines in cluster are not configured in same manner. This states to use "eth0" network interface for all cluster members. Or remove/disable Xen network interfaces on machines if they should not be there.
          Joe Fialli made changes -
          Link This issue depends on GLASSFISH-18047 [ GLASSFISH-18047 ]
          Hide
          Joe Fialli added a comment -

          Due to XEN being installed on some machines that compose a cluster for this issue,
          not all machines are selecting network interfaces with matching characteristics.
          The XEN introduced network interfaces are IPv6 only and the machines in cluster without
          XEN have dual stack network interface for eth0. Thus, half machines are using IPv6 only
          and other half are using IPv4 addresses (as preferred for dual stack).

          Suggested workaround of create-cluster --gms-bind-interface-address eth0 hit the
          reported bug GLASSFISH-18047.

          Show
          Joe Fialli added a comment - Due to XEN being installed on some machines that compose a cluster for this issue, not all machines are selecting network interfaces with matching characteristics. The XEN introduced network interfaces are IPv6 only and the machines in cluster without XEN have dual stack network interface for eth0. Thus, half machines are using IPv6 only and other half are using IPv4 addresses (as preferred for dual stack). Suggested workaround of create-cluster --gms-bind-interface-address eth0 hit the reported bug GLASSFISH-18047 .
          mzh777 made changes -
          Tags 312_gms 312_qa
          mzh777 made changes -
          Tags 312_gms 312_qa 312_qa
          mzh777 made changes -
          Tags 312_qa 312_failover 312_qa
          Hide
          Joe Fialli added a comment - - edited

          Downgraded this issue to minor.

          Ming did confirm that when the machine with XEN installed was removed from the cluster,
          that the test did pass.

          A review of the server logs showed that not all instances were on same subnet.
          All instances that were on a machine with XEN installed on it were incorrectly
          selecting the XEN virtual network interface for GMS communications.

          Summary of issue:

          The introduction of non-multicast mode for Group Management Services (GMS) in Glassfish 3.1.2 altered which network interface was automatically selected to be used on a multi-homed machine for clustering communications. This change can result in some clustered instances
          no longer being able to join their running cluster.

          In Glassfish 3.1-3.1.1, a network interface that did not support multicast was not considered as a candidate to be selected as the network interface to be used for cluster communications.
          Thus, the automatic selection of network interfaces was impacted. Specifically,
          virtual network interfaces that used to be ignored since interface did not support multicast,
          have been incorrectly selected as the default network interfaces for cluster communications.

          Workarounds:

          • Either disable/remove the network interfaces that are being selected incorrectly.

          Or

          • Specify which network interface to use on the machine(s) selecting the incorrect network interface. Here is pointer to documentation on how to specify which network interface
            to use on a multi-home machine.

          Link: http://docs.oracle.com/cd/E18930_01/html/821-2426/gjfnl.html#gjdlw

          Show
          Joe Fialli added a comment - - edited Downgraded this issue to minor. Ming did confirm that when the machine with XEN installed was removed from the cluster, that the test did pass. A review of the server logs showed that not all instances were on same subnet. All instances that were on a machine with XEN installed on it were incorrectly selecting the XEN virtual network interface for GMS communications. Summary of issue: The introduction of non-multicast mode for Group Management Services (GMS) in Glassfish 3.1.2 altered which network interface was automatically selected to be used on a multi-homed machine for clustering communications. This change can result in some clustered instances no longer being able to join their running cluster. In Glassfish 3.1-3.1.1, a network interface that did not support multicast was not considered as a candidate to be selected as the network interface to be used for cluster communications. Thus, the automatic selection of network interfaces was impacted. Specifically, virtual network interfaces that used to be ignored since interface did not support multicast, have been incorrectly selected as the default network interfaces for cluster communications. Workarounds: Either disable/remove the network interfaces that are being selected incorrectly. Or Specify which network interface to use on the machine(s) selecting the incorrect network interface. Here is pointer to documentation on how to specify which network interface to use on a multi-home machine. Link: http://docs.oracle.com/cd/E18930_01/html/821-2426/gjfnl.html#gjdlw
          Joe Fialli made changes -
          Priority Critical [ 2 ] Minor [ 4 ]
          Joe Fialli made changes -
          Tags 312_failover 312_qa 312_failover 312_qa 3_1_2-exclude
          Joe Fialli made changes -
          Tags 312_failover 312_qa 3_1_2-exclude 312_failover 312_qa 3_1_2-exclude 3_1_2-release-notes
          Rebecca Parks made changes -
          Tags 312_failover 312_qa 3_1_2-exclude 3_1_2-release-notes 312_failover 312_qa 3_1_2-exclude 3_1_2-release-note-added 3_1_2-release-notes
          Joe Fialli made changes -
          Summary EJB automatic timer migration doesn't work in instance crashing mode virtual network interfaces introduced by virtualization systems regress Glassfish 3.1.2 GMS auto selection of an appropriate network interface to use
          Component/s group_management_service [ 10598 ]
          Component/s failover [ 10629 ]
          Hide
          Joe Fialli added a comment -

          Fix for this is integrated into shoal-1.6.18 (shoal svn 1745).

          Note that shoal 1.6.17 is in glassfish 3.1.2 so this is not fixed in Glassfish 3.1.2.

          Show
          Joe Fialli added a comment - Fix for this is integrated into shoal-1.6.18 (shoal svn 1745). Note that shoal 1.6.17 is in glassfish 3.1.2 so this is not fixed in Glassfish 3.1.2.
          Hide
          Tom Mueller added a comment -

          Is this fixed in 4.0 since shoal 1.6.18 is in 4.0?

          Show
          Tom Mueller added a comment - Is this fixed in 4.0 since shoal 1.6.18 is in 4.0?
          Hide
          Joe Fialli added a comment -

          It was quite a complex environment that this issue was reported against.
          We don not have automated test to verify such an environment.

          Here is detailed commit message on how this issue was addressed in the Shoal GMS.

          > Altered algorithm for selecting network interface. Unless java.net.preferIPv6Addresses is set to true,
          > will favor network interface supporting IPv4 and multicast. Will settle for network interface that
          > does not support multicast if one exists. Lastly will settle for network interface that does not
          > support preferred IPv address format.
          >
          > Fix for GLASSFISH-18047: allow a network interface name as BIND_INTERFACE_ADDRESS.
          > Allows one to set network interface such as "eth0" if all machines involved have same network interface name for a cluster.

          The issue has been marked resolved but it is unverified.

          Show
          Joe Fialli added a comment - It was quite a complex environment that this issue was reported against. We don not have automated test to verify such an environment. Here is detailed commit message on how this issue was addressed in the Shoal GMS. > Altered algorithm for selecting network interface. Unless java.net.preferIPv6Addresses is set to true, > will favor network interface supporting IPv4 and multicast. Will settle for network interface that > does not support multicast if one exists. Lastly will settle for network interface that does not > support preferred IPv address format. > > Fix for GLASSFISH-18047 : allow a network interface name as BIND_INTERFACE_ADDRESS. > Allows one to set network interface such as "eth0" if all machines involved have same network interface name for a cluster. The issue has been marked resolved but it is unverified.
          Joe Fialli made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 4.0 [ 10970 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Joe Fialli
              Reporter:
              mzh777
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: