glassfish
  1. glassfish
  2. GLASSFISH-20620

Threading problem in SerialInitContextFactory

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      If a standalone java application connects to different glassfish clusters from multiple threads using the following snippet of code, it will, on rare occasions, end up with a bad endpoints list.

      Properties contextEnv = new Properties();
      contextEnv.setProperty("com.sun.appserv.iiop.endpoints", endpoints);
      InitialContext ctx = new InitialContext(contextEnv);
      

      The bad list results in a naming exception because the wrong cluster is contacted and the expected bean is not deployed to that cluster. The bad list contains all of the endpoints for two different clusters combined. This is the result of some synchronization problems in SerialInitContextFactory and GroupInfoServiceObserverImpl. I will attach updated files that fix the problem. These files have been updated from the latest version in the 3.1.2-minnow branch (under common/glassfish-naming). It looks like the code has been refactored in the 4.0 branch, so I don't know if the problem exists there. It looks like the problem exists in the 3.1.3 branch, but I have only tested with 3.1.2.

        Activity

        Hide
        sarnoth added a comment -

        I can't seem to attach files, so I will paste the modified methods here.
        This is from SerialInitContextFactory. In addition to the bug I was chasing I fixed an occurrence of double checked locking out of principle. I don't think it was actually contributing to this bug.

            /**
             * Create the InitialContext object.
             */
            @Override
            @SuppressWarnings("unchecked")
            public Context getInitialContext(Hashtable env) throws NamingException {
                final Hashtable myEnv = env == null ? new Hashtable() : env ;
        
                boolean membershipChangeForced = false ;
        
                fineLog( "getInitialContext: env={0}", env ) ;
                useLB = propertyIsSet(myEnv, IIOP_ENDPOINTS_PROPERTY)
                    || propertyIsSet(myEnv, LOAD_BALANCING_PROPERTY) ;
        
                synchronized( SerialInitContextFactory.class ) {
                    if (useLB && !initialized) {
                            // Always create one rrPolicy to be shared, if needed.
                            final List<String> epList = getEndpointList( myEnv ) ;
                            rrPolicy = new RoundRobinPolicy(epList);
        
                            GroupInfoService gis = null ;
                            try {
                                gis = (GroupInfoService) (getORB().resolve_initial_references(
                                    ORBLocator.FOLB_CLIENT_GROUP_INFO_SERVICE));
                            } catch (InvalidName ex) {
                                doLog(Level.SEVERE,
                                    "Exception in SerialInitContextFactory constructor {0}",
                                        ex);
                            }
        
                            giso = new GroupInfoServiceObserverImpl( gis, rrPolicy );
        
                            gis.addObserver(giso);
        
                            // fineLog( "getInitialContext: rrPolicy = {0}", rrPolicy );
        
                            // this should force the initialization of the resources providers
                            if (habitat!=null) {
                                for (NamingObjectsProvider provider :
                                    habitat.getAllByContract(NamingObjectsProvider.class)) {
                                    // no-op
                                }
                            }
        
                            // Get the actual content, not just the configured
                            // endpoints.
                            giso.forceMembershipChange();
                            membershipChangeForced = true ;
        
                            initialized = true ;
        
                            fineLog( "getInitialContext(initial): rrPolicy = {0}",
                                rrPolicy );
                    }
                }
        
                if (useLB || initialized)  {
                    // If myEnv already contains the IIOP_URL, don't get a new one:
                    // this getInitialContext call came from an internal
                    // new InitialContext call.
                    if (!myEnv.containsKey(IIOP_URL_PROPERTY)) {
                        Context ctx = SerialContext.getStickyContext() ;
                        if (ctx != null) {
                            return ctx ;
                        }
        
                        List<String> rrList;
                        synchronized(rrPolicy) {
                            // If the IIOP endpoint list is explicitly set in the env,
                            // update rrPolicy to use that information, otherwise just
                            // rotate rrPolicy to the next element.
                            if (myEnv.containsKey( IIOP_ENDPOINTS_PROPERTY ) ||
                                myEnv.containsKey( LOAD_BALANCING_PROPERTY )) {
                                final List<String> list = getEndpointList( myEnv ) ;
                                rrPolicy.setClusterInstanceInfoFromString(list);
                                if (!membershipChangeForced) {
                                    giso.forceMembershipChange() ;
                                }
                            }
        
                            rrList = rrPolicy.getNextRotation();
                            fineLog( "getInitialContext: rrPolicy = {0}", rrPolicy );
                        }
        
                        String corbalocURL = getCorbalocURL(rrList);
        
                        myEnv.put(IIOP_URL_PROPERTY, corbalocURL);
                    }
        
                    myEnv.put(ORBLocator.JNDI_CORBA_ORB_PROPERTY, getORB());
                } else {
                    if (defaultHost != null) {
                        myEnv.put( ORBLocator.OMG_ORB_INIT_HOST_PROPERTY, defaultHost ) ;
                    }
        
                    if (defaultPort != null) {
                        myEnv.put( ORBLocator.OMG_ORB_INIT_PORT_PROPERTY, defaultPort ) ;
                    }
                }
        
                return createInitialContext(myEnv);
            }
        

        And this is from GroupInfoServiceObserverImpl:

            private void doMembershipChange() {
                try {
                    synchronized(rr) {
                        List<ClusterInstanceInfo> instanceInfoList =
                            gis.getClusterInstanceInfo((String[])null, rr.getHostPortList() );
                        if (instanceInfoList != null && instanceInfoList.size() > 0) {
                            rr.setClusterInstanceInfo(instanceInfoList);
                        }
                    }
                } catch(Exception e) {
                    _logger.log(Level.SEVERE,
                        "groupinfoservice.membership.notification.problem", new Object[] {e});
                }
            }
        
        Show
        sarnoth added a comment - I can't seem to attach files, so I will paste the modified methods here. This is from SerialInitContextFactory. In addition to the bug I was chasing I fixed an occurrence of double checked locking out of principle. I don't think it was actually contributing to this bug. /** * Create the InitialContext object. */ @Override @SuppressWarnings( "unchecked" ) public Context getInitialContext(Hashtable env) throws NamingException { final Hashtable myEnv = env == null ? new Hashtable() : env ; boolean membershipChangeForced = false ; fineLog( "getInitialContext: env={0}" , env ) ; useLB = propertyIsSet(myEnv, IIOP_ENDPOINTS_PROPERTY) || propertyIsSet(myEnv, LOAD_BALANCING_PROPERTY) ; synchronized ( SerialInitContextFactory.class ) { if (useLB && !initialized) { // Always create one rrPolicy to be shared, if needed. final List< String > epList = getEndpointList( myEnv ) ; rrPolicy = new RoundRobinPolicy(epList); GroupInfoService gis = null ; try { gis = (GroupInfoService) (getORB().resolve_initial_references( ORBLocator.FOLB_CLIENT_GROUP_INFO_SERVICE)); } catch (InvalidName ex) { doLog(Level.SEVERE, "Exception in SerialInitContextFactory constructor {0}" , ex); } giso = new GroupInfoServiceObserverImpl( gis, rrPolicy ); gis.addObserver(giso); // fineLog( "getInitialContext: rrPolicy = {0}" , rrPolicy ); // this should force the initialization of the resources providers if (habitat!= null ) { for (NamingObjectsProvider provider : habitat.getAllByContract(NamingObjectsProvider.class)) { // no-op } } // Get the actual content, not just the configured // endpoints. giso.forceMembershipChange(); membershipChangeForced = true ; initialized = true ; fineLog( "getInitialContext(initial): rrPolicy = {0}" , rrPolicy ); } } if (useLB || initialized) { // If myEnv already contains the IIOP_URL, don't get a new one: // this getInitialContext call came from an internal // new InitialContext call. if (!myEnv.containsKey(IIOP_URL_PROPERTY)) { Context ctx = SerialContext.getStickyContext() ; if (ctx != null ) { return ctx ; } List< String > rrList; synchronized (rrPolicy) { // If the IIOP endpoint list is explicitly set in the env, // update rrPolicy to use that information, otherwise just // rotate rrPolicy to the next element. if (myEnv.containsKey( IIOP_ENDPOINTS_PROPERTY ) || myEnv.containsKey( LOAD_BALANCING_PROPERTY )) { final List< String > list = getEndpointList( myEnv ) ; rrPolicy.setClusterInstanceInfoFromString(list); if (!membershipChangeForced) { giso.forceMembershipChange() ; } } rrList = rrPolicy.getNextRotation(); fineLog( "getInitialContext: rrPolicy = {0}" , rrPolicy ); } String corbalocURL = getCorbalocURL(rrList); myEnv.put(IIOP_URL_PROPERTY, corbalocURL); } myEnv.put(ORBLocator.JNDI_CORBA_ORB_PROPERTY, getORB()); } else { if (defaultHost != null ) { myEnv.put( ORBLocator.OMG_ORB_INIT_HOST_PROPERTY, defaultHost ) ; } if (defaultPort != null ) { myEnv.put( ORBLocator.OMG_ORB_INIT_PORT_PROPERTY, defaultPort ) ; } } return createInitialContext(myEnv); } And this is from GroupInfoServiceObserverImpl: private void doMembershipChange() { try { synchronized (rr) { List<ClusterInstanceInfo> instanceInfoList = gis.getClusterInstanceInfo(( String []) null , rr.getHostPortList() ); if (instanceInfoList != null && instanceInfoList.size() > 0) { rr.setClusterInstanceInfo(instanceInfoList); } } } catch (Exception e) { _logger.log(Level.SEVERE, "groupinfoservice.membership.notification.problem" , new Object [] {e}); } }

          People

          • Assignee:
            michael.y.chen
            Reporter:
            sarnoth
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: