[GLASSFISH-18272] [perf] Excessive lock contention during execution of trade2 in-memory replication benchmark with JRockit VM Created: 30/Jan/12  Updated: 03/Dec/12  Resolved: 31/Jan/12

Status: Closed
Project: glassfish
Component/s: performance
Affects Version/s: 3.1.2_b19
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: amitagarwal Assignee: Mahesh Kannan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: PSRBUG

 Description   

Trade2 in-memory replication benchmark with JRockit VM gives 90% lower throughput than Sun Hotspot VM. Thread dump shows contention in following code-path,

"http-thread-pool-8080(4)" id=142 idx=0x220 tid=32438 prio=5 alive, blocked, native_blocked, daemon
– Blocked trying to get lock: java/util/HashMap@0x100903bb8[unlocked]
at jrockit/vm/Threads.yield()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1386)
at jrockit/vm/Locks.lockFat(Locks.java:1512)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at java/lang/reflect/Proxy.getProxyClass(Proxy.java:417)[optimized]
at java/lang/reflect/Proxy.newProxyInstance(Proxy.java:581)[optimized]
at sun/reflect/annotation/AnnotationParser.annotationForMap(AnnotationParser.java:239)[inlined]
at sun/reflect/annotation/AnnotationParser.parseAnnotation(AnnotationParser.java:229)[inlined]
at sun/reflect/annotation/AnnotationParser.parseAnnotations2(AnnotationParser.java:69)[inlined]
at sun/reflect/annotation/AnnotationParser.parseAnnotations(AnnotationParser.java:52)[optimized]
at java/lang/reflect/Field.declaredAnnotations(Field.java:1014)[inlined]
at java/lang/reflect/Field.getAnnotation(Field.java:1000)[inlined]
at org/jvnet/hk2/component/InjectionManager.inject(InjectionManager.java:137)[optimized]
^-- Holding lock: java/lang/reflect/Field@0x130c6bd90[biased lock]
at org/jvnet/hk2/component/InjectionManager.inject(InjectionManager.java:93)[inlined]
at com/sun/hk2/component/AbstractCreatorImpl.inject(AbstractCreatorImpl.java:126)[inlined]
at com/sun/hk2/component/ConstructorCreator.initialize(ConstructorCreator.java:91)[optimized]
at com/sun/hk2/component/AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)[optimized]
at com/sun/hk2/component/EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)[optimized]
at com/sun/hk2/component/AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)[optimized]
at org/jvnet/hk2/component/Habitat.getBy(Habitat.java:1056)[inlined]
at org/jvnet/hk2/component/Habitat.getByType(Habitat.java:1037)[inlined]
at org/jvnet/hk2/component/Habitat.getComponent(Habitat.java:781)[optimized]
at com/sun/ejb/containers/ContainerSynchronization.registerForTxCheckpoint(ContainerSynchronization.java:244)
at com/sun/ejb/containers/StatefulSessionContainer.afterBegin(StatefulSessionContainer.java:1805)
at com/sun/ejb/containers/BaseContainer.startNewTx(BaseContainer.java:4692)[optimized]
at com/sun/ejb/containers/BaseContainer.preInvokeTx(BaseContainer.java:4597)[optimized]
at com/sun/ejb/containers/BaseContainer.preInvoke(BaseContainer.java:1914)[optimized]
at com/sun/ejb/containers/EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:205)[inlined]
at com/sun/ejb/containers/EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:123)[optimized]
at $Proxy190.displayProperties()V(Unknown Source)[optimized]
at sun/reflect/GeneratedMethodAccessor104.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at com/sun/corba/ee/impl/presentation/rmi/StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:241)[inlined]
at com/sun/corba/ee/impl/presentation/rmi/StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)[optimized]
at com/sun/corba/ee/impl/presentation/rmi/codegen/CodegenStubBase.invoke(CodegenStubBase.java:227)[inlined]
at trade/_MySession_DynamicStub.displayProperties()V(trade/_MySession_DynamicStub.java)[optimized]
at trade_client/TradeServletAction.doPingSFSBSession(TradeServletAction.java:58)[optimized]
at trade_client/TradeServletAction.doPortfolio(TradeServletAction.java:966)[optimized]
at trade_client/TradeServletAction.doBuy(TradeServletAction.java:447)
at trade_client/TradeScenarioServlet.performTask(TradeScenarioServlet.java:307)[inlined]
at trade_client/TradeScenarioServlet.doGet(TradeScenarioServlet.java:72)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:770)[optimized]
at org/apache/catalina/core/StandardWrapper.service(StandardWrapper.java:1542)[inlined]
at org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:281)[optimized]
at org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:175)[optimized]
at org/apache/catalina/core/StandardPipeline.doInvoke(StandardPipeline.java:655)[inlined]
at org/apache/catalina/core/StandardPipeline.invoke(StandardPipeline.java:595)[optimized]
at org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:161)[optimized]
at org/apache/catalina/connector/CoyoteAdapter.doService(CoyoteAdapter.java:331)[optimized]
at org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:231)[optimized]
at com/sun/enterprise/v3/services/impl/ContainerMapper.service(ContainerMapper.java:174)[optimized]
at com/sun/grizzly/http/ProcessorTask.invokeAdapter(ProcessorTask.java:849)[optimized]
at com/sun/grizzly/http/ProcessorTask.doProcess(ProcessorTask.java:746)[optimized]
at com/sun/grizzly/http/ProcessorTask.process(ProcessorTask.java:1045)[inlined]
at com/sun/grizzly/http/DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)[optimized]
at com/sun/grizzly/DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)[inlined]
at com/sun/grizzly/DefaultProtocolChain.execute(DefaultProtocolChain.java:104)[inlined]
at com/sun/grizzly/DefaultProtocolChain.execute(DefaultProtocolChain.java:90)[inlined]
at com/sun/grizzly/http/HttpProtocolChain.execute(HttpProtocolChain.java:79)[optimized]
at com/sun/grizzly/ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)[optimized]
at com/sun/grizzly/SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com/sun/grizzly/ContextTask.run(ContextTask.java:71)
at com/sun/grizzly/util/AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com/sun/grizzly/util/AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)



 Comments   
Comment by amitagarwal [ 30/Jan/12 ]

Tried a patch provided by Scott, this resolves the contention and throughput improved significantly (from 800 ops/sec to over 8000 ops/sec).

Comment by Mahesh Kannan [ 31/Jan/12 ]

svn commit -m "Fix for 18272. Tested by Amit. Reviewed by Marina. QL & EJB devtests passed"
Sending ejb-container/src/main/java/com/sun/ejb/containers/ContainerSynchronization.java
Sending ejb-container/src/main/java/com/sun/ejb/containers/StatefulSessionContainer.java
Sending ejb-container/src/main/java/com/sun/ejb/containers/builder/StatefulContainerBuilder.java
Transmitting file data ...
Committed revision 52348.

Approved by: Joe Di Pol

What is the impact on the customer of the bug?
Slow performance when running ejb apps with SFSBs in a tx

What is the cost/risk of fixing the bug?
Not much. Touched three files

Is there an impact on documentation or message strings?
Not for documentation.

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
We need to run ha QA tests

Which is the targeted build of 3.1.2 for this fix?
3_1_2_b20





[GLASSFISH-18271] need to add new l0n package Created: 30/Jan/12  Updated: 01/Feb/12  Resolved: 01/Feb/12

Status: Resolved
Project: glassfish
Component/s: packaging
Affects Version/s: 3.1.2_b20
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: gmurr Assignee: Romain Grécourt
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 3_1_2-approved

 Description   

a new package glassfish-lbplugin-gui was added in 3.1.2. It has content that need to be localized.
I added a new l10n package "glassfish-lbplugin-gui-l10n" to the l10n internal distribution. Could you please integrate it into the build system?



 Comments   
Comment by Romain Grécourt [ 31/Jan/12 ]
  • What is the impact on the customer of the bug?
    No translation would be available for oracle-glassfish users when using lb-plugin-gui.
  • What is the cost/risk of fixing the bug?

The module being integrated is com.sun.glassfish.packager.l10n.ips:l10n-internal-distribution:3.1.2-b17. It is a zip of an ips repository. Low risk, one version change in pom.xml and updates of sun-glassfish and web-glassfish distribution xml files (internal svn repo).

  • Is there an impact on documentation or message strings?
    This integration will add localization support for a module added recently.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Install oracle-glassfish in a non english langage, then try to use the lb-plugin from admin-console. The pages should not be displayed in english but translated properly.
  • Which is the targeted build of 3.1.2 for this fix?
    b21.
Comment by Romain Grécourt [ 31/Jan/12 ]

Small update, current version of com.sun.glassfish.packager.l10n.ips:l10n-internal-distribution (3.1.2-b16) already contains glassfish-lbplugin-gui-l10n. The only changes needed are the updates of sun-web-distributions.xml and sun-glassfish-distributions.xml.





[GLASSFISH-18265] String keys are showing on Monitoring Service page Created: 30/Jan/12  Updated: 30/Jan/12  Resolved: 30/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: Anissa Lam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File after-fix.png     PNG File bug.png     Text File diff.text    
Tags: 3_1_2-approved

 Description   

Back in early Dec, when trying to fix GLASSFISH-17923, for the the report from transability (t13y), the following keys got removed from the Strings.properties file accidentally. This was not noticed until now.

#monitoring Service page, Component name

monitoring.Jvm=JVM
monitoring.Http=HTTP Service
monitoring.TransactionService=Transaction Service
monitoring.JmsConnector=JMS/Connector Service:
monitoring.Orb=ORB
monitoring.Web=Web Container
monitoring.Ejb=EJB Container
monitoring.Jdbc=JDBC Connection Pool
monitoring.Connector=Connector Connection Pool
monitoring.ConnectorService=Connector Service
monitoring.JmsService=JMS Service
monitoring.WebServices=Web Services Container
monitoring.Jpa=JPA
monitoring.Security=Security
monitoring.Jersey=RESTful Web Services
monitoring.ThreadPool=Thread Pool
monitoring.Deployment=Deployment

We should add these keys back to the Strings.properties file.
This will affect localization since these keys are probably not in the latest localized version.

The attached screen shows how the screen looks currently and what it will look after the fix.



 Comments   
Comment by Anissa Lam [ 30/Jan/12 ]
  • What is the impact on the customer of the bug?
    User will see the String Key instead of the actual word. eg. monitor.connector instead of Connector Connection Pool as the component name when setting the monitoring level. May not be that bad for English, but will affect all other locale.
  • What is the cost/risk of fixing the bug?
    It is almost no risk. Add the above Strings back to Strings.properties file.
  • Is there an impact on documentation or message strings?
    Not for documentation. But need localization change. Georges confirm that he is sending installer file for localization tomorrow, and can do this for GUI as well if checked in the change today.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Go to Configuration -> any configuration -> Monitoring and check the component name in the table.
  • Which is the targeted build of 3.1.2 for this fix?
    3_1_2_b20
Comment by Anissa Lam [ 30/Jan/12 ]

Log Message:
------------
GLASSFISH-18265. Add missing key to monitoring configuration page for component names in the table.

Revisions:
----------
52344

Modified Paths:
---------------
branches/3.1.2/admingui/core/src/main/resources/org/glassfish/admingui/core/Strings.properties





[GLASSFISH-18260] RuntimeException on screen when changing pool for a given jdbc resource Created: 27/Jan/12  Updated: 31/Jan/12  Resolved: 28/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip
DAS:
OEL6 with Jrockit 1.6.0_29
SSH Node and derby:
Solaris 10, JDK 1.6.0_14
DCOM Node:
Windows XP, JDK 1.6.0_22 , FF9


Attachments: Text File diff.text     XML File domain.xml     JPEG File jdbcResource-runtimeexception.JPG     File resourceEditTabs.inc     Text File server.log.txt    
Tags: 312_qa, 312_regression, 3_1_2-approved

 Description   

I tried switching jdbc resource jdbc/__default to use a custom created JDBC Pool and got a RuntimeException on the screen. The pool got changed. The following was printed to server.log:

Caused by: java.lang.IllegalArgumentException: URI is not absolute
at java.net.URI.toURL(URI.java:1080)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:157)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 80 more

Here are the steps to reproduce:

1. Start derby on a remote machine, tuppy.
2. Go to Admin Console, JDBC Connection Pools and click on New.
3. Enter the following:

  • name: tDerbyPool,
  • Resource Type: javax.sql.DataSource,
  • DB driver vendor: Derby-30.

Click Next. On next page uncheck Isolation Level and enter the following properties:

  • DatabaseName: sun-appserv-samples
  • Password: APP
  • ServerName: tuppy
  • ConnectionAttributes: ;create=true

Click Finish.

4. Go to JDBC Resources, jdbc/__default and change Pool Name to the newly created pool. Hit save - exception is printed on the screen.



 Comments   
Comment by Anissa Lam [ 27/Jan/12 ]

Can you create a new JDBC resource that reference this newly created Pool ?

Comment by lidiam [ 27/Jan/12 ]

Yes, no exception thrown when creating new resource with this pool.

Comment by lidiam [ 27/Jan/12 ]

Even though exception is thrown, the resource is referencing new pool afterwards and works fine. I deployed an application (scrumtoys) to a standalone instance on the remote machine where derby is running (and pool pointing to) and it works fine.

Comment by Anissa Lam [ 27/Jan/12 ]

Agree there shouldn't be exception shown on screen, but this wasn't affecting anything and the intended functionality is performed correctly.
Refreshing the screen will clear the exception and continue without any ill effect.
I am lowering this to P3.

Comment by Anissa Lam [ 27/Jan/12 ]

Is this consistently reproducible ?
Both Suma and I tried and cannot reproduce this. However, the database may not be running at that time. Please include instruction on starting the DB on tuppy.

Comment by lidiam [ 27/Jan/12 ]

It is consistently reproducible on this system. When switching JDBC Pool for an existing JDBC Resource the exception is thrown on the screen. I started database on tuppy prior to executing this test. Here is output from netstat:

tuppy(j2eetest):/export/home/j2eetest/3.1.2/glassfish3/glassfish/nodes/tup/int1/logs# netstat -a | grep 1527
.1527 *. 0 0 49152 0 LISTEN
tuppy.1527 jed-asqe-41.us.oracle.com.7896 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7897 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7898 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7899 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7900 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7901 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7902 6912 0 49232 0 ESTABLISHED
tuppy.43704 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43704 49152 0 49152 0 ESTABLISHED
tuppy.43705 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43705 49152 0 49152 0 ESTABLISHED
tuppy.43706 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43706 49152 0 49152 0 ESTABLISHED
tuppy.43707 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43707 49152 0 49152 0 ESTABLISHED
tuppy.43708 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43708 49152 0 49152 0 ESTABLISHED
tuppy.43709 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43709 49152 0 49152 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.59139 6912 0 49232 0 ESTABLISHED
tuppy.43741 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43741 49152 0 49152 0 ESTABLISHED
tuppy.43742 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43742 49152 0 49152 0 ESTABLISHED
tuppy(j2eetest):/export/home/j2eetest/3.1.2/glassfish3/glassfish/nodes/tup/int1/logs#

Comment by lidiam [ 27/Jan/12 ]

Attaching domain.xml.

Comment by shaline [ 27/Jan/12 ]

I am seeing this issue as well on b19 on solaris 11. I deleted a JDBC resource and saw the java.lang.IllegalArgumentException, but the resource was deleted, and after that any operation involving a JDBC resource throws a RuntimeException in the Console. This exception started to show up after deleting a existing JDBC resource.

Comment by Anissa Lam [ 27/Jan/12 ]

I can reproduce this problem now. This is not related to particular JDBC pool, or if the DB is running or switching the pool. Finally realize this is related to existence of additional instance/cluster.

I am setting this back to P2.

The bug will occur whenever there is other instance/cluster besides DAS.
All you need to do is press the SAVE button and you will see the exception in screen.
The property sheet will be saved, thus the switching of the pool will happen as user expected, as mentioned in the bug report. However, any change to the property table will be lost, since the exception occurs after saving the property sheet and before saving the property table.

Working on the fix now.

Comment by Anissa Lam [ 27/Jan/12 ]

Looking at the code, this is a more general bug and may affect other resource edit page.
Indeed, I tried and see that this bug affect every resource edit page. Save on any resource when there is other instance/cluster will cause the exception.
This is a must fix for 3.1.2

Comment by Anissa Lam [ 28/Jan/12 ]

I strongly believe there is a duplicate block of code thats due to copy & paste error. These block of code is repeated, and thus causing this issue.
I have tested this thoroughly, with DAS only and with cluster/instance created. Tested it on all resource edit page.
I am attaching the diff, and the file, resourceEditTabs.inc file.
To test this, just save the file as

glassfish/lib/install/applications/__admingui/common/resourceNode/resourceEditTabs.inc

and then restart the server.
I have requested Lidia and Shaline to test this out also.

Here is the change request:

  • What is the impact on the customer of the bug?
    Huge impact. User will not be able to edit any resource if there is other instance/cluster exists besides DAS.
  • What is the cost/risk of fixing the bug?
    I say the risk is min. It is a very localized change, in one file, affecting only the SAVE button of the resource edit page. It will not affect anything else. And this SAVE button is broken anyway without this change.
  • Is there an impact on documentation or message strings?
    No
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Test all resource edit page with and without any instance created.
  • Which is the targeted build of 3.1.2 for this fix?
    3_1_2_b20
Comment by Anissa Lam [ 28/Jan/12 ]

Shaline's email confirmed:

The problem is resolved now.. I tried the below scenarios on b19 with the fix, and the Exception did not show up:
--create and start cluster.
--create a pool and a resource and add both server and cluster targets.
--edit the resource, change the pool names and save.
--edit the pool, advanced pages and save.
----enable/disable resource.
--delete resource
--delete pools.

All these tests passed and no exceptions seen in the server.log or console .
I even Added a Resource using "Add Resource" button, edited the resource and added cluster target. Deleted the resource and there were no exceptions seen as before.

Comment by Anissa Lam [ 28/Jan/12 ]

Fix checked in to 3.1.2 branch. The trunk doesn't not have this dup. block of code and is fine.

Log Message:
------------
GLASSFISH-18260. Remove the duplicate block of code that causes editing of any resource page to throw exception when there is instance or cluster present.

Revisions:
----------
52330

Modified Paths:
---------------
branches/3.1.2/admingui/common/src/main/resources/resourceNode/resourceEditTabs.inc

Comment by lidiam [ 28/Jan/12 ]

After applying the fix exception is no longer printed on the screen when changing JDBC Pool. However, the following exception is printed to the server.log:

[#|2012-01-27T18:02:47.243-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=34;_ThreadName=Thread-4;|Exception while visiting com/sun/gjc/spi/ManagedConnection.class of size 24223
java.lang.NullPointerException
at org.glassfish.hk2.classmodel.reflect.impl.TypesImpl.getType(TypesImpl.java:78)
at org.glassfish.hk2.classmodel.reflect.impl.ModelClassVisitor.visit(ModelClassVisitor.java:119)
at org.objectweb.asm.ClassReader.accept(Unknown Source)
at org.objectweb.asm.ClassReader.accept(Unknown Source)
at org.glassfish.hk2.classmodel.reflect.Parser$5.on(Parser.java:363)
at org.glassfish.hk2.classmodel.reflect.util.JarArchive.onSelectedEntries(JarArchive.java:125)
at org.glassfish.hk2.classmodel.reflect.Parser.doJob(Parser.java:348)
at org.glassfish.hk2.classmodel.reflect.Parser.access$300(Parser.java:70)
at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:307)
at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:296)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

#]
Comment by lidiam [ 28/Jan/12 ]

Tried several more times and the above exception is not printed any more.

Comment by shaline [ 31/Jan/12 ]

Verified this on b20 nightly dated b20-01-30-2012, The java.lang.NullPointerException does show up in the server.log, the very first time we create a resource or edit a resource. But the resource is created and changes made to resource are saved successfully even with this exception in the server.log.





[GLASSFISH-18254] "version" command returns Open Source Edition when running Oracle GlassFish Server Created: 25/Jan/12  Updated: 26/Jan/12  Resolved: 26/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin
Affects Version/s: 3.1.2_b19
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: Anissa Lam Assignee: Tom Mueller
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 3_1_2-approved

 Description   

Installed Orcale GlassFish Server promoted build 19, version returns this is open source edition.

Here is whats returned
%asadmin version
Version = GlassFish Server Open Source Edition 3.1.2-b19 (build 19)
Command version executed successfully.

I checked today's nightly (1/25 )build, it also says Open Source Edition.



 Comments   
Comment by Joe Di Pol [ 25/Jan/12 ]

I just tried B19, and I see it too:

$ ./asadmin version
Authentication failed with password from login store: /export/home/dipol/.asadminpass
Enter admin password for user "admin"> 
Version = GlassFish Server Open Source Edition 3.1.2-b19 (build 19)
Command version executed successfully.

If I get the local version it works correctly:

$ ./asadmin version --local
Using locally retrieved version string from version class.
Version = Oracle GlassFish Server 3.1.2 (build 19)
Command version executed successfully.

And I just verified that both cases work correctly in 3.1.1

Comment by Tom Mueller [ 26/Jan/12 ]

This behavior is due to the GlassFishBranding class not being able to find the "BrandingVersion" resource bundle. It does the following:

vRes = (PropertyResourceBundle) PropertyResourceBundle.getBundle("BrandingVersion");

however, a MissingResourceException is thrown, even though the BrandingVersion.properties file exists within the branding-fragment.jar file that is shipped with OGS.

The reason that the bundle is not found is a mismatch in the OSGi data between the branding.jar file and the branding-fragment.jar file.

The branding-fragement.jar file has:

Fragment-Host: org.glassfish.branding

but the bundle symbolic name of the branding.jar file has been changed to org.glassfish.main.core.branding in the 3.1.2 release.

The Fragment-Host (defined in osgi.bundle) has to be changed to match that of branding.jar.

Comment by Tom Mueller [ 26/Jan/12 ]
  • What is the impact on the customer of the bug?

How likely is it that a customer will see the bug and how serious is the bug?
Is it a regression? Does it meet other bug fix criteria (security, performance, etc.)?

This is a regression. The customer will see the bug if running the version command, and the wrong version will show up in the console.

  • What is the cost/risk of fixing the bug?

How risky is the fix? How much work is the fix? Is the fix complicated?

Minimal. Just a change to the osgi.bundle file.

  • Is there an impact on documentation or message strings?

No.

  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

Just run the version command with the server running.

  • Which is the targeted build of 3.1.2 for this fix?

20.

Note: this doesn't have to be fixed in the trunk because the trunk uses a different versioning mechanism that doesn't involve the user of a JAR file.

Comment by Tom Mueller [ 26/Jan/12 ]

Fixed on the 3.1.2 branch (value-add) in revision 3657.





[GLASSFISH-18253] update installer links Created: 25/Jan/12  Updated: 27/Jan/12  Resolved: 27/Jan/12

Status: Resolved
Project: glassfish
Component/s: installation
Affects Version/s: 3.1.2_b19, 4.0_b20
Fix Version/s: 3.1.2_b20, 4.0_b21

Type: Bug Priority: Major
Reporter: Romain Grécourt Assignee: Romain Grécourt
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 3_1_2-approved, glassfish, installer

 Description   

Some outdated url are used in GlassFish's installer:

  • src/GlassFishV3Preview/resources/templates/ui.prefs: [..] href=https://glassfish.dev.java.net&gt; GlassFish Project Page</a> for more information. <BR><BR>Please click Next to proceed. <BR><BR> </BODY></HTML>

These urls are used for web and full profile. Also an other version of usage metrics is used under "src/main/resources" :

We should replace them with the following:



 Comments   
Comment by Romain Grécourt [ 25/Jan/12 ]
  • What is the impact on the customer of the bug?
    Every user clicking on GlassFish's installer links will see the bug.
  • What is the cost/risk of fixing the bug?
    Low.
  • Is there an impact on documentation or message strings?
    No.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    None.
  • Which is the targeted build of 3.1.2 for this fix?
    The next one (b20 I believe)
Comment by Romain Grécourt [ 27/Jan/12 ]

I've updated the following links for 3.1.2 and trunk (external, internal and sdk workspaces):





[GLASSFISH-18246] [Regression] SDK WebServices samples (bundled with GF 3.1.2) fail to run - Failed to parse the WSDL. Created: 24/Jan/12  Updated: 01/Feb/12  Resolved: 01/Feb/12

Status: Resolved
Project: glassfish
Component/s: sample_apps
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: Alex Pineda Assignee: Romain Grécourt
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Oracle Enterprise Linux 6, JDK1.6.0_30. Used java_ee_sdk-6u4-b18-unix.sh promoted build. Firefox Browser 3.6.25. Default "typical" installation with no password. CLASSPATH=$S1AS_HOME/lib/javaee.jar


Tags: 3_1_2-approved

 Description   

The WebServices samples worked and were functional with promoted build 16.

The problem is seen with promoted build 18, when "ant run" target is invoked. The sequence of steps are as follows (the following is for hello-jaxws2.2 sample),
o machine$ cd $S1AS_HOME/sample/javaee/webservices/hello-jaxws2.2
o ant compile
BUILD SUCCESSFUL
o ant package
BUILD SUCCESSFUL
o ant run
BUILD FAIL

The error message seen is

setup:
[mkdir] Created dir: /home/test/workspace/glassfish3/glassfish/samples/javaee6/webservices/hello-jaxws2.2/hello-jaxws-client/build/classes

jaxws:
[wsimport] Consider using <depends>/<produces> so that wsimport won't do unnecessary compilation
[wsimport] parsing WSDL...
[wsimport]
[wsimport]
[wsimport] [ERROR] http://localhost:8080/hello-jaxws2.2/HelloService?WSDL is unreachable
[wsimport]
[wsimport]
[wsimport] Failed to parse the WSDL.
[wsimport] Command invoked: wsimport /usr/local/tools/jdk1.6.0_30/jre/bin/java -d /home/ap2257/workspace/glassfish3/glassfish/samples/javaee6/webservices/hello-jaxws2.2/hello-jaxws-client/build/classes -g -Xendorsed -keep http://localhost:8080/hello-jaxws2.2/HelloService?WSDL -p endpoint
[wsimport] classLoader = java.net.URLClassLoader@54c4ad
[wsimport] SharedSecrets.getJavaNetAccess()=java.net.URLClassLoader$7@1568fb5

BUILD FAILED
/home/test/workspace/glassfish3/glassfish/samples/javaee6/webservices/hello-jaxws2.2/build.xml:61: The following error occurred while executing this line:
/home/test/workspace/glassfish3/glassfish/samples/javaee6/webservices/hello-jaxws2.2/hello-jaxws-client/build.xml:70: wsimport failed

The same error is seen on the other two samples (hello-singleton-ejb, hello-webserviceref)



 Comments   
Comment by scatari [ 25/Jan/12 ]

Romain, please evaluate to see if a workaround can be documented.

Comment by Romain Grécourt [ 26/Jan/12 ]

A possible workaround would be to provide "deployment.target" at comandline.
For install ant -Ddeployment.target=server run.

Comment by Romain Grécourt [ 26/Jan/12 ]

What is the impact on the customer of the bug?

User following instructions will not be able to run the sample.
It is a regression.

What is the cost/risk of fixing the bug?
The fix is very simple.
Is there an impact on documentation or message strings?
No.
Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
Try to run the sample again.
Which is the targeted build of 3.1.2 for this fix?
b20.





[GLASSFISH-18245] [Regression] SDK sample EJB Timer (bundled with GF 3.1.2) fails to run - javax.naming.NamingException: Lookup failed Created: 24/Jan/12  Updated: 01/Feb/12  Resolved: 01/Feb/12

Status: Resolved
Project: glassfish
Component/s: sample_apps
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: Alex Pineda Assignee: Romain Grécourt
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Oracle Enterprise Linux 6, JDK1.6.0_30. Used java_ee_sdk-6u4-b18-unix.sh promoted build. Firefox Browser 3.6.25. Default "typical" installation with no password. CLASSPATH=$S1AS_HOME/lib/javaee.jar


Tags: 3_1_2-approved

 Description   

The EJB Timer sample worked and was functional with promoted build 16.

The problem is seen with promoted build 18, when "ant run" target is invoked. The sequence of steps are:
o machine$ cd $S1AS_HOME/sample/javaee/ejb/automatic-timer
o ant compile
BUILD SUCCESSFUL
o ant package
BUILD SUCCESSFUL
o ant run
BUILD FAIL

The error message seen is as follows:
runjavaclient:
[java] Waiting for the timer to expire
[java] Logged timeouts :
[java] javax.naming.NamingException: Lookup failed for 'java:global/automat ic-timer-ejb/StatelessSessionBean' in SerialContext[myEnv=

{java.naming.factory.i nitial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.fact ory.url.pkgs=com.sun.enterprise.naming, java.naming.factory.state=com.sun.corba. ee.impl.presentation.rmi.JNDIStateFactoryImpl}

[Root exception is javax.naming.N ameNotFoundException: automatic-timer-ejb]
[java] at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialCon text.java:518)
[java] at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialCon text.java:455)
[java] at javax.naming.InitialContext.lookup(InitialContext.java:392)
[java] at enterprise.automatic_timer_client.AutomaticTimerJavaClient.ge tRecords(AutomaticTimerJavaClient.java:64)
[java] at enterprise.automatic_timer_client.AutomaticTimerJavaClient.ma in(AutomaticTimerJavaClient.java:53)
[java] Caused by: javax.naming.NameNotFoundException: automatic-timer-ejb
[java] at com.sun.enterprise.naming.impl.TransientContext.resolveContex t(TransientContext.java:310)
[java] at com.sun.enterprise.naming.impl.TransientContext.lookup(Transi entContext.java:218)
[java] at com.sun.enterprise.naming.impl.TransientContext.lookup(Transi entContext.java:219)
[java] at com.sun.enterprise.naming.impl.SerialContextProviderImpl.look up(SerialContextProviderImpl.java:77)
[java] at com.sun.enterprise.naming.impl.RemoteSerialContextProviderImp l.lookup(RemoteSerialContextProviderImpl.java:109)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces sorImpl.java:39)
[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMet hodAccessorImpl.java:25)
[java] at java.lang.reflect.Method.invoke(Method.java:597)
[java] at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatch ToMethod(ReflectiveTie.java:144)
[java] at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke( ReflectiveTie.java:174)
[java] at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherIm pl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528)
[java] at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherIm pl.dispatch(CorbaServerRequestDispatcherImpl.java:199)
[java] at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handl eRequestRequest(CorbaMessageMediatorImpl.java:1624)
[java] at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handl eRequest(CorbaMessageMediatorImpl.java:1486)
[java] at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handl eInput(CorbaMessageMediatorImpl.java:990)
[java] at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_ 1_2.callback(RequestMessage_1_2.java:214)
[java] at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handl eRequest(CorbaMessageMediatorImpl.java:742)
[java] at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispa tch(CorbaMessageMediatorImpl.java:539)
[java] at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWor k(CorbaMessageMediatorImpl.java:2324)
[java] at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$Worke rThread.performWork(ThreadPoolImpl.java:497)
[java] at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$Worke rThread.run(ThreadPoolImpl.java:540)
[java] Exception in thread "main" java.lang.NullPointerException
[java] at enterprise.automatic_timer_client.AutomaticTimerJavaClient.ma in(AutomaticTimerJavaClient.java:54)



 Comments   
Comment by scatari [ 25/Jan/12 ]

Romain, please evaluate to see if a workaround can be documented.

Comment by Romain Grécourt [ 26/Jan/12 ]

A possible workaround would be to provide "deployment.target" at comandline.
For instance: ant -Ddeployment.target=server run.

Comment by Romain Grécourt [ 26/Jan/12 ]

What is the impact on the customer of the bug?

User following instructions will not be able to run the sample.
It is a regression.

  • What is the cost/risk of fixing the bug?
    The fix is very simple.
  • Is there an impact on documentation or message strings?
    No.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Try to run the sample again.
  • Which is the targeted build of 3.1.2 for this fix?
    b20.




[GLASSFISH-18241] HA run Solaris 11 - LB reconfiguration thread stops detecting new loadbalancer.xml. This around the time that DAS commands had issues. Created: 24/Jan/12  Updated: 30/Jan/12  Resolved: 30/Jan/12

Status: Closed
Project: glassfish
Component/s: load_balancer
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: kshitiz_saxena
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to GLASSFISH-18240 HA run Solaris 11 - observed DAS comm... Resolved
Tags: 312_qa

 Description   

Please see issue 18240 http://java.net/jira/browse/GLASSFISH-18240 for setup details and for initial description of this issue.

The loadbalancer reconfiguration is observed to have stopped after the point that the 2 DAS related issues were observed. Each test depends on (and triggers) reconfiguration of the loadbalancer (running on the Webserver). LB reconfiguration is triggered after deploying the application that is used by the test.

Its not clear if this reconfiguration problem is caused by the DAS related issues and so this separate bug has been filed to track the reconfiguration problem.

Issue 18240 has the relevant logs attached to it. It also has notes on the specific logs.



 Comments   
Comment by varunrupela [ 30/Jan/12 ]

This issue is not reproducible as a result of fix to issue 18240





[GLASSFISH-18240] HA run Solaris 11 - observed DAS commands take > 5 minutes, SSLHandshakeException on some DAS commands, Blocked Threads on one instance Created: 24/Jan/12  Updated: 30/Jan/12  Resolved: 30/Jan/12

Status: Resolved
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: oleksiys
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Archive File grizzly-config.jar     Zip Archive jstacks-jmaps.zip     Zip Archive logs-from-test-before-issue.zip     Zip Archive logs-from-test-when-issue-showed-up.zip     HTML File ws-error-log-after-taking-thread-dump    
Issue Links:
Related
is related to GLASSFISH-18241 HA run Solaris 11 - LB reconfiguratio... Closed
Tags: 312_qa, 3_1_2-approved

 Description   

GF Build 18
Setup: 10 instance cluster
Platform: Solaris 11
JDK: 1.6.0_30

On running of the full HA test suite, One of the MQ tests (GFMQ15LocalTest) showed the following issues:

  • Some of the DAS commands took longer than 300 seconds.
    Note: The tests invoke DAS commands via java.lang.Runtime.exec(). The destroy() method of the resulting java.lang.Process object is invoked if the DAS command takes longer than 300 seconds.
  • Following this, this test's client log shows some admin commands returning with the error: "Failure: Command delete-admin-object failed on server instance instance106: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake". This exception was also observed on many asadmin commands that were run in the tests that followed later in the run..
  • The loadbalancer reconfiguration is observed to have stopped after the point that the above issues appeared. Each test depends on (and triggers) reconfiguration of the loadbalancer (running on the Webserver). LB reconfiguration is triggered after deploying the application that is used by the test. Since its not clear if this reconfiguration problem is related to the above issues, a separate bug will be filed to track it.

More Observations (after the full HA test suite run was complete, below info collected):

  • JStack of the Webserver processes show BLOCKED threads.
  • JStack of instance106 show BLOCKED threads.

Note: These issues showed up while the MQ module of the HA tests was running. MQ tests continued to run after these issues showed up. Tests from the Coherence module also ran before the HA run completed. The JStack, JMap information was collected after completion of the HA tests run.

The following logs are being attached:

  • Logs from a test that was run before the issues appeared.
  • Logs from the test during which the issues were observed.
  • JStack and JMap of 2 instances, the domain and the webserver. JStack was taken on instance106 (as SSLHandshakeException was associated with it) and for instance101 (just to check on one other instance). Also attached a second JStack for instance106. It was taken about 8 minutes after the first one.
  • The MQ Broker process associated with instance106 was found to be running after the completion of all the HA tests. A Jstack and jmap of that broker process is also attached.
  • Webserver log is also attached. A thread dump was saved on to the Webserver instance's error log when running jstack on it.

Note regarding the attached Test logs:

  • The file ant.output contains the client logs for each test. Browsing this file will show all the das commands (look for "30000 milliseconds" to find commands that took longer than 300 seconds to run) & mq commands that were run during the particular test
  • The instance logs are here - st-cluster/<instance-name>/logs/server.log
  • The MQ logs are here - st-cluster/<instance-name>/log.txt
  • The Webserver (i.e. Loadbalancer) logs are under sjsws.

Please send a mail to receive information on:

  • VNC Details of the setup
  • Link to results from the full HA test suite run.
  • Information on the order in which tests are being run.


 Comments   
Comment by Tom Mueller [ 24/Jan/12 ]

Instance106 failed to response to a stop-instance command. The DAS received an SSLHandshakeException when trying to shut down instance106.

The jstack from instance106 shows 10 threads that are blocked with this stack trace:

"http-thread-pool-24849(5)" daemon prio=3 tid=0x01d5a400 nid=0x57 waiting for monitor entry [0xcb5bf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.sun.grizzly.config.GrizzlyEmbeddedHttps$LazySSLInitializationFilter.execute(GrizzlyEmbeddedHttps.java:200)
	- waiting to lock <0xdc8cafd0> (a $Proxy67)
	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
	at java.lang.Thread.run(Thread.java:662)

Please investigate whether there might be a deadlock possibility in the LazySSLInitializationFilter.execute code.

Comment by oleksiys [ 25/Jan/12 ]

yeah, there is non-synchronized access to HashMap: puts and gets... looks like HashMap internal pointers got broken because of that.
I'll need to fix that in Grizzly.

Comment by oleksiys [ 25/Jan/12 ]

fixed in grizzly
http://java.net/jira/browse/GRIZZLY-1182

Comment by oleksiys [ 25/Jan/12 ]

In order to fix the issue we need to integrate Grizzly 1.9.46

  • What is the impact on the customer of the bug?
    This issue is there for a long time. May occur in situation when GF is configured w/ 2+ HTTPS listeners and clients make request to to these listeners at the same time.
  • What is the cost/risk of fixing the bug?
    We need to synchronize assess to Ciphers map. Risk is very low.

How risky is the fix? How much work is the fix? Is the fix complicated?
Low risk.

  • Is there an impact on documentation or message strings?
    No.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Same set of tests QA used to discover the issue.
  • Which is the targeted build of 3.1.2 for this fix?
    b20
Comment by oleksiys [ 26/Jan/12 ]

patch for grizzly 3.1.2

Comment by oleksiys [ 30/Jan/12 ]

resolved.

Project: glassfish
Repository: svn
Revision: 52346
Author: oleksiys
Date: 2012-01-30 18:48:17 UTC
Link:

Log Message:
------------
+ fix issue #18240
http://java.net/jira/browse/GLASSFISH-18240

"HA run Solaris 11 - observed DAS commands take > 5 minutes, SSLHandshakeException on some DAS commands, Blocked Threads on one instance"

Integrating Grizzly 1.9.46

Approved by Joe





[GLASSFISH-18216] Unlocalized strings in the a few related HTTP Load Balancers pages Created: 19/Jan/12  Updated: 08/Feb/12  Resolved: 26/Jan/12

Status: Closed
Project: glassfish
Component/s: l10n
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: sunny-gui Assignee: gmurr
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Server OS: OEL 6 x64 w/JRockit 28.2.0 32-bit
Bundle: ogs-3.1.2-web-b18-ml.zip
Server Locale: fr_FR.UTF-8
Browser Locale: zh_CN


Attachments: Zip Archive LoadBalancers_unloca.zip    
Tags: 3_1_2-review

 Description   

Unlocalized strings in the a few related HTTP Load Balancers pages

To reproduce:
1. Log into Admin Console page.
2. Go to HTTP Load Balancers in the left pane.

There are unlocalized strings in the following pages.
HTTP Load Balancers
New Load Balancers
General Tab
Settings Tab
Targets Tab, Manage Targets
Export Tab, Apply Changes Now

Attached zip file for all of the screen shots.



 Comments   
Comment by gmurr [ 19/Jan/12 ]

Will fix it with the new translations

Comment by gmurr [ 26/Jan/12 ]

fix will be available in 3.1.2_b20

Comment by sunny-gui [ 08/Feb/12 ]

Verified and fixed in build 21 with the bundle "ogs-3.1.2-b21-unix-ml.sh" in OEL 6 x64 w/JDK1.6.0_30 32-Bit, so close this issue.





[GLASSFISH-18184] Win 2008. "delete-local-instance <instance_name>" created an error: com.sun.enterprise.config.serverbeans.Server : wrong number of arguments Created: 13/Jan/12  Updated: 31/Jan/12  Resolved: 31/Jan/12

Status: Resolved
Project: glassfish
Component/s: other
Affects Version/s: 3.1.2_b17
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: easarina Assignee: Mahesh Kannan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File setup_cl.pl     File unsetup_cl.pl    
Tags: 3_1_2-review, 3_1_2_qa

 Description   

Win 2008 GF 3.1.2 b17. Created a cluster with two instances, started instances, then stopped a cluster and executed

delete-local-instance <instance_name>

That command returned the follow error:

remote failure: Exception while deleting the configuration com.sun.enterprise.config.serverbeans.Server : wrong number of arguments
Command delete-local-instance failed.

This problem happened sometimes, but if it happened, all such commands return this error, until DAS will be restarted.

I did not see this issue for previous GF 3.1.2 builds, but constantly see this issue for b17 on the different Win 2008 machines. In all cases was used JDK 1.7.0_02.

To reproduce the issue, I've run many times these two attached scripts: setup_cl.pl, unsetup_cl.pl.

In server.log I saw only the same error message. See bellow the debug output:

==================================================================================
C:\hudson\workspace\Cluster>glassfish3\glassfish\bin\asadmin delete-local-instance my-in1
CLASSPATH= C:\hudson\workspace\Cluster\glassfish3\glassfish\bin\..\modules\admin-cli.jar
Commands: [delete-local-instance, my-in1]
asadmin extension directory: C:\hudson\workspace\Cluster\glassfish3\glassfish\lib\asadmin
Prepare
Process program options
Parsing program options
Parse command options
params: {}
operands: [my-in1]
Prevalidate command options
Inject command options
Validate command options
nodeDirChild: C:\hudson\workspace\Cluster\glassfish3\glassfish\nodes\localhost-domain1
instanceDir: C:\hudson\workspace\Cluster\glassfish3\glassfish\nodes\localhost-domain1\my-in1
asadmin --host localhost --port 4848 --interactive=true --echo=false --terse=false delete-local-instance my-in1
Execute command
Prepare
Parsing program options
URI: /__asadmin/uptime?Xhelp=true
URL: com.sun.enterprise.admin.util.HttpConnectorAddress@21f4c81c
URL: http://localhost:4848/__asadmin/uptime?Xhelp=true
Password options: null
Using auth info: User: null, Password: <null>
Response Content-Type: text/xml
------- RAW METADATA RESPONSE ---------
<?xml version="1.0" encoding="UTF-8" standalone="no"?><action-report description="uptime help" exit-code="SUCCESS"><message-part message=""><property name="Gene
ratedHelp" value="true"/><command name="uptime"><option default="false" name="milliseconds" optional="true" type="BOOLEAN"/></command></message-part></action-re
port>
------- RAW METADATA RESPONSE ---------
fetchCommandModel: got command opts: com.sun.enterprise.admin.util.CommandModelData@caad9a
doHttpCommand succeeds
Process program options
Parsing program options
Parse command options
params: {milliseconds: [true]
}
operands: []
Prevalidate command options
Inject command options
Validate command options
asadmin --host localhost --port 4848 --interactive=true --echo=false --terse=false uptime --milliseconds=true
Execute command
doUpload set to false
URI: /__asadmin/uptime?milliseconds=true
URL: com.sun.enterprise.admin.util.HttpConnectorAddress@6a13a848
URL: http://localhost:4848/__asadmin/uptime?milliseconds=true
Password options: null
Using auth info: User: null, Password: <null>
------- RAW RESPONSE ---------
Signature-Version: 1.0
message: 318834
milliseconds_value: 318834
keys: milliseconds
milliseconds_name: milliseconds
use-main-children-attribute: false
exit-code: SUCCESS

------- RAW RESPONSE ---------
PROCESSING MANIFEST...
doHttpCommand succeeds
server uptime: 318834
Prepare
Parsing program options
URI: /__asadmin/get?Xhelp=true
URL: com.sun.enterprise.admin.util.HttpConnectorAddress@2d14a694
URL: http://localhost:4848/__asadmin/get?Xhelp=true
Password options: null
Using auth info: User: null, Password: <null>
Response Content-Type: text/xml
------- RAW METADATA RESPONSE ---------
<?xml version="1.0" encoding="UTF-8" standalone="no"?><action-report description="get help" exit-code="SUCCESS"><message-part message=""><property name="Generat
edHelp" value="true"/><command name="get"><option default="false" name="monitor" optional="true" short="m" type="BOOLEAN"/><operand max="1" min="1" name="patter
n" type="STRING"/></command></message-part></action-report>
------- RAW METADATA RESPONSE ---------
fetchCommandModel: got command opts: com.sun.enterprise.admin.util.CommandModelData@26c455ab
doHttpCommand succeeds
Process program options
Parsing program options
Parse command options
params: {}
operands: [servers.server.my-in1]
Prevalidate command options
Inject command options
Validate command options
asadmin --host localhost --port 4848 --interactive=true --echo=false --terse=false get --monitor=false servers.server.my-in1
Execute command
doUpload set to false
URI: /__asadmin/get?DEFAULT=servers.server.my-in1
URL: com.sun.enterprise.admin.util.HttpConnectorAddress@23d4616f
URL: http://localhost:4848/__asadmin/get?DEFAULT=servers.server.my-in1
Password options: null
Using auth info: User: null, Password: <null>
------- RAW RESPONSE ---------
Signature-Version: 1.0
message:
children: servers.server.my-in1.config-ref=my-c1-config;servers.server
.my-in1.lb-weight=100;servers.server.my-in1.name=my-in1;servers.serve
r.my-in1.node-ref=localhost-domain1
use-main-children-attribute: true
children-type: null
exit-code: SUCCESS

Name: servers.server.my-in1.node-ref=localhost-domain1
message:

Name: servers.server.my-in1.name=my-in1
message:

Name: servers.server.my-in1.lb-weight=100
message:

Name: servers.server.my-in1.config-ref=my-c1-config
message:

------- RAW RESPONSE ---------
PROCESSING MANIFEST...
doHttpCommand succeeds
Prepare
Parsing program options
URI: /__asadmin/_unregister-instance?Xhelp=true
URL: com.sun.enterprise.admin.util.HttpConnectorAddress@4c48d0c9
URL: http://localhost:4848/__asadmin/_unregister-instance?Xhelp=true
Password options: null
Using auth info: User: null, Password: <null>
Response Content-Type: text/xml
------- RAW METADATA RESPONSE ---------
<?xml version="1.0" encoding="UTF-8" standalone="no"?><action-report description="_unregister-instance help" exit-code="SUCCESS"><message-part message=""><prope
rty name="GeneratedHelp" value="true"/><command name="_unregister-instance"><option name="node" optional="true" type="STRING"/><option name="target" optional="t
rue" type="STRING"/><operand max="1" min="1" name="name" type="STRING"/></command></message-part></action-report>
------- RAW METADATA RESPONSE ---------
fetchCommandModel: got command opts: com.sun.enterprise.admin.util.CommandModelData@4083633f
doHttpCommand succeeds
Process program options
Parsing program options
Parse command options
params: {node: [localhost-domain1]
}
operands: [my-in1]
Prevalidate command options
Inject command options
Validate command options
asadmin --host localhost --port 4848 --interactive=true --echo=false --terse=false _unregister-instance --node localhost-domain1 my-in1
Execute command
doUpload set to false
URI: /__asadmin/_unregister-instance?node=localhost-domain1&DEFAULT=my-in1
URL: com.sun.enterprise.admin.util.HttpConnectorAddress@71e8de2f
URL: http://localhost:4848/__asadmin/_unregister-instance?node=localhost-domain1&DEFAULT=my-in1
Password options: null
Using auth info: User: null, Password: <null>
------- RAW RESPONSE ---------
Signature-Version: 1.0
message: Exception while deleting the configuration com.sun.enterprise
.config.serverbeans.Server : wrong number of arguments
use-main-children-attribute: false
exit-code: FAILURE

------- RAW RESPONSE ---------
PROCESSING MANIFEST...
remote failure: Exception while deleting the configuration com.sun.enterprise.config.serverbeans.Server : wrong number of arguments
Command delete-local-instance failed.



 Comments   
Comment by easarina [ 13/Jan/12 ]

If between instance creation and instance deleting many other commands were executed, then, with a big probability, this problem will be seen. At least, when I've executed an automated test, the problem was seen in 4 executions from 5 executions totally. And I did not see this issue for the previous builds.

Comment by Tom Mueller [ 13/Jan/12 ]

What other commands?

Comment by easarina [ 13/Jan/12 ]

There were a lot of the different commands. But I was able to reproduce the issue just using create and delete instance commands, see the attached scripts.

Comment by Tom Mueller [ 13/Jan/12 ]

The root cause of this issue is in the HK2 ConfigSupport._deleteChild method.

This method has the following code:

     for (Method m : parentProxyType.getMethods()) {
         final Class returnType = m.getReturnType();
         if (Collection.class.isAssignableFrom(returnType)) {
              // this could be it...
              if (!(m.getGenericReturnType() instanceof ParameterizedType))
                     throw new IllegalArgumentException("List needs to be parameterized");
              final Class itemType = Types.erasure(Types.getTypeArgument(m.getGenericReturnType(), 0));
              if (itemType.isAssignableFrom(childType)) {

Given a childType such as Server, this code is looking for a method such as:

    List<Server> getServer()

However, the Servers config bean, which is being operated on in this case, has more than one method that returns List<Server>

interface Servers ... {
    public List<Server> getServer();
    public List<Server> getServersOnNode(Node node);
}

If the loop encounters the 2nd method first, an "IllegalArgumentException: wrong number of arguments" is thrown because the code try to invoke the method with no arguments but getServersOnNode takes one argument. If the loop encounters the 1st method first, then it works fine.

Suggested fix: add a check to the loop to make sure the found method takes the expected number of parameters (0).

Comment by Tom Mueller [ 13/Jan/12 ]

Assigning to Mahesh since this is an HK2 problem.

The bug is in the HK2 config module, in the ConfigSupport.java file at about line 792. To fix the bug, a check has to be added to see if m.getParameterTypes().length == 0.

To reproduce the problem, you have to run the script about 10-20 times.

Comment by Mahesh Kannan [ 31/Jan/12 ]

svn diff
Index: config/src/main/java/org/jvnet/hk2/config/ConfigSupport.java
===================================================================
— config/src/main/java/org/jvnet/hk2/config/ConfigSupport.java (revision 2533)
+++ config/src/main/java/org/jvnet/hk2/config/ConfigSupport.java (working copy)
@@ -787,7 +787,7 @@
// type will not work.
for (Method m : parentProxyType.getMethods()) {
final Class returnType = m.getReturnType();

  • if (Collection.class.isAssignableFrom(returnType)) {
    + if (Collection.class.isAssignableFrom(returnType) && (m.getParameterTypes().length == 0)) {
    // this could be it...
    if (!(m.getGenericReturnType() instanceof ParameterizedType))
    throw new IllegalArgumentException("List needs to be parameterized");

svn commit -m "Fix for 18184. Admin/cli tests passed. Approved by Joe Di Pol, Reviewed by Tom Muller"
Sending config/src/main/java/org/jvnet/hk2/config/ConfigSupport.java
Transmitting file data .
Committed revision 2539.

Approved by: Joe Di Pol

What is the impact on the customer of the bug?
This problem happened sometimes, but if it happened, all such commands return this error, until DAS will be restarted.

What is the cost/risk of fixing the bug?
Not much. Touched one file and one line

Is there an impact on documentation or message strings?
Not for documentation.

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
The one listed in the bug report itself

Which is the targeted build of 3.1.2 for this fix?
3_1_2_b20

Comment by Mahesh Kannan [ 31/Jan/12 ]

Integrated new HK2 version

svn commit -m "Integrate new hk2 version (fix for 18184). Approved by Joe. QL Passed"
Sending pom.xml
Transmitting file data .
Committed revision 52356.





[GLASSFISH-18093] SDK sample, servletcontainerinitializer-war, fails to run Created: 28/Dec/11  Updated: 01/Feb/12  Resolved: 01/Feb/12

Status: Resolved
Project: glassfish
Component/s: sample_apps
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: Alex Pineda Assignee: Romain Grécourt
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Installed java_ee_sdk-6u4-b16-unix.sh on Mac system (MacOS 10.7.2). Using Firefox 3.6.2 browser. Same issue is seen on other Linux systems.


Tags: 3_1_2-approved

 Description   

After installing and configuring $S1AS_HOME/samples/bp-project/build.properties file to point to the correct location of the GlassFish libraries, using the SDK build. An attempt to run the $S1AS_HOME/samples/javaee6/web/servlet/servletcontainerinitializer-war, fails with the following error:

stop-domain:
[echo] Stopping default domain for /home/sqe/workspace/glassfish3/glassfish
[exec] Waiting for the domain to stop ...
[exec] Command stop-domain executed successfully.
[javac] Compiling 1 source file to /home/sqe/workspace/glassfish3/glassfish/samples/javaee6/web/servlet/servletcontainerinitializer-war/containerinitializerlibrary/build
[javac] /home/sqe/workspace/glassfish3/glassfish/samples/javaee6/web/servlet/servletcontainerinitializer-war/containerinitializerlibrary/src/web/servlet/servletcontainerinitializer_lib/SharedLibrary.java:44: package javax.servlet does not exist
[javac] import javax.servlet.*;
[javac] ^
[javac] /home/sqe/workspace/glassfish3/glassfish/samples/javaee6/web/servlet/servletcontainerinitializer-war/containerinitializerlibrary/src/web/servlet/servletcontainerinitializer_lib/SharedLibrary.java:45: package javax.servlet.annotation does not exist
[javac] import javax.servlet.annotation.*;
[javac] ^
[javac] /home/sqe/workspace/glassfish3/glassfish/samples/javaee6/web/servlet/servletcontainerinitializer-war/containerinitializerlibrary/src/web/servlet/servletcontainerinitializer_lib/SharedLibrary.java:48: cannot find symbol
[javac] symbol: class ServletContainerInitializer
[javac] public class SharedLibrary implements ServletContainerInitializer {
[javac] ^
[javac] /home/sqe/workspace/glassfish3/glassfish/samples/javaee6/web/servlet/servletcontainerinitializer-war/containerinitializerlibrary/src/web/servlet/servletcontainerinitializer_lib/SharedLibrary.java:47: cannot find symbol
[javac] symbol: class HandlesTypes
[javac] @HandlesTypes(WebServlet.class)
[javac] ^
[javac] /home/sqe/workspace/glassfish3/glassfish/samples/javaee6/web/servlet/servletcontainerinitializer-war/containerinitializerlibrary/src/web/servlet/servletcontainerinitializer_lib/SharedLibrary.java:50: cannot find symbol
[javac] symbol : class ServletContext
[javac] location: class web.servlet.servletcontainerinitializer_lib.SharedLibrary
[javac] public void onStartup(Set<Class<?>> c, ServletContext ctx) {
[javac] ^
[javac] 5 errors

BUILD FAILED

The exact commands executed were:

  • machine $ cd $S1AS_HOME/samples/javaee6/web/servlet/servletcontainerinitializer-war
  • machine $ asadmin start-domain
  • machine $ ant compile
    BUILD SUCCESS
  • machine $ ant package
    BUILD SUCCESS
  • machine $ ant run
    BUILD FAILED


 Comments   
Comment by scatari [ 28/Dec/11 ]

Please make sure that <InstallHome>/glassfish/lib/javaee.jar is included in the CLASSPATH as a workaround. We will scan our samples scripts to see if this is done.

Comment by Alex Pineda [ 28/Dec/11 ]

It would be interesting to know why this is happening as the (Hudson) test scripts have worked with all previous releases (3.1.1) and builds (b13). I will use the suggested workaround as a temporary solution until a decision or conclusion is made.

Comment by Alex Pineda [ 28/Dec/11 ]

Seeing the same problem on the Web SDK distribution (javaee_sdk_6u4-web-b16-jdk7-solarix-x86.sh) with this sample. The suggested workaround resolves the "run" issue.

Comment by Alex Pineda [ 18/Jan/12 ]

Not sure what excluding the issue on 3.1.2 means. Did you scan the samples? Do I need to apply the workaround as a the solution to this issue? Please advise.

Comment by Romain Grécourt [ 24/Jan/12 ]

For the being I'm not sure I'll have time to fix this (I've reproduced the issue, but I've not investigated the right fix yet). Let me talk to Sathyan to see if we can do this for 3.1.2.

Comment by Romain Grécourt [ 26/Jan/12 ]

The cause of this issue is the following:

<javac srcdir="./containerinitializerlibrary/src"
                destdir="./containerinitializerlibrary/build"
                classpath="${javaee.home}/modules/javax.servlet.jar"
                debug="on"
                failonerror="true"/>

javax.servlet.jar no more exists, it has been renamed to "javax.servlet-api.jar".

Comment by Romain Grécourt [ 26/Jan/12 ]

What is the impact on the customer of the bug?

User following instructions will not be able to run the sample.

What is the cost/risk of fixing the bug?
The fix is very simple (a har was renamed).

Is there an impact on documentation or message strings?
No.

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
Try to run the sample again.

Which is the targeted build of 3.1.2 for this fix?
b20.





[GLASSFISH-18054] [508] Layout Table in the Login Page not defined clearly. Created: 20/Dec/11  Updated: 25/Jan/12  Resolved: 25/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b14
Fix Version/s: 3.1.2_b17, 3.1.2_b20, 4.0

Type: Bug Priority: Minor
Reporter: shaline Assignee: andriy.zhdanov
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: solaris sparc 10
FF 8.0.1
GF 3.1.2 SCF b14


Attachments: File common-tasks-layout-table.bmp     JPEG File edit-node.JPG     JPEG File login.JPG     JPEG File new-jdbc-pool.JPG     File new-jdbc-resource.bmp     JPEG File new-load-balancer.JPG     JPEG File server-general-information.JPG     File wodstock.diff     File woodstock2.diff     File working-internal.diff     File working.diff    
Issue Links:
Related
is related to GLASSFISH-18252 [508] Layout Table in the CommonTasks... Resolved
Tags: 3_1_2-508, 3_1_2_approved

 Description   

After we enable secure admin, and access console from a remote browser, the Login Screen in console has a layout table which could be confused by screen reader as data table.OGHAG toolbar Table Analyzer reports ERROR for the Layout table in the Login page:
Attached the OGHAG Table Report for the login page:



 Comments   
Comment by shaline [ 22/Dec/11 ]

Few more tables in the Console have this issue in 3.1.2 SCF b15.
They are:
Add Resources Page
New Node Page
New JDBC Pool Page
New JDBC resource Page
Edit Node Page
New Http Load Balancer Page
Common Tasks Page.
Attached the Screen shots for these pages for reference.

Comment by andriy.zhdanov [ 04/Jan/12 ]

Thought fix based on http://webaim.org/standards/508/checklist:

Either add role="presentation" to a table or appropriate <th> element

Comment by andriy.zhdanov [ 06/Jan/12 ]
  • What is the impact on the customer of the bug?

Minimal (508 issue).

  • What is the cost/risk of fixing the bug?

No risk.

  • Is there an impact on documentation or message strings?

No

  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

Sanity test login page.

  • Which is the targeted build of 3.1.2 for this fix?

3_1_2_b17

Comment by andriy.zhdanov [ 06/Jan/12 ]

Fix in woodstock (branches/Woodstock_402_Branch):

Committed revision 1599

Comment by andriy.zhdanov [ 06/Jan/12 ]

Fixed loginform in glassfish:

Committed revision 51930.
Committed revision 51931.
Committed revision 51932.

Comment by Anissa Lam [ 09/Jan/12 ]

Woodstock 4.0.2.9 has been integrated to Glassfish on both 3.1.2 branch and trunk.
Marking the bug fixed.

Comment by shaline [ 19/Jan/12 ]

I see the error from the OGHAG table inspector , for the Login Page and the Common Tasks Window. In the rest of the screens reported in the issue, like New Nodes page, New JDBC Pools page etc are all fixed. For Login Page and Common Tasks, the error still shows up on GF 3.1.2 Promoted b18.

Comment by andriy.zhdanov [ 24/Jan/12 ]

Fix for OGF loginpage. The fix for commonTasks table must be in woodstock.

  • What is the impact on the customer of the bug?

Minimal (508 issue).

  • What is the cost/risk of fixing the bug?

No risk.

  • Is there an impact on documentation or message strings?

No

  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

Sanity test login page.

  • Which is the targeted build of 3.1.2 for this fix?

3_1_2_b20

Comment by andriy.zhdanov [ 25/Jan/12 ]

Fix for CommonTasks section in woodstock

Comment by andriy.zhdanov [ 25/Jan/12 ]

Fixed login page in OGF:
Committed revision 3653
Committed revision 3654

Fix for CommonTasks is tracked in separate issue GLASSFISH-18252





[GLASSFISH-17971] classloader errors on instance restart or deployment after a cluster restart Created: 10/Dec/11  Updated: 27/Jan/12  Resolved: 27/Jan/12

Status: Resolved
Project: glassfish
Component/s: jca
Affects Version/s: 3.1.2_b14
Fix Version/s: 3.1.2_b20, 4.0_b22

Type: Bug Priority: Major
Reporter: sherryshen Assignee: Jagadish
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHL 5 and JDK 1.6.0_24


Tags: 312_qa, 3_1_2-approved

 Description   

3.1.2 nightly build b14-12_09_2011

The intermittent failures were observed in tx recovery tests of jdbc/jms app with dblog
while verifying the fix for GLASSFISH-17789. When a recovery test failed, db lock
was not released and caused the failure of later tests. Since the intermittent failures
apear often and affected the test execution. File a p2 bug to track issue.
I have sent test details in emails to dev and will add related part to this bug.



 Comments   
Comment by sherryshen [ 10/Dec/11 ]

auto-delegated recovery of jdbc/jms app with dblog
and test source in
appserver-sqe/pe/transaction/recovery/cliapp3

This 1 test is compared w.r.t. different builds from my observation.
1) promoted build 10, the test passed consistently.
2) nightly build 11 with the fix for self recovery, e.g. b11-11_21_2011,
the test failed consistently.
filed GLASSFISH-17789 and verified multiple fixes on later build.
3) nightly build 14 with the fix for GLASSFISH-17789, e.g. b14-12_09_2011
the test passed in some runs, and failed in other runs.
filed this bug, GLASSFISH-17971

The examples of passed and failed cases on 14-12_09_2011 are given below.
Run A. passed case.
http://sqe-hudson.us.oracle.com:8080/hudson/view/Sherry_core/job/sherry-tx-lc-dbg/
Good news is that all tx recovery tests with db log in this run.

Run B. failed case
I stopped the hudson job manually after I saw the failure so that log can be viewed easily.
Run B and Run A are in the same order of execution as below
--cliweb pass
--cliapp3 fail
--rest tests

http://sqe-hudson.us.oracle.com:8080/hudson/job/sherry-tx/ws/glassfish3/glassfish/nodes/localhost-domain1/clustered_instance_2/logs/server.log/*view*/

[#|2011-12-09T19:31:36.969-0800|INFO|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|_ThreadID=17;_ThreadName=Thread-2;|
Recovery of Inbound Transactions started.|#]

[#|2011-12-09T19:31:37.273-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|_ThreadID=17;_ThreadName=Thread-2;|
RAR8061: failed to load resource-adapter-config or RA [ __ds_jdbc_ra ], com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: No Classloader found for RA [ __ds_jdbc_ra ]|#]

[#|2011-12-09T19:31:37.274-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|_ThreadID=17;_ThreadName=Thread-2;|
RAR8060: Unable to lookup pool [ poolNontx ], javax.naming.NamingException: Lookup failed for '__SYSTEM/pools/poolNontx'
in SerialContext[myEnv=

{java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is javax.naming.NameNotFoundException: pools]|#]

[#|2011-12-09T19:31:37.274-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|_ThreadID=17;_ThreadName=Thread-2;|RAR6017 :
Failed to get connection pool object jdbc/nontx via JNDI lookup : com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Lookup failed for '__SYSTEM/pools/poolNontx' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming}

|#]

[#|2011-12-09T19:31:37.275-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.recovery|_ThreadID=17;_ThreadName=Thread-2;|
RAR7109: Error while loading jdbc resources during recovery : jdbc/nontx|#]

[#|2011-12-09T19:31:37.279-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=17;_ThreadName=Thread-2;|
JTS5014: Recoverable JTS instance, serverId = [100]|#]

[#|2011-12-09T19:31:37.317-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|_ThreadID=18;_ThreadName=Thread-2;|
RAR8061: failed to load resource-adapter-config or RA [ __ds_jdbc_ra ], com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: No Classloader found for RA [ __ds_jdbc_ra ]|#]

[#|2011-12-09T19:31:37.318-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|ThreadID=18;_ThreadName=Thread-2;|RAR8060: Unable to lookup pool [ poolNontx ], javax.naming.NamingException: Lookup failed for '_SYSTEM/pools/poolNontx' in SerialContext[myEnv=

{java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is javax.naming.NameNotFoundException: pools]|#]

[#|2011-12-09T19:31:37.318-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|ThreadID=18;_ThreadName=Thread-2;|RAR6017 : Failed to get connection pool object jdbc/nontx via JNDI lookup : com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Lookup failed for '_SYSTEM/pools/poolNontx' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming}

|#]

[#|2011-12-09T19:31:37.318-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=18;_ThreadName=Thread-2;|
JTS5070: Exception occurred while obtaining JDBC resource [jdbc/nontx] for transaction log.|#]

[#|2011-12-09T19:31:37.319-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=18;_ThreadName=Thread-2;|
javax.naming.NamingException: Lookup failed for 'jdbc/nontx' in SerialContext[myEnv=

{java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming}
[Root exception is javax.naming.NamingException: Unable to lookup resource : jdbc/nontx [Root exception is com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Lookup failed for '__SYSTEM/pools/poolNontx' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming}

]]
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:518)
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:455)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at com.sun.jts.CosTransactions.LogDBHelper.<init>(LogDBHelper.java:101)
at com.sun.jts.CosTransactions.LogDBHelper.<clinit>(LogDBHelper.java:88)
at com.sun.jts.CosTransactions.RecoveryManager.dbXARecovery(RecoveryManager.java:1180)
at com.sun.jts.CosTransactions.ResyncThread.run(RecoveryManager.java:1897)

Comment by sherryshen [ 10/Dec/11 ]

Another examples captured from previous email show that the intermittent failure also
on self-recovery of jdbc/jms app in earlier tests.

Run A:
http://sqe-hudson.us.oracle.com:8080/hudson/job/sherry-tx/22/artifact/appserver-sqe/reports/pe-ee/i386_apg-sqe2_Linux/html/test_results_txn.html
#22 b13-12_04_2011 10 tests in the following order
1) delegated and self recovery with jdbc app. (4 tests), pass
2) self recovery with jdbc/jms app (4 tests), fail
3) delegated recovery with jdbc/jms app (2 tests), fail due to GLASSFISH-17789.
I put failure tests at last since failures made db in bad condition.

Run B: new failure in self recovery and proved to be intermittent from other runs.
http://sqe-hudson.us.oracle.com:8080/hudson/job/sherry-tx/23/artifact/appserver-sqe/reports/pe-ee/i386_apg-sqe1_Linux/html/test_results_txn.html
#23 re-continuous build 403 with the fix for GLASSFISH-17789.
the same 10 tests and execution order and source as Run A.
1) delegated and self recovery with jdbc app. (4 tests), pass
2) self recovery with jdbc/jms app (4 tests), see new failure in comparing with b13-12_04_2011.
3) delegated recovery with jdbc/jms app (2 tests), possibly due to unreleased db lock in 2).

Comment by sherryshen [ 15/Dec/11 ]

Track the intermittent failure of another run here even though
the cause may be different.

On b14 promoted build, intermittent failure is observed in self recovery
of jdbc/jms app (cliapp2) with dblog.

verify-data:
[echo] Executing test on port 38080
[java] WS HOME appserver-sqe
[java] main: testSuiteID=tx-cliapp2-
[java] main: testCase=verify_xa
[java] invoking webclient at http://localhost:38080/cliapp2/verify?xa
[java] Processing line: RESULTS:0
[java] FAILURE
[java] did not get expectedResult=RESULTS:9
[java] Generating report at /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/test_results.xml
[java]
[java]
[java] -----------------------------------------
[java] - tx-cliapp2-t3mql-dblog-killASInstanceAutomaticSelfRecovery: FAIL -
[java] -----------------------------------------
[java] Total PASS: 0
[java] Total FAIL: 1
[java] Total DNR: 0
[java] -----------------------------------------

I did not find Classloader error in this run.
The test results and server.log are saved on RunID 20.
http://agni-1.us.oracle.com/JSPWiki/Wiki.jsp?page=V312CoreTest
The hudson job for this run is at
http://sqe-hudson.us.oracle.com:8080/hudson/view/Sherry_core/job/sherry-lc-oracle/7/

Comment by marina vatkina [ 15/Dec/11 ]

The test failed because instance2 was down on deploy and the insert didn't happen (see deploy output from the console log below). How can I match the timing of this test to other tests in instance2 log?

deploy-common-ee:
[echo] So you are using Enterprise Version eh ?
[exec] asadmin --host localhost --port 4848 --user admin --passwordfile /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build-config/adminpassword.txt --interactive=false --echo=true --terse=false deploy --force=false --precompilejsp=false --verify=false --retrieve /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build/ee/i386_apg-sqe2_Linux/cliapp2/archive --generatermistubs=false --availabilityenabled=false --asyncreplication=true --target sqe-cluster --keepreposdir=false --keepfailedstubs=false --isredeploy=false --logreportederrors=true /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build/ee/i386_apg-sqe2_Linux/cliapp2/archive/cliapp2App.ear
[exec] Application deployed with name cliapp2App.
[exec] WARNING: Command _deploy did not complete successfully on server instance clustered_instance_2: remote failure: Failed to load the application on instance clustered_instance_2. The application will not run properly. Please fix your application and redeploy.
[exec] Exception while loading the app : EJB Container initialization error. Please see server.log for more details.

Comment by marina vatkina [ 20/Dec/11 ]

There is something wrong with the classloaders in 3.1.2 after many restarts and redeploys.

Sherry's hudson job fails because instance can't be started (intermittently):

[#|2011-12-09T19:31:37.273-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|_ThreadID=17;_ThreadName=Thread-2;|RAR8061:
failed to load resource-adapter-config or RA [ __ds_jdbc_ra ],
com.sun.appserv.connectors.internal.api.ConnectorRuntimeException:
No Classloader found for RA [ __ds_jdbc_ra ]|#]

I see (deterministically) in transaction recovery devtests ClassNotFoundException: com.sun.corba.ee.impl.oa.rfm.ReferenceManagerConfigurator after several cluster restarts and app redeploys:

Caused by: java.lang.ClassNotFoundException: com.sun.corba.ee.impl.oa.rfm.ReferenceManagerConfigurator
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1509)
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1359)
at com.sun.corba.ee.spi.orb.ORB$2.evaluate(ORB.java:880)

To reproduce the latter:

% cd v2/appserv-tests/devtests/transaction/ee/dblogs/mdb
% ant all-local

Comment by Sanjeeb Sahoo [ 21/Dec/11 ]

I doubt very much this is a class loader issue.
Does this test pass in 3.1.1? Did this test pass in earlier builds of 3.1.2?
Where does the class com.sun.corba.ee.impl.oa.rfm.ReferenceManagerConfigurator get loaded from in environment where it worked?

Comment by marina vatkina [ 21/Dec/11 ]

This is a new test, and in 3.1.1 MDBs didn't start ORB (they don't need ORB, but JMS does, but that's subject of another issue).

The class is in glassfish-corba-orb.jar

Comment by Sanjeeb Sahoo [ 21/Dec/11 ]

No, this is not a classloader issue. Asking a class loader to load an arbitrary class can result in ClassNotFoundException. Returning to ejb team to do further evaluation. Based on what I hear from Marina, the likely place where a fix is needed is corba, but I will let Marina take the call. An important question to answer is if this a regression? In other words, does the test pass in any previous build (or version)?

Comment by marina vatkina [ 21/Dec/11 ]

It's by no means an EJB issue (though is triggered by the test that executes an app with MDBs).

The ORB class is in a private package:

glassfish-corba-orb.jar
META-INF/MANIFEST.MF:
Private-Package: <...>, com.sun.corba.ee.impl.oa.rfm, <...>
ORB-Class-Provider: com.sun.corba.ee.impl.oa.rfm.ReferenceManagerConfi
gurator, <...>

So the question is why is it being loaded by the WebappCL?

The full stack trace:
[#|2011-12-20T21:30:29.428-0800|WARNING|glassfish3.1.2|javax.enterprise.resource.corba.ORBUtil|_ThreadID=18;_ThreadName=Thread-2;|IOP01210054: Error while attempting to load class com.sun.corba.ee.impl.oa.rfm.ReferenceManagerConfigurator
org.omg.CORBA.BAD_OPERATION: WARNING: IOP01210054: Error while attempting to load class com.sun.corba.ee.impl.oa.rfm.ReferenceManagerConfigurator vmcid: OMG minor code: 54 completed: No
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:248)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:95)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.handleFullLogging(WrapperGenerator.java:387)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.access$400(WrapperGenerator.java:107)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator$2.invoke(WrapperGenerator.java:511)
at com.sun.corba.ee.spi.orbutil.proxy.CompositeInvocationHandlerImpl.invoke(CompositeInvocationHandlerImpl.java:99)
at $Proxy158.classActionException(Unknown Source)
at com.sun.corba.ee.spi.orb.OperationFactory$ClassAction.operate(OperationFactory.java:297)
at com.sun.corba.ee.spi.orb.OperationFactory$ComposeAction.operate(OperationFactory.java:520)
at com.sun.corba.ee.impl.orb.PrefixParserAction.apply(PrefixParserAction.java:96)
at com.sun.corba.ee.spi.orb.PropertyParser.parse(PropertyParser.java:84)
at com.sun.corba.ee.spi.orb.ParserImplBase.init(ParserImplBase.java:77)
at com.sun.corba.ee.impl.orb.ORBConfiguratorImpl.runUserConfigurators(ORBConfiguratorImpl.java:179)
at com.sun.corba.ee.impl.orb.ORBConfiguratorImpl.configure(ORBConfiguratorImpl.java:170)
at com.sun.corba.ee.impl.orb.ORBImpl.postInit(ORBImpl.java:625)
at com.sun.corba.ee.impl.orb.ORBImpl.set_parameters(ORBImpl.java:704)
at com.sun.corba.ee.impl.orb.ORBImpl.setParameters(ORBImpl.java:691)
at com.sun.corba.ee.spi.osgi.ORBFactory.initialize(ORBFactory.java:107)
at org.glassfish.enterprise.iiop.impl.GlassFishORBManager.initORB(GlassFishORBManager.java:581)
at org.glassfish.enterprise.iiop.impl.GlassFishORBManager.getORB(GlassFishORBManager.java:263)
at org.glassfish.enterprise.iiop.impl.GlassFishORBFactoryImpl.createORB(GlassFishORBFactoryImpl.java:93)
at org.glassfish.enterprise.iiop.api.GlassFishORBHelper.getORB(GlassFishORBHelper.java:152)
at org.glassfish.enterprise.iiop.api.GlassFishORBHelper.getProtocolManager(GlassFishORBHelper.java:219)
at com.sun.ejb.containers.BaseContainer.initializeProtocolManager(BaseContainer.java:824)
at com.sun.ejb.containers.BaseContainer.<init>(BaseContainer.java:508)
at com.sun.ejb.containers.MessageBeanContainer.<init>(MessageBeanContainer.java:142)

P.S. I do see a message similar to what Sherry obseved earlier than CNF error on the same instance startup:
[#|2011-12-20T21:29:46.664-0800|WARNING|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.recovery|_ThreadID=17;_ThreadName=Thread-2;|RAR8037: exception while getting transaction-support for RAR [ jmsra ] , com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: No Classloader found for RA [ jmsra ]|#]

Comment by Harshad Vilekar [ 24/Dec/11 ]

There is no change in packaging for glassfish-corba-orb.jar
META-INF/MANIFEST.MF in the latest build matches with 3.1.1.

I ran the test twice, and application deploy worked fine - both before and after cluster restart.

> [java] Deployed /home/hv51393/repo/v2-appserv-tests/build/module/archive/tx-ee-dblogs-mdb-web.war

This is not ORB issue.

Comment by marina vatkina [ 27/Dec/11 ]

There is no JTS in the picture, it's an ORB init error on MDB deployment.

Harshad, which condition (classActionException) gets triggered by this path:

at $Proxy158.classActionException(Unknown Source)
at com.sun.corba.ee.spi.orb.OperationFactory$ClassAction.operate(OperationFactory.java:297)
at com.sun.corba.ee.spi.orb.OperationFactory$ComposeAction.operate(OperationFactory.java:520)
at com.sun.corba.ee.impl.orb.PrefixParserAction.apply(PrefixParserAction.java:96)
at com.sun.corba.ee.spi.orb.PropertyParser.parse(PropertyParser.java:84)
at com.sun.corba.ee.spi.orb.ParserImplBase.init(ParserImplBase.java:77)
at com.sun.corba.ee.impl.orb.ORBConfiguratorImpl.runUserConfigurators(ORBConfiguratorImpl.java:179)
at com.sun.corba.ee.impl.orb.ORBConfiguratorImpl.configure(ORBConfiguratorImpl.java:170)
at com.sun.corba.ee.impl.orb.ORBImpl.postInit(ORBImpl.java:625)
at com.sun.corba.ee.impl.orb.ORBImpl.set_parameters(ORBImpl.java:704)
at com.sun.corba.ee.impl.orb.ORBImpl.setParameters(ORBImpl.java:691)
at com.sun.corba.ee.spi.osgi.ORBFactory.initialize(ORBFactory.java:107)

Comment by Harshad Vilekar [ 28/Dec/11 ]

I'm unable to duplicate the failure (Red Hat Enterprise Linux 6.1, JDK 1.6.0_30, GF 3.1.2 latest build). However, following the code path, the above exception seems to be coming from:

com.sun.corba.ee.impl.osgi.loader.OSGiListener.evaluate():196 return secureLoadClass( bundle, className ); << Throw exception here
com.sun.corba.ee.impl.osgi.loader.OSGiListener.evaluate():187 Bundle bundle = getBundleForClass( className ) ;
com.sun.corba.ee.spi.orb.OperationFactory.operate(): 294 Class<?> result = resolver.evaluate( className ) ;

> % cd v2/appserv-tests/devtests/transaction/ee/dblogs/mdb
> % ant all-local

What configuration (OS/JDK/GlassFish Build) is the test failing consistently ?

Comment by marina vatkina [ 28/Dec/11 ]

Interesting. Hudson tests on x86 passed today. I run my tests on mac

Comment by Harshad Vilekar [ 03/Jan/12 ]

Sherry, How is your test result looking with the latest 3.1.2 build ? Do you see following exception in the server log for any of the instances ?
> org.omg.CORBA.BAD_OPERATION: WARNING: IOP01210054: Error while attempting to load class

Comment by sherryshen [ 03/Jan/12 ]

Harshad, I did not notice this error in previous and recent 3.1.2 build.
> org.omg.CORBA.BAD_OPERATION: WARNING: IOP01210054: Error while attempting to load class

Comment by Harshad Vilekar [ 04/Jan/12 ]

Please reopen if the test fails again.

Comment by sherryshen [ 11/Jan/12 ]

Reopen the bug since the intermittent failures are shown in the execution of 3.1.2 b17.

1) One of previously reported intermittent failure is shown on 3.1.2
build 17 tests, e.g.
marina vatkina added a comment - 15/Dec/11 07:07 AM
The test failed because instance2 was down on deploy
The related test details are provided as below.

2) The test env can be view from hudson job.
--One run with Derby with 1 intermittent failure
http://sqe-hudson.us.oracle.com:8080/hudson/job/sherry-lc-core/

--Another run with Oracle DB with 3 intermittent failures.
http://sqe-hudson.us.oracle.com:8080/hudson/view/Sherry_core/job/sherry-lc-oracle/

3) tx test report shows what types of tests have intermittent failure
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/html/test_results_txn.html
jms/jdbc app with txlog in database has intermittent failure.
jdbc only app with txlog in database, all passed.
jms/jdbc app or jdbc only app with txlog in file system, all passed.

4) client test output shows when failure occurred.
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/output/txn_recovery_dblog.output
The tests are execution in the following order
tx-cliapp3-t1mql-dblog-killASInstanceAutomaticDelegatedRecoveryID pass
tx-cliapp2-t2mql-dblog-killASInstanceAdmincliDelegatedRecoveryID pass
tx-cliapp2-t3mql-dblog-killASInstanceAutomaticSelfRecoveryID pass
tx-cliapp2-t4mql-dblog-killMQBrokerAutomaticSelfRecovery1ID fail
tx-cliapp2-t4mql-dblog-killMQBrokerAutomaticSelfRecovery2ID fail
tx-cliapp2-t5mql-dblog-killDBServerAdmincliSelfRecoveryID fail
tx-cliweb4-t1-killASInstanceAdmincliDelegatedRecoveryID pass

From the first failure at, tx-cliapp2-t4mql-dblog-killMQBrokerAutomaticSelfRecovery1ID,
instance2 was down on deploy.

deploy-common-ee:
[echo] So you are using Enterprise Version eh ?
[exec] asadmin --host localhost --port 4848 --user admin --passwordfile /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build-config/adminpassword.txt --interactive=false --echo=true --terse=false deploy --force=false --precompilejsp=false --verify=false --retrieve /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build/ee/i386_apg-sqe2_Linux/cliapp2/archive --generatermistubs=false --availabilityenabled=false --asyncreplication=true --target sqe-cluster --keepreposdir=false --keepfailedstubs=false --isredeploy=false --logreportederrors=true /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build/ee/i386_apg-sqe2_Linux/cliapp2/archive/cliapp2App.ear
[exec] Application deployed with name cliapp2App.
[exec] WARNING: Command _deploy did not complete successfully on server instance clustered_instance_2: remote failure: Failed to load the application on instance clustered_instance_2. The application will not run properly. Please fix your application and redeploy.
[exec] Exception while loading the app : EJB Container initialization error. Please see server.log for more details.
[exec] Command deploy completed with warnings.

[echo] Deployment on target server sqe-cluster successful

5) Related server.log
Some server.log files are saved at
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/log/

http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/log/server_in2.log_2012-01-11T11-31-37_dbg
--deployed normally for cliapp2 t3

[#|2012-01-11T09:50:34.619-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=21;_ThreadName=Thread-2;|cliapp2App was successfully deployed in 6,704 milliseconds.|#]

--Failure appeared, after that deploy has failure on cliapp2 t4.

[#|2012-01-11T09:56:14.803-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.ejb.com.sun.ejb.containers|
_ThreadID=21;_ThreadName=Thread-2;|EJB5181:Portable JNDI names for EJB SessionBean: [java:global/cliapp2App/cliapp2-ejb/SessionBean, java:global/cliapp2App/cliapp2-ejb/SessionBean!tx.recovery.cliapp2.SessionLocal]|#]

[#|2012-01-11T09:56:17.421-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.corba.ORBUtil|
_ThreadID=21;_ThreadName=Thread-2;|
IOP00410016: Unable to create IIOP listener on the specified host all interfaces and port 43,700
org.omg.CORBA.COMM_FAILURE: SEVERE: IOP00410016:
Unable to create IIOP listener on the specified host all interfaces and port 43,700 vmcid: OMG minor code: 16 completed: No
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:248)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:95)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.handleFullLogging(WrapperGenerator.java:387)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.access$400(WrapperGenerator.java:107)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator$2.invoke(WrapperGenerator.java:511)
at com.sun.corba.ee.spi.orbutil.proxy.CompositeInvocationHandlerImpl.invoke(CompositeInvocationHandlerImpl.java:99)
at $Proxy160.createListenerFailed(Unknown Source)
at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.initialize(SocketOrChannelAcceptorImpl.java:98)
at com.sun.corba.ee.impl.transport.CorbaTransportManagerImpl.getAcceptors(CorbaTransportManagerImpl.java:248)
at com.sun.corba.ee.impl.transport.CorbaTransportManagerImpl.addToIORTemplate(CorbaTransportManagerImpl.java:267)
at com.sun.corba.ee.spi.oa.ObjectAdapterBase.initializeTemplate(ObjectAdapterBase.java:103)
at com.sun.corba.ee.impl.oa.poa.POAImpl.initialize(POAImpl.java:553)
at com.sun.corba.ee.impl.oa.poa.POAImpl.makeRootPOA(POAImpl.java:352)
at com.sun.corba.ee.impl.oa.poa.POAFactory$1.evaluate(POAFactory.java:286)
at com.sun.corba.ee.impl.orbutil.closure.Future.evaluate(Future.java:61)
at com.sun.corba.ee.impl.resolver.LocalResolverImpl.resolve(LocalResolverImpl.java:55)
at com.sun.corba.ee.impl.resolver.CompositeResolverImpl.resolve(CompositeResolverImpl.java:59)
at com.sun.corba.ee.impl.orb.ORBImpl.resolve_initial_references(ORBImpl.java:1266)
at com.sun.corba.ee.impl.naming.cosnaming.TransientNameService.initialize(TransientNameService.java:124)
at com.sun.corba.ee.impl.naming.cosnaming.TransientNameService.<init>(TransientNameService.java:93)
at org.glassfish.enterprise.iiop.impl.PEORBConfigurator.configure(PEORBConfigurator.java:161)
at com.sun.corba.ee.impl.orb.ORBConfiguratorImpl.runUserConfigurators(ORBConfiguratorImpl.java:185)
at com.sun.corba.ee.impl.orb.ORBConfiguratorImpl.configure(ORBConfiguratorImpl.java:170)
at com.sun.corba.ee.impl.orb.ORBImpl.postInit(ORBImpl.java:625)
at com.sun.corba.ee.impl.orb.ORBImpl.set_parameters(ORBImpl.java:704)
at com.sun.corba.ee.impl.orb.ORBImpl.setParameters(ORBImpl.java:691)
at com.sun.corba.ee.spi.osgi.ORBFactory.initialize(ORBFactory.java:107)
at org.glassfish.enterprise.iiop.impl.GlassFishORBManager.initORB(GlassFishORBManager.java:581)
at org.glassfish.enterprise.iiop.impl.GlassFishORBManager.getORB(GlassFishORBManager.java:263)
at org.glassfish.enterprise.iiop.impl.GlassFishORBFactoryImpl.createORB(GlassFishORBFactoryImpl.java:93)
at org.glassfish.enterprise.iiop.api.GlassFishORBHelper.getORB(GlassFishORBHelper.java:152)
at org.glassfish.enterprise.iiop.api.GlassFishORBHelper.getProtocolManager(GlassFishORBHelper.java:219)
at com.sun.ejb.containers.BaseContainer.initializeProtocolManager(BaseContainer.java:824)
at com.sun.ejb.containers.BaseContainer.<init>(BaseContainer.java:508)
at com.sun.ejb.containers.MessageBeanContainer.<init>(MessageBeanContainer.java:142)
at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:121)
at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:230)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:299)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:105)
at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186)
at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:460)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createServerSocket(IIOPSSLSocketFactory.java:292)
at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.initialize(SocketOrChannelAcceptorImpl.java:91)
... 61 more

#]
Comment by sherryshen [ 11/Jan/12 ]

From P2 to P3.

Comment by sherryshen [ 12/Jan/12 ]

See similar error in Solaris 11 and RequestAndApplicationScopeEJBMDB.zip
http://java.net/jira/browse/GLASSFISH-18155
org.omg.CORBA.COMM_FAILURE: SEVERE: IOP00410016:
Unable to create IIOP listener on the specified host asqe-sblade-8.us.oracle.com and port 0
vmcid: OMG minor code: 16 completed: No

Comment by Harshad Vilekar [ 12/Jan/12 ]

instance2 log is showing Address already in use error - indicating the port is not available. Is that expected by the test ? If not, consider adding a small delay between instance restart and see if you get consistent results.

-----------------
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/log/server_in2.log_2012-01-11T11-31-37

Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createServerSocket(IIOPSSLSocketFactory.java:292)
at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.initialize(SocketOrChannelAcceptorImpl.java:91)
... 61 more
-----------------

Comment by sherryshen [ 12/Jan/12 ]

I can't tell if IOP00410016 is root cause for tx intermittent failures.

I gave a pointer of the log so that other may find root cause around.

I do have time delay in tests for restart instance, e.g. around 15 seconds for most cases.

I checked server.log for this case, the IOP00410016 is shown at 55 seconds after last instance restart.
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/log/in2/server.log_2012-01-11T11-31-37
[#|2012-01-11T09:55:22.204-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=12;_ThreadName=Thread-2;|GMS1092: GMS View Change Received for group: sqe-cluster : Members in view for JOINED_AND_READY_EVENT(before change analysis) are :
1: MemberId: clustered_instance_1, MemberType: CORE, Address: 10.133.184.142:9134:228.9.200.101:10177:sqe-cluster:clustered_instance_1
2: MemberId: clustered_instance_2, MemberType: CORE, Address: 10.133.184.142:9161:228.9.200.101:10177:sqe-cluster:clustered_instance_2
3: MemberId: clustered_instance_3, MemberType: CORE, Address: 10.133.184.142:9178:228.9.200.101:10177:sqe-cluster:clustered_instance_3
4: MemberId: server, MemberType: SPECTATOR, Address: 10.133.184.142:9187:228.9.200.101:10177:sqe-cluster:server

#]

.....

[#|2012-01-11T09:56:17.421-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.corba.ORBUtil|_ThreadID=21;_ThreadName=Thread-2;|IOP00410016: Unable to create IIOP listener on the specified host all interfaces and port 43,700
org.omg.CORBA.COMM_FAILURE: SEVERE: IOP00410016: Unable to create IIOP listener on the specified host all interfaces and port 43,700 vmcid: OMG minor code: 16 completed: No
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
......
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createServerSocket(IIOPSSLSocketFactory.java:292)
at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.initialize(SocketOrChannelAcceptorImpl.java:91)
... 61 more

#]

I compared 2 runs for IOP00410016 on in2.
--3 intermittent tx failure with 16 times of IOP00410016
/net/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl/log/in2
-bash-3.2$ grep IOP00410016 server.log* | wc -l
16
-bash-3.2$

--1 intermittent tx failure with 22 times of IOP00410016
-bash-3.2$ pwd
/net/asqe-logs/export1/3.1.2/Results/build17/core/lc_derby_rhl/log/in2
-bash-3.2$ grep IOP00410016 server.lo* | wc -l
22
-bash-3.2$
The test results are saved at
http://agni-1.us.oracle.com/JSPWiki/Wiki.jsp?page=V312CoreTest
RunID 39 Oracle DB and RunID 40 Derby

Comment by sherryshen [ 13/Jan/12 ]

As discussed with Harshad, I added a sleep time (30 sec) between restart-cluster to see
if the address can be released, w.r.t.
Caused by: java.net.BindException: Address already in use
From Harshad, some platforms take longer time than others to release the address.
I repeated the previous core-local-cluster executions on the same platform, RHL5,
tx tests got 100% pass without IOP00410016 in server.log, e.g.
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_derby_rhl2/html/summaryreport.html

Comment by sherryshen [ 13/Jan/12 ]

Another run with 30 seconds between cluster restart is finished with
2 intermittent failures for tx tests and 22 IOP004 error in execution
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl2/html/summaryreport.html

hudson@apg-sqe2% find . -name "." -exec grep -l -i "IOP004" {} \;
./in2/server.log_2012-01-13T11-21-16
hudson@apg-sqe2% pwd
/net/asqe-logs/export1/3.1.2/Results/build17/core/lc_oracle4_rhl2/log
hudson@apg-sqe2% grep IOP004 ./in2/server.log_2012-01-13T11-21-16 | wc -l
22
hudson@apg-sqe2%

The failure appeared in cliapp2 with tx log on file system at this time.
deploy-common-ee:
[echo] So you are using Enterprise Version eh ?
[exec] asadmin --host localhost --port 4848 --user admin --passwordfile /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build-config/adminpassword.txt --interactive=false --echo=true --terse=false deploy --force=false --precompilejsp=false --verify=false --retrieve /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build/ee/i386_apg-sqe2_Linux/cliapp2/archive --generatermistubs=false --availabilityenabled=false --asyncreplication=true --target sqe-cluster --keepreposdir=false --keepfailedstubs=false --isredeploy=false --logreportederrors=true /export/hudson/workspace/sherry-lc-oracle/appserver-sqe/build/ee/i386_apg-sqe2_Linux/cliapp2/archive/cliapp2App.ear
[exec] Application deployed with name cliapp2App.
[exec] WARNING: Command _deploy did not complete successfully on server instance clustered_instance_2: remote failure: Failed to load the application on instance clustered_instance_2. The application will not run properly. Please fix your application and redeploy.
[exec] Exception while loading the app : EJB Container initialization error. Please see server.log for more details.
[exec] Command deploy completed with warnings.
[echo] Deployment on target server sqe-cluster successful
.....
[java] -----------------------------------------
[java] - tx-cliapp2-t4mql-killMQBrokerAutomaticSelfRecovery1: FAIL -
[java] -----------------------------------------

Comment by Harshad Vilekar [ 14/Jan/12 ]

The latest run is showing following error in the instance2 server log. Assigning to the jms team to review.
======================
[#|2012-01-13T09:50:06.885-0800|WARNING|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.jtsxa|_ThreadID=22;_ThreadName=Thread-2;|JTS5067: Unexpected error occurred in commit
javax.transaction.xa.XAException
at com.sun.messaging.jmq.jmsclient.XAResourceForMC.commit(XAResourceForMC.java:248)
at com.sun.jts.jtsxa.OTSResourceImpl.commit(OTSResourceImpl.java:114)
at com.sun.jts.CosTransactions.RegisteredResources.distributeCommit(RegisteredResources.java:763)
at com.sun.jts.CosTransactions.TopCoordinator.commit(TopCoordinator.java:2099)
at com.sun.jts.CosTransactions.CoordinatorTerm.commit(CoordinatorTerm.java:412)
at com.sun.jts.CosTransactions.TerminatorImpl.commit(TerminatorImpl.java:250)
at com.sun.jts.CosTransactions.CurrentImpl.commit(CurrentImpl.java:633)
at com.sun.jts.jta.TransactionManagerImpl.commit(TransactionManagerImpl.java:332)
at com.sun.enterprise.transaction.jts.JavaEETransactionManagerJTSDelegate.commitDistributedTransaction(JavaEETransactionManagerJTSDelegate.java:185)
at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:861)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:5136)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4901)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2045)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1994)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:222)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at $Proxy211.testxa(Unknown Source)
at tx.recovery.cliapp2.TestServlet.doGet(TestServlet.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C01020000009F012FD86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(45712)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4103)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.commit(ProtocolHandler.java:3600)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.commit(ProtocolHandler.java:3517)
at com.sun.messaging.jmq.jmsclient.XAResourceForMC.commit(XAResourceForMC.java:226)
... 42 more
Caused by: com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C01020000009F012FD86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(45712)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4088)
... 45 more
======================

Comment by amyk [ 18/Jan/12 ]

To recovery a prepared transaction in MQ, it needs to connect to the broker that owns the transaction (see also RFE 6944363).

However further looking at where the "Uknown XID" JMSException (referenced in Marina's below email) occurred and the full stacktrace (see the end of this comment section) in the server log, the "Uknown XID" JMSException did not occur at transaction recovery time, it occurred in the servlet doGet() at container commit which is after the test injected failure between PREPARE and COMMIT to the connected broker while TM injected a sleep between PREPARE and COMMIT. Hence after the sleep, at COMMIT time, the MQ client runtime auto-reconnect to another broker (note: the test uses conventional cluster) which doesn't know about the PREPARED transaction because it owned by the killed broker, hence the "Unknown XID" exception. This is expected in conventional cluster. Afterward, the test will do transaction recovery to recover the PREPARED transaction.

----------------------------------------
>[#|2012-01-13T09:48:36.826-0800|WARNING|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.utils.RecoveryHooks|_ThreadID=22;_ThreadName=Thread-2;|JTS5057: FailPoint : [null]|#]
>
> [#|2012-01-13T09:48:36.826-0800|WARNING|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.utils.RecoveryHooks|_ThreadID=22;_ThreadName=Thread-2;|JTS5057: FailPoint : [2]|#]
>
> ==> the above line means tx sleeps between prepare and commit
>
> (2) right after above:
>
> [#|2012-01-13T09:48:51.194-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=23;_ThreadName=Thread-2;|[I500]: Caught JVM Exception: java.io.EOFException: Trying to read 72 bytes. Already read 0 bytes.|#]
>
> [#|2012-01-13T09:48:51.194-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=24;_ThreadName=Thread-2;|[I500]: Caught JVM Exception: java.io.EOFException: Trying to read 72 bytes. Already read 0 bytes.|#]
>
> [#|2012-01-13T09:48:51.196-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.inbound.message|_ThreadID=26;_ThreadName=Thread-2;|MQJMSRA_EC1101: onEvent:Connection Event:E206:[E206]: Connection closed. The connection is closed due to a network problem, broker crashed, or internal error: localhost:48686(60082), com.sun.messaging.jms.notification.ConnectionClosedEvent[source=BrokerAddress=localhost:48686(60082), ConnectionID=5909906714122617856, ReconnectEnabled: false, IsConnectedToHABroker: false]|#]
>
> <...>
> (3) After mostly repeating the above logging entries, there is this recovery happening somehow:
>
> [#|2012-01-13T09:49:24.221-0800|INFO|glassfish3.1.2|javax.jms.connection|_ThreadID=24;_ThreadName=Thread-2;|[I107]: Connection recover state: RECOVER_TRANSPORT_CONNECTED, broker: localhost:58686(45712)|#]
>
> [#|2012-01-13T09:49:24.221-0800|INFO|glassfish3.1.2|javax.jms.connection|_ThreadID=24;_ThreadName=Thread-2;|[I107]: Connection recover state: RECOVER_STARTED, broker: localhost:58686(45712)|#]
>
> [#|2012-01-13T09:49:24.221-0800|INFO|glassfish3.1.2|javax.jms.connection|_ThreadID=29;_ThreadName=Thread-2;|[I107]: Connection recover state: RECOVER_IN_PROCESS, broker: localhost:58686(45712)|#]
>
> [#|2012-01-13T09:49:24.227-0800|INFO|glassfish3.1.2|javax.jms.connection|_ThreadID=29;_ThreadName=Thread-2;|[I107]: Connection recover state: RECOVER_SUCCEEDED, broker: localhost:58686(45712)|#]
>
> [#|2012-01-13T09:49:24.228-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.outbound.connection|_ThreadID=28;_ThreadName=Thread-2;|MQJMSRA_CL1101: onEvent:Connection Event for mc=1 :xacId=:event:E301:[E301]: Connection reconnected to the broker: localhost:58686(45712), com.sun.messaging.jms.notification.ConnectionReconnectedEvent[source=BrokerAddress=localhost:58686(45712), ConnectionID=8108507757229997056, ReconnectEnabled: true, IsConnectedToHABroker: false]|#]
>
> [#|2012-01-13T09:49:24.228-0800|INFO|glassfish3.1.2|javax.jms.connection|_ThreadID=29;_ThreadName=Thread-2;|[I107]: Connection recover state: RECOVER_INACTIVE, broker: localhost:58686(45712)|#]
>
> [#|2012-01-13T09:49:24.234-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=26;_ThreadName=Thread-2;|[C4003]: Error occurred on connection creation [localhost:48686]. - cause: java.net.ConnectException: Connection refused|#]
>
> [#|2012-01-13T09:49:29.257-0800|SEVERE|glassfish3.1.2|javax.resourceadapter.mqjmsra.inbound.message|_ThreadID=26;_ThreadName=Thread-2;|MQJMSRA_EC4001: onException:reconnect success
>
> <...>
> (4) And then what seems like transaction being committed:
>
> [#|2012-01-13T09:50:06.840-0800|WARNING|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.utils.RecoveryHooks|_ThreadID=22;_ThreadName=Thread-2;|JTS5057: FailPoint : [null]|#]
>
> [#|2012-01-13T09:50:06.882-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=22;_ThreadName=Thread-2;|[I500]: Caught JVM Exception: com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C01020000009F012FD86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(45712)|#]
>
> [#|2012-01-13T09:50:06.882-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=22;_ThreadName=Thread-2;|MQRA:XARFMC:commit:XAException-Exception=[COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C01020000009F012FD86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(45712)|#]
>
> [#|2012-01-13T09:50:06.884-0800|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=22;_ThreadName=Thread-2;|com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C01020000009F012FD86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(45712)
> at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4103)
> at com.sun.messaging.jmq.jmsclient.ProtocolHandler.commit(ProtocolHandler.java:3600)
> at com.sun.messaging.jmq.jmsclient.ProtocolHandler.commit(ProtocolHandler.java:3517)
> at com.sun.messaging.jmq.jmsclient.XAResourceForMC.commit(XAResourceForMC.java:226)
> at com.sun.jts.jtsxa.OTSResourceImpl.commit(OTSResourceImpl.java:114)

>>>>>>>>>The full stacktrace of the "Uknown XID" JMSException
[#|2012-01-13T10:49:15.531-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=21;_ThreadName=Thread-2;|[I500]: Caught JVM Exception: com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C0102000000AE6265D86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(42645)|#]

[#|2012-01-13T10:49:15.531-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=21;_ThreadName=Thread-2;|MQRA:XARFMC:commit:XAException-Exception=[COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C0102000000AE6265D86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(42645)|#]

[#|2012-01-13T10:49:15.533-0800|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=21;_ThreadName=Thread-2;|com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C0102000000AE6265D86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(42645)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4103)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.commit(ProtocolHandler.java:3600)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.commit(ProtocolHandler.java:3517)
at com.sun.messaging.jmq.jmsclient.XAResourceForMC.commit(XAResourceForMC.java:226)
at com.sun.jts.jtsxa.OTSResourceImpl.commit(OTSResourceImpl.java:114)
at com.sun.jts.CosTransactions.RegisteredResources.distributeCommit(RegisteredResources.java:763)
at com.sun.jts.CosTransactions.TopCoordinator.commit(TopCoordinator.java:2099)
at com.sun.jts.CosTransactions.CoordinatorTerm.commit(CoordinatorTerm.java:412)
at com.sun.jts.CosTransactions.TerminatorImpl.commit(TerminatorImpl.java:250)
at com.sun.jts.CosTransactions.CurrentImpl.commit(CurrentImpl.java:633)
at com.sun.jts.jta.TransactionManagerImpl.commit(TransactionManagerImpl.java:332)
at com.sun.enterprise.transaction.jts.JavaEETransactionManagerJTSDelegate.commitDistributedTransaction(JavaEETransactionManagerJTSDelegate.java:185)
at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:861)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:5136)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4901)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2045)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1994)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:222)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at $Proxy126.testxa(Unknown Source)
at tx.recovery.cliapp3.TestServlet.doGet(TestServlet.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.sun.messaging.jms.JMSException: [COMMIT_TRANSACTION_REPLY(47)] [C4036]: A broker error occurred. :[404] Uknown XID 6170672D737165322C636C757374657265645F696E7374616E63655F322C5034333730302C0102000000AE6265D86170672D737165322C636C757374657265645F696E7374616E63655F322C503433373030 user=guest, broker=localhost:58686(42645)
at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4088)
... 45 more

#]
Comment by sherryshen [ 20/Jan/12 ]

2 executions of core cluster tests are completed on 3.1.2 b18.
http://agni-1.us.oracle.com/JSPWiki/Wiki.jsp?page=V312CoreTest

One run has 100% pass on tx, see RunID45,
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_rhl/html/summaryreport.html

Another run has 2 failures on tx recovery, RunID 46.
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_oel/html/summaryreport.html
http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_oel/html/test_results_txn.html
tx-cliapp3-t1mql-dblog-killASInstanceAutomaticDelegatedRecoveryID
tx-cliapp2-t2mql-dblog-killASInstanceAdmincliDelegatedRecoveryID
The test scenarios for the 2 failures are TX_01L and TX_02L in
http://agni-1.us.oracle.com/asqe-logs/export1/3.1/docs/sqe/txn/GF31TXRecoveryTestSpec.html

http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_oel/output/txn_recovery_dblog.output
The client output for the first failure show the following:

1) deploy is OK on 3 instances

deploy-common-ee:
[echo] So you are using Enterprise Version eh ?
[exec] asadmin --host localhost --port 4848 --user admin --passwordfile /export/hudson/workspace/sonia-oel6-v312-clu-smon/appserver-sqe/build-config/adminpassword.txt --interactive=false --echo=true --terse=false deploy --force=false --precompilejsp=false --verify=false --retrieve /export/hudson/workspace/sonia-oel6-v312-clu-smon/appserver-sqe/build/ee/amd64_asqeopteron1_Linux/cliapp3/archive --generatermistubs=false --availabilityenabled=false --asyncreplication=true --target sqe-cluster --keepreposdir=false --keepfailedstubs=false --isredeploy=false --logreportederrors=true /export/hudson/workspace/sonia-oel6-v312-clu-smon/appserver-sqe/build/ee/amd64_asqeopteron1_Linux/cliapp3/archive/cliapp3App.ear
[exec] Application deployed with name cliapp3App.
[exec] Command deploy executed successfully.
[echo] Deployment on target server sqe-cluster successful

2) insert on in2
operate-data:
[echo] Executing test on port 48080
[echo] build.classes.dir /export/hudson/workspace/sonia-oel6-v312-clu-smon/appserver-sqe/build/ee/amd64_asqeopteron1_Linux/cliapp3/classes
[java] WS HOME appserver-sqe
[java] main: testSuiteID=tx-cliapp3-
[java] main: testCase=operate_xa
[java] invoking webclient at http://localhost:48080/cliapp3/test?xa

3) verify on on1
verify-data:
[echo] Executing test on port 38080
[java] WS HOME appserver-sqe
[java] main: testSuiteID=tx-cliapp3-
[java] main: testCase=verify_xa
[java] invoking webclient at http://localhost:38080/cliapp3/verify?xa
[java] Processing line: got exception
[java] Processing line: javax.ejb.EJBException: Transaction aborted

[java] - tx-cliapp3-t1mql-dblog-killASInstanceAutomaticDelegatedRecovery: FAIL -

The server.log on in2 show that No Classloader found for RA [ __ds_jdbc_ra ],
which one of the error that I reported initially on build 14.

http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_oel/log/in2/server.log_2012-01-19T17-00-26

[#|2012-01-19T13:51:36.731-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|
_ThreadID=17;_ThreadName=Thread-2;|RAR8061: failed to load resource-adapter-config or RA [ __ds_jdbc_ra ], com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: No Classloader found for RA [ __ds_jdbc_ra ]|#]

[#|2012-01-19T13:51:36.738-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|ThreadID=17;_ThreadName=Thread-2;|RAR8060: Unable to lookup pool [ poolNontx ], javax.naming.NamingException: Lookup failed for '_SYSTEM/pools/poolNontx' in SerialContext[myEnv=

{java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is javax.naming.NameNotFoundException: pools]|#]

[#|2012-01-19T13:51:36.741-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|ThreadID=17;_ThreadName=Thread-2;|RAR6017 : Failed to get connection pool object jdbc/nontx via JNDI lookup : com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Lookup failed for '_SYSTEM/pools/poolNontx' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming}

|#]

[#|2012-01-19T13:51:36.917-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.recovery|_ThreadID=17;_ThreadName=Thread-2;|RAR7109: Error while loading jdbc resources during recovery : jdbc/nontx|#]

[#|2012-01-19T13:51:37.071-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=17;_ThreadName=Thread-2;|JTS5014: Recoverable JTS instance, serverId = [100]|#]

.....

[#|2012-01-19T13:51:37.174-0800|SEVERE|glassfish3.1.2|
javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=18;_ThreadName=Thread-2;|
javax.naming.NamingException:
Lookup failed for 'jdbc/nontx' in SerialContext[myEnv=

{java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is javax.naming.NamingException: Unable to lookup resource : jdbc/nontx [Root exception is com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Lookup failed for '__SYSTEM/pools/poolNontx' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming}

]]
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:518)
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:455)
at javax.naming.InitialContext.lookup(InitialContext.java:392)

Comment by sherryshen [ 21/Jan/12 ]

The tests are executed in a cluster of 3 instances.
The cliapp3 is the first test in the execution.
Since multiple tests are in execution, I listed a few time stamps of in2 log
for both fail and pass cases to help reading the log.
We need to look for why "No Classloader found for RA" happened.
I saw that "Recovery of Inbound Transactions started." appears early in failed case,
but did not know if this is related to the failure.

http://sqe-hudson.us.oracle.com:8080/hudson/job/sonia-oel6-v312-clu-smon/
#3 b18, the first test of execution, cliapp3, failed

http://sqe-hudson.us.oracle.com:8080/hudson/job/sherry-lc/
#9, b18, the first test of execution, cliapp3, passed

1) cliapp3 failed in one run.

http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_oel/log/in2/server.log_2012-01-19T17-00-26

1.1) deployed cliapp3

[#|2012-01-19T13:47:54.206-0800|INFO|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|
_ThreadID=18;_ThreadName=Thread-2;|Recovery of Inbound Transactions started.|#]

.......

[#|2012-01-19T13:51:35.424-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=12;_ThreadName=Thread-2;|
GMS1025: Adding Joined And Ready member: clustered_instance_3 group: sqe-cluster StartupState: GROUP_STARTUP |#]

[#|2012-01-19T13:51:37.162-0800|SEVERE|glassfish3.1.2|
javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service|
_ThreadID=18;_ThreadName=Thread-2;|RAR8061:
failed to load resource-adapter-config or RA [ __ds_jdbc_ra ],
com.sun.appserv.connectors.internal.api.ConnectorRuntimeException:
No Classloader found for RA [ __ds_jdbc_ra ]|#]

........

[#|2012-01-19T13:51:42.090-0800|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|
_ThreadID=10;_ThreadName=Thread-2;|JMS010: ADDRESSLIST in setJmsServiceProvider:
mq://localhost:48686/,mq://localhost:58686/,mq://localhost:38686/|#]

[#|2012-01-19T13:51:42.090-0800|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|
_ThreadID=10;_ThreadName=Thread-2;|JMS08: JMS Service Connection URL is : mq://localhost:48686/,mq://localhost:58686/,mq://localhost:38686/|#]

[#|2012-01-19T13:51:42.984-0800|INFO|glassfish3.1.2|org.hibernate.validator.util.Version|_ThreadID=10;_ThreadName=Thread-2;|
Hibernate Validator 4.2.0.Final|#]

[#|2012-01-19T13:51:43.851-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=10;_ThreadName=Thread-2;|
MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version: 4.5.2 (Build 2-d) Compile: Thu Dec 8 17:30:48 PST 2011|#]

[#|2012-01-19T13:51:43.856-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=10;_ThreadName=Thread-2;|
MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is LOCAL, connection mode is TCP|#]

[#|2012-01-19T13:51:48.985-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=10;_ThreadName=Thread-2;|
[C4003]: Error occurred on connection creation [localhost:48686]. - cause: java.net.ConnectException: Connection refused|#]

........

[#|2012-01-19T13:53:20.885-0800|INFO|glassfish3.1.2|
javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|
_ThreadID=21;_ThreadName=Thread-2;|
cliapp3App was successfully deployed in 19,341 milliseconds.|#]

1.2) insert to database, and tx at prepared stage.
[#|2012-01-19T13:53:44.590-0800|WARNING|glassfish3.1.2|
javax.enterprise.system.core.transaction.com.sun.jts.utils.RecoveryHooks|
_ThreadID=22; _ThreadName=Thread-2;|JTS5057: FailPoint : [2]|#]

1.3) deployed cliapp2, the next test, don't need to look further.
[#|2012-01-19T14:07:40.600-0800|INFO|glassfish3.1.2|
javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|
_ThreadID=21;_ThreadName=Thread-2;|
cliapp2App was successfully deployed in 14,274 milliseconds.|#]

2) cliapp3 passed in another run.

http://agni-1.us.oracle.com/asqe-logs/export1/3.1.2/Results/build18/core/lc_sm_rhl/log/in2/server.log_2012-01-19T17-08-17
2.1) deployed cliapp3

[#|2012-01-19T15:04:59.796-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=12;_ThreadName=Thread-2;|
GMS1025: Adding Joined And Ready member: clustered_instance_3 group: sqe-cluster StartupState: GROUP_STARTUP |#]

......
[#|2012-01-19T15:05:02.392-0800|INFO|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|
_ThreadID=17;_ThreadName=Thread-2;|Recovery of Inbound Transactions started.|#]

[#|2012-01-19T15:05:03.052-0800|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|
_ThreadID=10;_ThreadName=Thread-2;|JMS010: ADDRESSLIST in setJmsServiceProvider:
mq://localhost:48686/,mq://localhost:58686/,mq://localhost:38686/|#]

[#|2012-01-19T15:05:03.053-0800|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|
_ThreadID=10;_ThreadName=Thread-2;|JMS08: JMS Service Connection URL is : mq://localhost:48686/,mq://localhost:58686/,mq://localhost:38686/|#]

[#|2012-01-19T15:05:03.404-0800|INFO|glassfish3.1.2|org.hibernate.validator.util.Version|_ThreadID=10;_ThreadName=Thread-2;|
Hibernate Validator 4.2.0.Final|#]

[#|2012-01-19T15:05:04.761-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=10;_ThreadName=Thread-2;|
MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version: 4.5.2 (Build 2-d) Compile: Thu Dec 8 17:30:48 PST 2011|#]

[#|2012-01-19T15:05:04.761-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=10;_ThreadName=Thread-2;|
MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is LOCAL, connection mode is TCP|#]

[#|2012-01-19T15:05:05.499-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|
_ThreadID=17;_ThreadName=Thread-2;|JTS5014: Recoverable JTS instance, serverId = [100]|#]

[#|2012-01-19T15:05:06.502-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|
_ThreadID=18;_ThreadName=Thread-2;|JTS5078: Table for transaction logging was not added.|#]
........

[#|2012-01-19T15:05:07.697-0800|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=10;_ThreadName=Thread-2;|
MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter Started:LOCAL|#]

.......

[#|2012-01-19T15:05:41.618-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|
_ThreadID=21;_ThreadName=Thread-2;|cliapp3App was successfully deployed in 7,444 milliseconds.|#]

Comment by sherryshen [ 23/Jan/12 ]

Since GLASSFISH-17971 is a following up of GLASSFISH-17789.
I summarized my observations from both bugs below:
auto-delegated recovery with dblog in
appserver-sqe/pe/transaction/recovery/cliapp3
--it always passed on b10, 10+ runs, the build had part of the support of tx db log.
--it always failed on b11, 10+ runs, the build had more support of tx db log.
--it had intermittent failure on b14, b18 with "No Classloader found for RA".

Comment by Jagadish [ 25/Jan/12 ]

I am able to reproduce the issue and have a fix. I tried the fix multiple times, locally and no reported exceptions were seen.
Thanks to Sherry for trying this couple of times. I have also started a new run.
http://sqe-hudson.us.oracle.com:8080/hudson/job/sherry-tx-lc-dbg/63/

Fix is to make sure that the global connector-class-loader object is initialized (assigned) only after the connector-class-loader chain (of all resource-adapters) is fully constructed.

Comment by Jagadish [ 25/Jan/12 ]

What is the impact on the customer of the bug?

  • There is timing issue that results in getting empty connector-class-loader during server-startup, especially during transaction recovery.

How likely is it that a customer will see the bug and how serious is the bug?

  • Some times during instance/server startup when transaction recovery is involved.

Is it a regression? Does it meet other bug fix criteria (security, performance, etc.)?

  • This seems to be the behavior since GlassFish 3.0, got exposed while enabling transaction recovery tests.

What is the cost/risk of fixing the bug?

  • Minimal, made sure that QL (Web, Classic), connector-dev, jdbc-dev, resources-admin-cli, connector-standalone-cts (Web, Classic) are passing.

How risky is the fix? How much work is the fix? Is the fix complicated?

  • Fix is not complicated, making sure that the global connector-class-loader is assigned a fully constructed class-loader instead of assigning a class-loader and then adding resource-adapter jars to the class-loader.

Is there an impact on documentation or message strings?

  • None.

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

  • This bug is raised as part of a QA test run.

Which is the targeted build of 3.1.2 for this fix?

  • B20.
Comment by Jagadish [ 27/Jan/12 ]

Fixed in Trunk as well 3.1.2

FIX INFORMATION :
TRUNK :
svn log -v -r 52294
r52294 | jr158900 | 2012-01-26 16:02:13 +0530 (Thu, 26 Jan 2012) | 5 lines
Changed paths:
M /trunk/main/appserver/connectors/connectors-internal-api/src/main/java/com/sun/appserv/connectors/internal/api/ConnectorClassLoaderServiceImpl.java
M /trunk/main/appserver/connectors/connectors-internal-api/src/main/java/com/sun/appserv/connectors/internal/api/ConnectorsClassLoaderUtil.java

3.1.2 branch :
svn log -v -r 52290
r52290 | jr158900 | 2012-01-26 11:07:13 +0530 (Thu, 26 Jan 2012) | 6 lines
Changed paths:
M /branches/3.1.2/connectors/connectors-internal-api/src/main/java/com/sun/appserv/connectors/internal/api/ConnectorClassLoaderServiceImpl.java
M /branches/3.1.2/connectors/connectors-internal-api/src/main/java/com/sun/appserv/connectors/internal/api/ConnectorsClassLoaderUtil.java





[GLASSFISH-17830] For GF 3.1.2 new support, the output of CLI are in English Created: 25/Nov/11  Updated: 08/Feb/12  Resolved: 26/Jan/12

Status: Closed
Project: glassfish
Component/s: i18n
Affects Version/s: 3.1.2_b11
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: sunny-gui Assignee: gmurr
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Server OS: OEL 6 x64 w/JDK1.6.0_29 32bit
Bundle: java_ee_sdk-6u4-b11-jdk7-linux-ml.sh
Server Locale: fr_FR.UTF-8


Attachments: HTML File report.all.html    
Tags: 3_1_2-review

 Description   

For GF 3.1.2 new support, the output of CLI are in English.

To reproduce:
After installed the bundle execute the commands like this,

  1. ./disable-secure-admin --user admin --passwordfile pwdfile --help

the output are in English.

Attached the report file for your reference.



 Comments   
Comment by gmurr [ 26/Jan/12 ]

fix will be available in 3.1.2_b20

Comment by sunny-gui [ 08/Feb/12 ]

Verified and fixed in build 21 with the bundle "ogs-3.1.2-b21-unix-ml.sh" in OEL 6 x64 w/JDK1.6.0_30 32-Bit, so close this issue.





[GLASSFISH-15804] Multiple failover not happening on OEL6 with a cluster of 10 instances Created: 02/Feb/11  Updated: 25/Feb/12  Resolved: 25/Feb/12

Status: Resolved
Project: glassfish
Component/s: rmi_iiop_failover
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: gopaljorapur Assignee: Harshad Vilekar
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL 6 + Jrockit 1.6.0_29


Attachments: File ant.output     Zip Archive instancedeleteEJBMultipleFO.zip     Zip Archive MultiEJBApp.zip    
Issue Links:
Dependency
blocks GLASSFISH-18392 Multiple Failover of IIOP does not ha... Open
Tags: 312_failover, 312_qa, 3_1-exclude, 3_1_2-exclude

 Description   

1. Create New IC
2. Look up Bean1
3. Run some method
4. Kill the instance that handled the request
5. Look up Bean2
6. Run some method
7. Kill the instance that handled the request
8. Run some method on the bean looked up in step 5

Following exception is seen

Feb 2, 2011 2:34:01 PM com.sun.dft.glassfish.ipc.api.CallbackClient killInstance
INFO: Invoking CallbackServer.killInstance on instance instance104
javax.ejb.EJBException: java.rmi.MarshalException: CORBA COMM_FAILURE 1330446344 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: IOP00410008: Connection abort vmcid: OMG minor code: 8 completed: Maybe
at com.sun.appserver.ee.tests.ejb.stateful._SFSB3Remote_Wrapper.getName(com/sun/appserver/ee/tests/ejb/stateful/_SFSB3Remote_Wrapper.java)
at com.sun.appserver.ee.tests.client.Client.runTest2(Unknown Source)
at com.sun.appserver.ee.tests.client.Client.runMultipleFOTest(Unknown Source)
at com.sun.appserver.ee.tests.client.Client.main(Unknown Source)
Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1330446344 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: IOP00410008: Connection abort vmcid: OMG minor code: 8 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:259)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:213)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
at com.sun.appserver.ee.tests.ejb.stateful._SFSB3Remote_Remote_DynamicStub.getName(com/sun/appserver/ee/tests/ejb/stateful/_SFSB3Remote_Remote_DynamicStub.java)
... 4 more
Caused by: org.omg.CORBA.COMM_FAILURE: FINE: IOP00410008: Connection abort vmcid: OMG minor code: 8 completed: Maybe
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:248)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:95)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.handleFullLogging(WrapperGenerator.java:387)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.access$400(WrapperGenerator.java:107)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator$2.invoke(WrapperGenerator.java:511)
at com.sun.corba.ee.spi.orbutil.proxy.CompositeInvocationHandlerImpl.invoke(CompositeInvocationHandlerImpl.java:99)
at $Proxy32.connectionAbort(Unknown Source)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1537)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1084)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
Caused by: org.omg.CORBA.COMM_FAILURE: FINE: IOP00410030: Exception in a blocking read on connection SocketOrChannelConnectionImpl[ java.nio.channels.SocketChannel[connected local=/10.133.184.153:56073 remote=gf-ha-dev-sb6-18/10.133.184.157:23700] ESTABLISHED true true] with a temporary selector vmcid: OMG minor code: 30 completed: No
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:248)
at com.sun.corba.ee.spi.orbutil.logex.corba.CorbaExtension.makeException(CorbaExtension.java:95)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.handleFullLogging(WrapperGenerator.java:387)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator.access$400(WrapperGenerator.java:107)
at com.sun.corba.ee.spi.orbutil.logex.WrapperGenerator$2.invoke(WrapperGenerator.java:511)
at com.sun.corba.ee.spi.orbutil.proxy.CompositeInvocationHandlerImpl.invoke(CompositeInvocationHandlerImpl.java:99)
at $Proxy32.exceptionBlockingReadWithTemporarySelector(Unknown Source)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.blockingRead(SocketOrChannelConnectionImpl.java:1604)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1501)
... 3 more



 Comments   
Comment by Ken Cavanaugh [ 02/Feb/11 ]

No time to investigate this for GF 3.1. Moving to 3.2.

Comment by Harshad Vilekar [ 29/Apr/11 ]

1. Please attach the complete test case to duplicate this issue.
2. Is this SUSE Linux specific ? What is the behavior on other platform(s) ?
3. What is the behavior with smaller cluster size ?

I'm unable to duplicate this with 5 Instance Cluster. Part of the dev test is:
-------------------------
InitialContext ic = makeIC() ;
Location slsb = lookup( ic, BeanType.SLSB ) ;

String inst1 = invokeMethod(slsb) ;
gfCluster.stopInstance(inst1);

Location sfsb = lookup( ic, BeanType.SFSB ) ;
String inst2 = invokeMethod(sfsb) ;
gfCluster.stopInstance(inst2);
String inst3 = invokeMethod(sfsb) ;
-------------------------

Comment by Harshad Vilekar [ 06/May/11 ]

I'm unable to duplicate this failure with the iiop-folb-devtests: 15804sfsb, 15804slsb. Both the tests PASS.

If you can still duplicate the issue with your test, please attach requested information to the bug report.

Comment by mzh777 [ 06/Jan/12 ]

The same exceptions were shown in execution of GF3.1.2 promote build 16 on the OEL6 + JRockit configuration. Reopen the issue.

Comment by mzh777 [ 06/Jan/12 ]

Attach the MultiEJBApp.zip containing test App source and ant.output containing steps performed in the test.

Comment by sb110099 [ 11/Jan/12 ]

This issue is seen again in 3.1.2 , the issue is reproducible on OL6 +JRockit setup.
(The test passes on Solaris11 setup).

Changing the affect version to 3.1.2_b16 and fixVersion to 3.1.2_b18.

Thanks,
Sudipa

Comment by Harshad Vilekar [ 26/Jan/12 ]

The failure could be duplicated with the updated iiop-folb-devtest, with 5 instance cluster - running on the single host. The exception is thrown when the second instance is killed (step 7). If step 7 is replaced by stop-instance, then the failure is not seen.

Comment by Harshad Vilekar [ 27/Jan/12 ]

Reviewed the test code that is failing. Adding a small sleep before and after step 7 is expected to resolve this issue.

Comment by mzh777 [ 30/Jan/12 ]

Added 10s delays before and after the second failover in the multiple failover test. The test is passing on the 3 machines-10 instances-Oracle JDK setup.

Comment by Joe Di Pol [ 30/Jan/12 ]

Due to the lateness in the release, and the fact that this is not a regression over 3.1.1 we will not be fixing this in 3.1.2. Also, the fact that this appears to only occur with two immediate failures mitigates the customer impact.

Comment by mzh777 [ 03/Feb/12 ]

The IIOP multiple failover in dynamiccluster delete instance scenario (com.sun.dft.glassfish.iiop.dynamiccluster.failover.instancedelete.EJBAdvanced.IIOPFailoverEJBMultipleFO) still failed with similar exception even with time delay. The steps of test to reproduce the errors are in instancedeleteEJBMultipleFO/ant.output. See attached instancedeleteEJBMultipleFO.zip for details. Test app is still the same (see MultiEJBApp.zip).

Comment by mzh777 [ 03/Feb/12 ]

Output of IIOP multiple failover test with dynamiccluster delete instance.

Comment by Harshad Vilekar [ 25/Feb/12 ]

Ming confirmed that the test scenario described in this bug report is passing - after adding the delay. Marking this as resolved.

For "dynamiccluster delete instance" case - new issue is created: GLASSFISH-18392.





[GLASSFISH-15467] Recently added SDK sample (clusterjsp) is not complete. Missing ant targets to compile, package, run and cleanup Created: 06/Jan/11  Updated: 01/Feb/12  Resolved: 01/Feb/12

Status: Resolved
Project: glassfish
Component/s: sample_apps
Affects Version/s: 3.1_b36
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Major
Reporter: Alex Pineda Assignee: Romain Grécourt
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

SDK build 35/36. Glassfish distribution


Tags: 3_1-exclude, 3_1_1-scrubbed, 3_1_2-approved

 Description   

Recently a new sample was added to the SDK distributions - clusterjsp, however, the contents of this sample are not consistent with the other samples. Clusterjsp is missing targets to compile, package, run and cleanup in build.xml.

Given the timing of this release, the suggestion is that we remove this sample from the GF3.1 SDK distributions.



 Comments   
Comment by scatari [ 06/Jan/11 ]

Please see http://java.net/jira/browse/GLASSFISH-14109 for more information on the original request. As commented in 14109, this release will include clusterjsp in binary format and the build scripts will be added in 3.2 due to time constraints.

It is important to give developers the access to this sample, removing the sample from distribution will defeat that.

This sample even though lacking the build system, serves the purpose. Add to that if someone tries to build this sample using standard ant tasks, then a clear message gets displayed regarding the support level in this release.

clusterjsp build does not include any compilable artifacts, hence can be disassembed and assembled by developers easily.

Adding "exclude" to be fixed in 3.2.

Comment by scatari [ 06/Jan/11 ]

Please see http://java.net/jira/browse/GLASSFISH-14109 for more information on the original request. As commented in 14109, this release will include clusterjsp in binary format and the build scripts will be added in 3.2 due to time constraints.

It is important to give developers the access to this sample, removing the sample from distribution will defeat that.

This sample even though lacking the build system, serves the purpose. Add to that if someone tries to build this sample using standard ant tasks, then a clear message gets displayed regarding the support level in this release.

clusterjsp build does not include any compilable artifacts, hence can be disassembed and assembled by developers easily.

Adding "exclude" to be fixed in 3.2.

Comment by scatari [ 05/Jul/11 ]

Not critical for product functionality. Targeting for post 3.1.1 release.

Comment by scatari [ 13/Oct/11 ]

Romain,
Please evaluate this further to be fixed in 3.1.2.

Comment by Romain Grécourt [ 09/Jan/12 ]

version 1.1 of samples has been released. This version was integrated into samples's trunk and 3.1.2 branche today. Fixed samples will available by next promoted build (#17).

Comment by Alex Pineda [ 23/Jan/12 ]

Unable to deploy sample, thus, re-opening the bug. The ant targets for compile, package and clean work without failure, however, when I execute "ant run", I see the following error:

-pre-deploy:
[exec] Result: 1
[delete] Deleting: /home/ap2257/workspace/glassfish3/glassfish/samples/javaee6/ha/clusterjsp/asadmin-get-output
[exec] Result: 1
[delete] Deleting: /home/ap2257/workspace/glassfish3/glassfish/samples/javaee6/ha/clusterjsp/asadmin-get-output

deploy:
[exec] Deprecated syntax, instead use:
[exec] asadmin --user admin --passwordfile /home/ap2257/workspace/glassfish3/glassfish/samples/bp-project/passwordfile --host localhost --port 4848 deploy [options] ...
[exec] remote failure: Unable to find a valid target with name cluster
[exec] Command deploy failed.

BUILD FAILED
/home/ap2257/workspace/glassfish3/glassfish/samples/bp-project/app-server-ant.xml:605: exec returned: 1

The issue appears that clusterjsp expects to deploy on a cluster that is not being created by ant targets.

Comment by Romain Grécourt [ 24/Jan/12 ]

The cluster and the 2 instances needed to run this sample are currently configured using "setup" targets from "clusterjsp" directory.
You need to run the following targets to be able to run this sample: "ant setup run" (as run calls package).

Be careful as if you ran the sample without having ran "setup", it means that you deployed the "clusterjsp" application on the DAS. Then when running "ant setup run", deployment to cluster will fail because the application will be already deployed on DAS.

You'll have to delete and re-create domain1 before trying again (because of some application ref). I think we need to prevent that usecase. I see some solutions:

  • add domain creation / deletion to setup / unsetup process (so users can easily clean their glassfish)
  • add a call to target "setup" from the target "all"
  • add a dependency to "setup" from target "-pre-deploy"

Or I've just missed something in the changes I made, please let me know.

Comment by Alex Pineda [ 24/Jan/12 ]

Because you've added targets to this sample, I believe you need to update the Clusterjsp instructions located for example at /export/test/glassfish3/glassfish/samples/javaee/ha/clusterjsp/docs/index.html, and list the sequence of ant targets that showcase this sample. The current instructions are all manual and the point to the pre-compiled version of the sample (clusterjsp.ear).

Comment by Romain Grécourt [ 26/Jan/12 ]

What is the impact on the customer of the bug?

User following instructions will not be able to run the sample.

What is the cost/risk of fixing the bug?
The fix is very simple: documenting the steps.

Is there an impact on documentation or message strings?
No.

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
Try to run the sample again.

Which is the targeted build of 3.1.2 for this fix?
b20.

Comment by Rebecca Parks [ 26/Jan/12 ]

This issue has been removed from the list of 3.1.2 Release Note issues. I have no objection, I'm just curious why.

Comment by Romain Grécourt [ 27/Jan/12 ]

I removed the release note tags because I thought that the fix will remove the need of a release note. Please feel free to put the tags back if you think it would still need a release note.

Comment by Alex Pineda [ 27/Jan/12 ]

A few of points

1. We should remove this issue from the Release Notes, because the initial problem reported by me is now fixed.

2. In the justification section, it says no impact to documentation, but that is not true. I pointed out that we need to update/edit the samples/javaee/ha/clusterjsp/docs/index.html and I believe this constitutes a change in the product and adds some risk if mistakes are made.

3. It's not quite true that all QA has to do is run the sample again. The change will require that QA verify the instructions of how to build/compile/deploy the sample. So, there's a bit more impact on QA than stated.





Generated at Tue Apr 28 18:32:54 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.