[GLASSFISH-18636] Unable to run Trade2 benchmark due to orb errors Created: 16/Apr/12  Updated: 16/May/13  Resolved: 16/May/13

Status: Closed
Project: glassfish
Component/s: ejb_container
Affects Version/s: 4.0_b31
Fix Version/s: 4.0_b35

Type: Bug Priority: Major
Reporter: amitagarwal Assignee: marina vatkina
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File server.log    

 Description   

From b31 onwards we have noticed following exceptions in server.log file while running Trade2 performance benchmark. Due to this error Trade2 benchmark does not proceed as expected,

[#|2012-04-09T00:36:09.229-0700|WARNING|44.0|javax.enterprise.system.container.ejb.com.sun.ejb.containers|_ThreadID=48;_ThreadName=Thread-2;|javax.ejb.TransactionRolledbackLocalException: Client's
transaction aborted
at com.sun.ejb.containers.BaseContainer.useClientTx(BaseContainer.java:4656)
at com.sun.ejb.containers.BaseContainer.preInvokeTx(BaseContainer.java:4535)
at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1907)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:205)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:123)
at $Proxy197.login(Unknown Source)
at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:239)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:150)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
at trade._Trade_DynamicStub.login(trade/_Trade_DynamicStub.java)
at trade.TradeAccessBean.login(TradeAccessBean.java:182)
at trade_client.TradeAction.doLogin(TradeAction.java:311)
at trade_client.TradeServletAction.doLogin(TradeServletAction.java:595)
at trade_client.TradeScenarioServlet.performTask(TradeScenarioServlet.java:224)
at trade_client.TradeScenarioServlet.doGet(TradeScenarioServlet.java:72)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1589)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:286)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:337)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:240)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:172)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:163)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:164)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:816)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
at java.lang.Thread.run(Thread.java:722)

#]

[#|2012-04-09T00:36:09.230-0700|INFO|44.0|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=41;_ThreadName=Thread-2;|javax.transaction.TransactionRolledbackException: CORBA TR
ANSACTION_ROLLEDBACK 9998 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x2000 minor code: 1806 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:276)
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.wrapException(Util.java:695)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:249)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:150)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
at trade._Trade_DynamicStub.login(trade/_Trade_DynamicStub.java)
at trade.TradeAccessBean.login(TradeAccessBean.java:182)
at trade_client.TradeAction.doLogin(TradeAction.java:311)
at trade_client.TradeServletAction.doLogin(TradeServletAction.java:595)
at trade_client.TradeScenarioServlet.performTask(TradeScenarioServlet.java:224)
at trade_client.TradeScenarioServlet.doGet(TradeScenarioServlet.java:72)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1589)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:286)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:337)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:240)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:172)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:163)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:164)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:816)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x2000 minor code: 1806 completed: Maybe
Caused by: javax.ejb.TransactionRolledbackLocalException: Client's transaction aborted
at com.sun.ejb.containers.BaseContainer.useClientTx(BaseContainer.java:4656)
at com.sun.ejb.containers.BaseContainer.preInvokeTx(BaseContainer.java:4535)
at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1907)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:205)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:123)
at $Proxy197.login(Unknown Source)
at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:239)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:150)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
at trade._Trade_DynamicStub.login(trade/_Trade_DynamicStub.java)
at trade.TradeAccessBean.login(TradeAccessBean.java:182)
at trade_client.TradeAction.doLogin(TradeAction.java:311)
at trade_client.TradeServletAction.doLogin(TradeServletAction.java:595)
at trade_client.TradeScenarioServlet.performTask(TradeScenarioServlet.java:224)
at trade_client.TradeScenarioServlet.doGet(TradeScenarioServlet.java:72)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1589)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:286)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:337)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:240)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:172)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:163)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:164)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:816)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
at java.lang.Thread.run(Thread.java:722)

#]


 Comments   
Comment by marina vatkina [ 16/Apr/12 ]

Can you try the latest nightly? Are there any other exceptions in the server.log before the TransactionRolledbackLocalException?

Comment by amitagarwal [ 16/Apr/12 ]

We see this issue with latest build b32 as well, so I feel this will be with latest nightly too.
I have attached a copy of server.log for your reference.

Comment by amitagarwal [ 16/Apr/12 ]

Forgot to mentioned that build b30 was fine, but build b31 onwards we started seeing this issue.

Comment by marina vatkina [ 16/Apr/12 ]

Can you attach the rotated logs as well, so that I can possibly see the starting point of these exceptions?

Can you attach the TradeBean.java?

Comment by amitagarwal [ 16/Apr/12 ]

There are lots of rotated log files. You can browse them through this link,
http://alacrity.us.oracle.com/net/alacrity.us.oracle.com/export/multicomplex/logs/p/4.0se.b31_Ois64/trade2_2se.20120408_231320/results.trade2_2se/results_4/output/1646/logs/perf-x4150-2/

We don't have source code of Trade2 benchmark. And TradeBean.java is part of it.

Comment by marina vatkina [ 16/Apr/12 ]

Is there a ejb-jar.xml?

Comment by marina vatkina [ 16/Apr/12 ]

The recent changes in the EJB Container should not affect SLSBs (which TradeBean is according to google searches (see. e.g. http://svn.apache.org/repos/asf/geronimo/daytrader/branches/m2standard/daytrader-ejb/src/main/resources/META-INF/ejb-jar.xml).
Requesting web container team to check if any transactional support had changed from b31 onward.

Comment by Shing Wai Chan [ 16/Apr/12 ]

b30 is on Mar 28. There is no traditional changes in web container.
From stack trace, it may be related to ejb container or transactions.

Comment by amitagarwal [ 17/Apr/12 ]

You can access the .ear file here to check xmls and jars.
http://alacrity.us.oracle.com/net/alacrity/export/multicomplex/workload/workload/benchmarks/trade2/jars/v3/Trade2.ear

Comment by marina vatkina [ 17/Apr/12 ]

Can you run this ear without multi-threaded access? Will you see the same error? Or is there a readme how to run it locally?

Comment by amitagarwal [ 23/Apr/12 ]

I tried this with one user and get same exceptions, but some user requests get served too.
Configuring this locally requires lots of steps and db installation too. Is it ok if we share our setup with you for few hours for debugging? Please let us know? You can coordinate with Scott too in US timezone.

Comment by Scott Oaks [ 16/May/13 ]

Fixed in b35.

Generated at Wed Jun 03 00:15:30 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.