Issue Details (XML | Word | Printable)

Key: GLASSFISH-18381
Type: Bug Bug
Status: Resolved Resolved
Resolution: Cannot Reproduce
Priority: Critical Critical
Assignee: Sanjeeb Sahoo
Reporter: blackbeltdev
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
glassfish

JPA OSGi WAB redployment hangs Glassfish and unable to shutdown

Created: 20/Feb/12 05:08 PM   Updated: 25/Mar/13 12:33 PM   Resolved: 25/Mar/13 12:33 PM
Component/s: OSGi-JavaEE
Affects Version/s: 4.0_b82_EE7MS7
Fix Version/s: 4.0_b82_EE7MS7

Time Tracking:
Not Specified

File Attachments: 1. Zip Archive AddressBook.zip (228 kB) 20/Feb/12 05:08 PM - blackbeltdev
2. Java Archive File cpms-web-static.jar (1.95 MB) 20/Feb/12 05:08 PM - blackbeltdev
3. File cpms-web.war (138 kB) 20/Feb/12 05:08 PM - blackbeltdev

Environment:

Windows 7 64-bit

java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1808-windows-x86_64, compiled mode)


Tags: osgi osgi-javaee jpa 3_1_2-exclude 3_1_2-next
Participants: blackbeltdev, Hong Zhang, Joe Di Pol, Sanjeeb Sahoo and TangYong


 Description  « Hide

Re-deploying a JPA enabled WAB bundle causes Glassfish to hang. I noticed this interesting message in the logs:

[#|2012-02-20T09:37:21.989-0600|INFO|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=17;_ThreadName=Thread-4;|Deferring refresh to framework restart, so enhanced bytes won't come into effect until then for bundle 292 if there are existing wires to this bundle.|#]

This is completely reproducible every time:
1) Start glassfish
2) Deploy WAB (works fine)
3) Edit source file (e.g. add println/logging)
4) Re-deploy
5) Hangs with logging similar to the below
6) Unable to stop glassfish outside of killing Java process

From what I can tell it looks like it is stuck on the following thread:

Stack Trace
admin-thread-pool-4848(1) [131] (WAITING)
java.lang.Object.wait line: 485
org.apache.felix.framework.Felix.acquireGlobalLock line: 4943
org.apache.felix.framework.StatefulResolver.resolve line: 219
org.apache.felix.framework.BundleWiringImpl.searchDynamicImports line: 1539
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation line: 1439
org.apache.felix.framework.BundleWiringImpl.access$400 line: 72
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass line: 1843
java.lang.ClassLoader.loadClass line: 247
com.sun.enterprise.container.common.GenericAdminAuthenticator.loginAsAdmin line: 0
com.sun.enterprise.v3.admin.AdminAdapter.authenticate line: 268
com.sun.enterprise.v3.admin.AdminAdapter.authenticate line: 310
com.sun.enterprise.v3.admin.AdminAdapter.service line: 210
com.sun.grizzly.tcp.http11.GrizzlyAdapter.service line: 179
com.sun.enterprise.v3.server.HK2Dispatcher.dispath line: 117
com.sun.enterprise.v3.services.impl.ContainerMapper$Hk2DispatcherCallable.call line: 354
com.sun.enterprise.v3.services.impl.ContainerMapper.service line: 195
com.sun.grizzly.http.ProcessorTask.invokeAdapter line: 849
com.sun.grizzly.http.ProcessorTask.doProcess line: 746
com.sun.grizzly.http.ProcessorTask.process line: 1045
com.sun.grizzly.http.DefaultProtocolFilter.execute line: 228
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter line: 137
com.sun.grizzly.DefaultProtocolChain.execute line: 104
com.sun.grizzly.DefaultProtocolChain.execute line: 90
com.sun.grizzly.http.HttpProtocolChain.execute line: 79
com.sun.grizzly.ProtocolChainContextTask.doCall line: 54
com.sun.grizzly.SelectionKeyContextTask.call line: 59
com.sun.grizzly.ContextTask.run line: 71
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork line: 532
com.sun.grizzly.util.AbstractThreadPool$Worker.run line: 513
java.lang.Thread.run line: 662

[#|2012-02-20T09:37:16.167-0600|INFO|glassfish3.1.2|org.glassfish.osgiejb|_ThreadID=17;_ThreadName=Thread-4;|removedService: Found 1 no. of EJBs|#]

[#|2012-02-20T09:37:18.462-0600|INFO|glassfish3.1.2|org.glassfish.osgijavaeebase|_ThreadID=17;_ThreadName=Thread-4;|Deleted C:\Users\KIRK~1.RAS\AppData\Local\Temp\osgiapp3320546087592904764|#]

[#|2012-02-20T09:37:18.464-0600|INFO|glassfish3.1.2|org.glassfish.osgijavaeebase|_ThreadID=17;_ThreadName=Thread-4;|Undeployed bundle com.textura.cpms.web [292]|#]

[#|2012-02-20T09:37:21.233-0600|INFO|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=17;_ThreadName=Thread-4;|Bundle having id 292 is a JPA bundle|#]

[#|2012-02-20T09:37:21.350-0600|INFO|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=17;_ThreadName=Thread-4;|Exploded bundle com.textura.cpms.web [292] at C:\Users\KIRK~1.RAS\AppData\Local\Temp\osgiapp5360629438350331033 |#]

[#|2012-02-20T09:37:21.551-0600|INFO|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=17;_ThreadName=Thread-4;|Source = C:\Users\KIRK~1.RAS\AppData\Local\Temp\osgiapp5360629438350331033\WEB-INF\classes, Target = C:\Users\KIRK~1.RAS\AppData\Local\Temp\enhanced-osgiapp8372524464557779997\WEB-INF\classes|#]

[#|2012-02-20T09:37:21.912-0600|WARNING|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=17;_ThreadName=Thread-4;|Unable to delete C:\Users\KIRK~1.RAS\AppData\Local\Temp\osgiapp5360629438350331033|#]

[#|2012-02-20T09:37:21.977-0600|INFO|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=25;_ThreadName=Thread-4;|Deleted C:\Users\KIRK~1.RAS\AppData\Local\Temp\enhanced-osgiapp8372524464557779997 |#]

[#|2012-02-20T09:37:21.989-0600|INFO|glassfish3.1.2|org.glassfish.osgijpa|_ThreadID=17;_ThreadName=Thread-4;|Deferring refresh to framework restart, so enhanced bytes won't come into effect until then for bundle 292 if there are existing wires to this bundle.|#]

[#|2012-02-20T09:37:21.989-0600|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=17;_ThreadName=Thread-4;|Updated C:\dev\java\tools\glassfish-3.1.2-b23\glassfish3\glassfish\domains\domain1\autodeploy\bundles\cpms-web.war|#]

[#|2012-02-20T09:37:21.993-0600|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=17;_ThreadName=Thread-4;|Started bundle: file:/C:/dev/java/tools/glassfish-3.1.2-b23/glassfish3/glassfish/domains/domain1/autodeploy/bundles/cpms-web.war|#]

[#|2012-02-20T09:37:40.346-0600|INFO|glassfish3.1.2|org.glassfish.osgijavaeebase|_ThreadID=19;_ThreadName=Thread-4;|Expanded at file:/C:/Users/KIRK~1.RAS/AppData/Local/Temp/osgiapp1648809821027798109/|#]

Nothing happens for a long time until...

[#|2012-02-20T10:43:40.474-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8080(2).|#]

[#|2012-02-20T10:43:41.477-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8080(4).|#]

[#|2012-02-20T10:43:43.479-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8080(1).|#]

[#|2012-02-20T10:44:32.144-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=14;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: admin-thread-pool-4848(1).|#]

[#|2012-02-20T10:44:33.144-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=14;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: admin-thread-pool-4848(1).|#]

[#|2012-02-20T10:44:34.144-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=14;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: admin-thread-pool-4848(1).|#]

[#|2012-02-20T10:44:35.144-0600|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=14;_ThreadName=Thread-4;|GRIZZLY0023: Interrupting idle Thread: admin-thread-pool-4848(1).|#]

[#|2012-02-20T10:44:35.587-0600|SEVERE|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=26;_ThreadName=Thread-4;|service exception
java.lang.RuntimeException: ClientAbortException: java.io.IOException: Connection aborted by peer
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:246)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper$Hk2DispatcherCallable.call(ContainerMapper.java:355)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:747)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1046)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:105)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:91)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:56)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: ClientAbortException: java.io.IOException: Connection aborted by peer
at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:439)
at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.flush(GrizzlyOutputBuffer.java:405)
at com.sun.grizzly.tcp.http11.GrizzlyOutputStream.flush(GrizzlyOutputStream.java:140)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:243)
... 18 more
Caused by: java.io.IOException: Connection aborted by peer
at sun.nio.ch.SocketDispatcher.write1(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:58)
at sun.nio.ch.IOUtil.write(IOUtil.java:199)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:417)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:489)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:467)
at com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:1276)
at com.sun.grizzly.tcp.Response.action(Response.java:268)
at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:434)
at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.flush(GrizzlyOutputBuffer.java:405)
at com.sun.grizzly.tcp.http11.GrizzlyOutputStream.flush(GrizzlyOutputStream.java:140)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:243)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper$Hk2DispatcherCallable.call(ContainerMapper.java:354)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
... 5 more

I have attached the source along with the bundles.

1) Host bundle for application (uses Vaadin 6.7.5)
2) Static resources as Bundle Fragment



blackbeltdev added a comment - 20/Feb/12 05:10 PM

FYI.. I don't notice this behavior with non-JPA WAB bundles. I'm able to redeploy WABs back to back with no problem.


Joe Di Pol added a comment - 20/Feb/12 10:23 PM

3.1.2 has wrapped up. Tagging for consideration in next update.


Hong Zhang added a comment - 21/Feb/12 02:12 AM

assign to sahoo for initial evaluation


blackbeltdev added a comment - 21/Feb/12 06:02 AM

Hmmm...

What's interesting is that the eclipsecon2011 demo doesn't exhibit this behavior. I'm able to continuously update the ejb_service bundle without having any issues. Maybe its related to deploying everything as part of the WAB like I did rather than split code between the API, Impl, and Web bundles in eclipsecon2011 demo.

On a related note I'd like to know how to split the eclipsecon2011 bundles further. I'm working on a long email to post to the users@glassfish.java.net email list.

Basically I'd like to know how to split the EJB Service implementation and JPA entities into their own bundles. In other words I'd like to create a persistence bundle that contains the entities (and persistence.xml) only and create another EJB service bundle that consumes the persistence bundle rather than package the service layer and entities together for increased modularity.


Sanjeeb Sahoo added a comment - 21/Feb/12 07:18 AM

Checkout https://svn.java.net/svn/glassfish~svn/trunk/fighterfish/sample/uas and see how to split EJBs, JPAs and WABs have been split into separate bundles.


Sanjeeb Sahoo added a comment - 21/Feb/12 08:39 AM

Your WAB depends on so many other packages and you never mentioned about them. I had to deploy groovy-all-1.8.6.jar, slf4j-api-1.6.1.jar, slf4j-nop-1.6.1.jar, vaadin-6.7.5.jar before I could deploy your WAB. Anyway, after doing all these, I was able to successfully deploy and redeploy the WAB. I didn't notice any hangI tried accessing app/address-book/ and saw some JPA query getting executed in server, so I am assuming the app is more-or-less working fine.

Did you try to undeploy while the app was still getting deployed? See GLASSFISH-18159 for potential deadlock in such a situation. Deployment is asynchronous where as undeployment is synchronous. So, you must wait for a message like the one shown below to appear before proceeding to use the app:
deployed bundle com.textura.cpms.web [289] at file:/tmp/osgiapp8158913360060872663/


blackbeltdev added a comment - 21/Feb/12 03:57 PM - edited

Sorry about not mentioning the dependencies. I tried to be as precise as possible and forgot the basics!

The other thing I should have mentioned that the method I used to deploy was to copy the WAR file to the autodeploy/bundle directory. The "redeploy" method was to simply overwrite the WAR file with a newer version.

Like I said above replacing the EJB JAR file from the EcliipseCon sample code did not have this problem. It only seems to be a problem with WABs that contain EJB/JPA probably due to the amount of time involved, i.e. my other simple WABs undeploy so fast it didn't run into the deadlock condition.

I will try to undeploy and deploy as distinct steps and see what happens.


blackbeltdev added a comment - 21/Feb/12 04:27 PM

I think you hit the nail on the head. If I delete the WAR, wait, and then copy the file to autodeploy then there is no problem. However if I replace the existing file it will deadlock. You can see these threads are simultaneously deploying and undeploying (notice two threads are waiting on Felix.acquireBundleLock):

Stack Trace
fileinstall-C:\dev\java\tools\glassfish-3.1.2-b23\glassfish3\glassfish\domains\domain1/autodeploy/bundles/ [65] (WAITING)
java.lang.Object.wait line: 485
org.apache.felix.framework.Felix.acquireBundleLock line: 4870
org.apache.felix.framework.Felix.startBundle line: 1744
org.apache.felix.framework.BundleImpl.start line: 944
org.apache.felix.fileinstall.internal.DirectoryWatcher.process line: 1175
org.apache.felix.fileinstall.internal.DirectoryWatcher.process line: 1153
org.apache.felix.fileinstall.internal.DirectoryWatcher.processAllBundles line: 1146
org.apache.felix.fileinstall.internal.DirectoryWatcher.process line: 456
org.apache.felix.fileinstall.internal.DirectoryWatcher.run line: 263

Stack Trace
FelixFrameworkWiring [131] (WAITING)
sun.misc.Unsafe.park line: not available [native method]
java.util.concurrent.locks.LockSupport.park line: 156
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt line: 811
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly line: 969
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly line: 1281
java.util.concurrent.FutureTask$Sync.innerGet line: 218
java.util.concurrent.FutureTask.get line: 83
org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer.removedBundle line: 183
org.osgi.util.tracker.BundleTracker$Tracked.customizerRemoved line: 508
org.osgi.util.tracker.BundleTracker$Tracked.customizerRemoved line: 424
org.osgi.util.tracker.AbstractTracked.untrack line: 352
org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged line: 464
org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback line: 868
org.apache.felix.framework.util.EventDispatcher.fireEventImmediately line: 789
org.apache.felix.framework.util.EventDispatcher.fireBundleEvent line: 514
org.apache.felix.framework.Felix.fireBundleEvent line: 4244
org.apache.felix.framework.Felix.stopBundle line: 2351
org.apache.felix.framework.Felix$RefreshHelper.stop line: 4629
org.apache.felix.framework.Felix.refreshPackages line: 3951
org.apache.felix.framework.FrameworkWiringImpl.run line: 172
java.lang.Thread.run line: 662

Stack Trace
pool-8-thread-1 [71] (WAITING)
java.lang.Object.wait line: 485
org.apache.felix.framework.Felix.acquireGlobalLock line: 4943
org.apache.felix.framework.StatefulResolver.resolve line: 219
org.apache.felix.framework.BundleWiringImpl.searchDynamicImports line: 1539
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation line: 1439
org.apache.felix.framework.BundleWiringImpl.access$400 line: 72
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass line: 1843
java.lang.ClassLoader.loadClass line: 247
java.lang.ClassLoader.defineClass1 line: not available [native method]
java.lang.ClassLoader.defineClassCond line: 631
java.lang.ClassLoader.defineClass line: 615
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass line: 2128
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation line: 1432
org.apache.felix.framework.BundleWiringImpl.access$400 line: 72
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass line: 1843
java.lang.ClassLoader.loadClass line: 247
java.lang.Class.getDeclaredMethods0 line: not available [native method]
java.lang.Class.privateGetDeclaredMethods line: 2427
java.lang.Class.getDeclaredMethods line: 1791
org.glassfish.apf.impl.ComponentDefinition.initializeMethods line: 132
org.glassfish.apf.impl.ComponentDefinition.<init> line: 76
org.glassfish.apf.impl.JavaEEScanner.getComponentInfo line: 70
org.glassfish.apf.impl.AnnotationProcessorImpl.process line: 177
org.glassfish.apf.impl.AnnotationProcessorImpl.process line: 134
com.sun.enterprise.deployment.archivist.Archivist.processAnnotations line: 598
com.sun.enterprise.deployment.archivist.Archivist.readAnnotations line: 442
com.sun.enterprise.deployment.archivist.Archivist.readAnnotations line: 429
com.sun.enterprise.deployment.archivist.Archivist.readRestDeploymentDescriptors line: 405
com.sun.enterprise.deployment.archivist.Archivist.readDeploymentDescriptors line: 380
com.sun.enterprise.deployment.archivist.Archivist.open line: 243
com.sun.enterprise.deployment.archivist.Archivist.open line: 252
com.sun.enterprise.deployment.archivist.Archivist.open line: 213
com.sun.enterprise.deployment.archivist.ApplicationFactory.openArchive line: 165
org.glassfish.javaee.core.deployment.DolProvider.load line: 185
org.glassfish.javaee.core.deployment.DolProvider.load line: 94
com.sun.enterprise.v3.server.ApplicationLifecycle.loadDeployer line: 827
com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos line: 769
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy line: 368
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy line: 240
org.glassfish.osgijavaeebase.OSGiDeploymentRequest.deploy line: 183
org.glassfish.osgijavaeebase.OSGiDeploymentRequest.execute line: 118
org.glassfish.osgijavaeebase.AbstractOSGiDeployer.deploy line: 121
org.glassfish.osgijavaeebase.OSGiContainer.deploy line: 154
org.glassfish.osgijavaeebase.JavaEEExtender.deploy line: 107
org.glassfish.osgijavaeebase.JavaEEExtender.access$200 line: 61
org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer$1.call line: 151
org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer$1.call line: 148
java.util.concurrent.FutureTask$Sync.innerRun line: 303
java.util.concurrent.FutureTask.run line: 138
java.util.concurrent.ThreadPoolExecutor$Worker.runTask line: 886
java.util.concurrent.ThreadPoolExecutor$Worker.run line: 908
java.lang.Thread.run line: 662


Sanjeeb Sahoo added a comment - 21/Feb/12 04:52 PM

I am unable to reproduce even when I just deploy/update the war using autodpeloy/bundles/.


blackbeltdev added a comment - 21/Feb/12 05:31 PM

With multi-threaded code and machine differences there's always the possibility of different runtime characteristics. I remember when I got my first dual core machine an application that worked fine on my single core machine started deadlocking on a regular basis

Now I'm running with Intel Quad Core Q9300 @ 2.53 GHz


Sanjeeb Sahoo added a comment - 21/Feb/12 05:47 PM

Those are always possibilities for MT issues. You should wait for GLASSFISH-18159 to be integrated to see if that makes any difference.


blackbeltdev added a comment - 21/Feb/12 06:22 PM

Will do. I tried it about 10 times so far and if I 'delete WAR, wait, copy file' it works every time. If I just replace the file it deadlocks every time.

BTW is 'users@glassfish.java.net' still the best place to post questions or should I use the Glassfish forums. I just sent an email about splitting up the EJB/JPA from the EclipseCon demo into separate packages so the entities and service are more modular.

Thanks for your help!


Sanjeeb Sahoo added a comment - 21/Feb/12 06:35 PM

Did you checkout the sample I had mentioned in an earlier comment in this bug? Did it not answer your question abut how to split into smaller bundles?


blackbeltdev added a comment - 21/Feb/12 06:52 PM

Arg sorry I missed that one. I'll check it out!

Thanks again!


blackbeltdev added a comment - 21/Feb/12 11:57 PM

I was able to get the fighterfish UAS sample working! I had to make a couple of maven changes and after some trial and error figured out I need to update

$GLASSFISH_HOME/glassfish/config/osgi.properties:
org.glassfish.osgijpa.extension.useHybridPersistenceProviderResolver=true

As an OSGi noob even after (mostly) reading "OSGi in Depth" I foundthe JPA setup extremely confusing and non-intuitive (not Glassfish specific).

Maybe add a link to the "fighterfish" samples here as the JPA section is a little too brief on how to go about splitting up the entities and EJBs:
http://glassfish.java.net/public/GF-OSGi-Features.pdf

I would also suggestion that you update your excellent OSGi EclipseCon tutorial to use the "fighterfish" sample instead (which is much more realistic) I think it would help the new users tremendously to get started on Enterprise OSGi which is still IMO not very well documented with lots of conflicting advice (outside of Glassfish I mean).

What we really need is the PetClinic Sample ported to hybrid JavaEE/OSGi with CDI/JPA/OSGi.

Not having Hibernate JPA support kind of hurts us from adopting Glassfish. Is there any timeline for implementing Hibernate in the near term? I realize I can embed Hibernate libraries and entities in the bundle and make it work (kind of like the 'ejbservice' sample does) but that defeats some of the modularization benefits of OSGi.

Thanks again!


Sanjeeb Sahoo added a comment - 22/Feb/12 04:06 PM

What changes did you have to do to the pom.xml to get it to work? I didn't anticipate them. Pl mention that so that I can fix them. May be by creating a bug under osgi-javaee category here.

I will fix the document to reference the sample from each technology section.

I don't know if I have the time to redo the eclipsecon tutorial - may be when I get to present them next time, I will do them. It takea a looo...t of time to create a high quality screencast.

I totally agree about having a petstore type hybrid app leveraging osgi/cdi/ejb/jpa. It comes down to resource availability.

Hibernate dev has not been very helpful in making their software OSGi-ready. I don't have the time to help them, so I don't complain. Your best bet is to embed Hibernate in jpa bundles.

Thanks much,
Sahoo


blackbeltdev added a comment - 23/Feb/12 03:48 PM - edited

Thank you Sahoo. You have been very helpful. The pom changes were mostly unique to our location and a few minor changes to eliminate some duplication. I think the biggest changes I made was to add

<relativePath>../../parent-pom/pom.xml</relativePath>

to each project so I didn't need to install artifacts in local maven repo.

And parent-pom had 1.0.1-SNAPSHOT rather than 1.0.0-SNAPSHOT

<version>1.0.1-SNAPSHOT</version>

Now that I have a working project my understanding has greatly increased. Maybe I can take a stab at a newbie guide since its fresh in my head. I totally missed the appendix had the samples listed already.

I went though the exercise to embed Hibernate in a bundle and got that working. I now have a greater appreciation of the difficulty. That was painful! It was really difficult to get maven/bnd to create the proper manifest file (I had to hack it a bit to make it work).

One last thing that wasn't clear to me is about JPA bootstrapping. In the ejbservice1 example you used managed JPA deployment which allowed you to simplify coding:

@PersistenceContext
    private EntityManager em;

whereas in ejbservice2 you used unmanaged JPA and injected EntityManagerFactory

@Inject
    @OSGiService(dynamic = true, serviceCriteria = "(persistence-unit=sample.uas.entities)")
    private EntityManagerFactory emf;

so the code becomes more brittle (i.e. have to manually create/close EntityManager):

EntityManager em = emf.createEntityManager();
    try {
    } finally {
        em.close();
    }

I was wondering if this is the only option or is it possible to use managed JPA with CDI to inject EntityManager in the EJB similar to ejbservice1 example? Something like (doesn't work):

@Inject
    @OSGiService(dynamic = true, serviceCriteria = "(persistence-unit=sample.uas.entities)")
    private EntityManager em;

In the Enterprise OSGi in Action book they use managed JPA with declarative transactions exclusively in their examples (Apache Aries) using Blueprint, e.g.

package fancyfoods.persistence;

public class InventoryImpl implements Inventory {

    private EntityManager em;

    public void setEntityManager(EntityManager em) {
        this.em = em;
    }
}
<bean
                id="inventory"
                class="fancyfoods.persistence.InventoryImpl">
                <tx:transaction
                        method="*"
                        value="Required" />
                <jpa:context
                        property="entityManager"
                        unitname="fancyfoods" />
        </bean>

So I'm hopeful this is possible to do with CDI as well.

Thanks again

References:

http://aries.apache.org/modules/jpaproject.html
http://www.manning.com/cummins/SourceCodeEnterpriseOSGiinAction.zip


Sanjeeb Sahoo added a comment - 24/Feb/12 04:58 PM

I would have liked you to raise general questions unrelated to this issue in our mailing list so that they could benefit others. Anyway, I will answer them here since you have asked so meticulously. If you don't want to manage the lifecycle of EntityManager while injecting EMF as a service, I suggest you take a look the following code which uses interceptors to localise the lifecycle management:

https://svn.java.net/svn/glassfish~svn/trunk/fighterfish/test/testapp/test.app16.mdb/src/main/java/org/glassfish/fighterfish/test/app16/mdb/TestApp16MDB_BMT.java

Pl ask related questions in forum. I am traveling now, so expect some delays in replies.

Thanks,
Sahoo


TangYong added a comment - 13/Oct/12 04:23 AM

I think that because GLASSFISH-18159 should have been integreated into current trunk, it is time to re-back the issue and next week, firstly, I will spend some time to see whether the problem will be reproduced.


TangYong added a comment - 25/Mar/13 08:09 AM

For the issue, I have confirmed it in depth today using v4 SNAPSHOT as following:

1. About re-producing steps in description

Firstly, re-producing steps in description are not very enough. And by investigations many times, right steps are as following:

[Precondition]: the means of "re-deploy" should be directly putting bundles related the issue into "glassfish4\glassfish\domains\domain1\autodeploy\bundles" . (Here, I assumed that the user used domain1 only not clustering scene or used multi-domains scene)

[Steps]
1) asadmin start-domain domain1
2) asadmin start-database
3) putting the following bundles into autodeploy\bundles

  • slf4j-jcl-1.6.4.jar
  • slf4j-api-1.6.4.jar
  • com.springsource.org.apache.commons.logging-1.1.1.jar
  • groovy-all-1.8.6.jar
  • vaadin-6.7.5.jar
  • cpms-web-static.jar[1]
  • cpms-web.war
    [1]: you can also firstly put cpms-web.war, then put cpms-web-static.jar, and executing "asadmin osgi update <WAB's bundle id>". I adopted this way.

Noting: please do not put jcl-over-slf4j-1.6.4.jar into autodeploy\bundles, because this will produce a conflict with slf4j-jcl-1.6.4.jar and cause a StackOverflowError while you accessed the vaadin application client.

BTW: In reality, these dependent bundles needed not to be deployed as OSGi bundle alone, and right doing is to make them as inner class path or embed them into the war's inner class path.

In addition, the demo itself is very interesting and uses fragment(cpms-web-static.jar) skill to attach vaadin's themes including widgetsets into host bundle(cpms-web.war) in order to reach dynamic themes switching(I guess) in the future.

4) accessing "http://localhost:8080/app/address-book/" and should be fine

5) modifying cpms-web.war's source(eg. I changed the war's servlet mapping name from address-book into address-book1), then, executing "mvn clean install"

6) replacing the cpms-web.war in autodeploy\bundles with new built cpms-web.war based 5)

2 About the "Hangs with logging similar to the below"
After executing 6), I have not seen any "Hangs with logging similar to the below", however, in server.log, there are the following exceptions:

[2013-03-25T16:35:23.156+0900] [glassfish 4.0] [WARNING] [] [org.glassfish.osgijavaeebase] [tid: _ThreadID=96 _ThreadName=pool-19-thread-1] [timeMillis: 1364196923156] [levelValue: 900] [[
Failed to deploy bundle com.textura.cpms.web [301]
org.glassfish.osgijavaeebase.DeploymentException: Deployment of com.textura.cpms.web [301] failed because of following reason: Failed while deploying bundle com.textura.cpms.web [301] : java.nio.channels.ClosedByInterruptException
at org.glassfish.osgijavaeebase.AbstractOSGiDeployer.deploy(AbstractOSGiDeployer.java:127)
at org.glassfish.osgijavaeebase.OSGiContainer.deploy(OSGiContainer.java:154)
at org.glassfish.osgijavaeebase.JavaEEExtender.deploy(JavaEEExtender.java:109)
at org.glassfish.osgijavaeebase.JavaEEExtender.access$200(JavaEEExtender.java:61)
at org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer$1.call(JavaEEExtender.java:153)
at org.glassfish.osgijavaeebase.JavaEEExtender$HybridBundleTrackerCustomizer$1.call(JavaEEExtender.java:150)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:387)
at com.sun.enterprise.util.io.FileUtils.copy(FileUtils.java:952)
at org.glassfish.internal.deployment.GenericHandler.expand(GenericHandler.java:100)
at org.glassfish.osgijavaeebase.OSGiDeploymentRequest.expandIfNeeded(OSGiDeploymentRequest.java:259)
at org.glassfish.osgijavaeebase.OSGiDeploymentRequest.prepare(OSGiDeploymentRequest.java:154)
at org.glassfish.osgijavaeebase.OSGiDeploymentRequest.execute(OSGiDeploymentRequest.java:117)
at org.glassfish.osgijavaeebase.AbstractOSGiDeployer.deploy(AbstractOSGiDeployer.java:123)
... 10 more
]]

At the same time, after I executed "asadmin stop-domain", I can stop glassfish normally and need not to kill the java process from outside.

So, My suggestion is that
1. please blackbeltdev sees the above my confirmation and tried current v4 as possible
2. if not happening on v4, please sahoo closed the issue.
3. about the exception I mentioned, whether needing to re-open an issue?

Thanks
--Tang


Sanjeeb Sahoo added a comment - 25/Mar/13 09:22 AM - edited

Tang,

The fix made in GLASSFISH-18159 means we won't deadlock, but we introduced a timeout functionality. The interrupted exception suggests some other thread interrupted the deployment process because of timeout. See the log file closely to see if there is any message to this effect. The default value for timeout is 10 seconds. See GLASSFISH-20023 to see how you can change it.

Sahoo


TangYong added a comment - 25/Mar/13 10:22 AM

Sahoo,

I am sorry for forgetting to say a point:

The exception can not be re-produced after I applied GLASSFISH-19688, and I felt a litter curious.

So, whether can close the issue?

In addition,

> See GLASSFISH-20023 to see how you can change it.
Your means is whether an user can set the value of "org.glassfish.osgijavaeebase.deployment.timeout" except in osgi.proerties file?

If such being the case, we should create a new feature to resolve it.

Thanks
--Tang


Sanjeeb Sahoo added a comment - 25/Mar/13 11:43 AM

If you can't reproduce, then close the issue.


TangYong added a comment - 25/Mar/13 12:33 PM

The issue can not be re-produced after fixing GLASSFISH-19688 which will be fixed in 4.0_b82_EE7MS7.

If seeing the issue again in the future, I will re-open the issue.