[GLASSFISH-21196] Remote Deploy of Maven EAR issues java.util.concurrent.TimeoutException Created: 15/Sep/14  Updated: 30/Sep/14

Status: Open
Project: glassfish
Component/s: admin, concurrency, deployment
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: gesker Assignee: Chris Kasso
Resolution: Unresolved Votes: 11
Labels: concurrency, deploy, jdk8, maven, netbeans, timeout
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Workstation: Win 8.1 Pro, jdk8_20, GlassFish_4.1, NetBeans_8.0.1
Server: Ubuntu Server 14.04, jdk8_20, GlassFish_4.1



 Description   

Maven Enterprise Project (of about 27 MB in size) cannot be deployed from NetBeans using DAS port.

This same EAR can successfully be deployed to GlassFish_4.1 on server using the Web – Admin console.

When attempting to deploy to the remote server from NetBeans on workstation the logs on the server record:

Exception while running a command
java.io.IOException: java.util.concurrent.TimeoutException
at org.glassfish.admin.payload.PayloadFilesManager.extractFile(PayloadFilesManager.java:584)
at org.glassfish.admin.payload.PayloadFilesManager.access$600(PayloadFilesManager.java:93)
at org.glassfish.admin.payload.PayloadFilesManager$DataRequestType$1.processPart(PayloadFilesManager.java:749)
at org.glassfish.admin.payload.PayloadFilesManager.processPartsExtended(PayloadFilesManager.java:618)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$UploadedFilesManager.extractFiles(CommandRunnerImpl.java:2074)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$UploadedFilesManager.<init>(CommandRunnerImpl.java:2046)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$UploadedFilesManager.<init>(CommandRunnerImpl.java:2025)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1155)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:109)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1846)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1722)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:534)
at com.sun.enterprise.v3.admin.AdminAdapter.onMissingResource(AdminAdapter.java:224)
at org.glassfish.grizzly.http.server.StaticHttpHandlerBase.service(StaticHttpHandlerBase.java:189)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:231)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:231)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.util.concurrent.TimeoutException
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:90)
at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:351)
at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:695)
at org.glassfish.grizzly.portunif.BackChannelFilter.handleRead(BackChannelFilter.java:80)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:351)
at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:695)
at org.glassfish.grizzly.portunif.BackChannelFilter.handleRead(BackChannelFilter.java:80)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:351)
at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:695)
at org.glassfish.grizzly.http.io.InputBuffer.blockingRead(InputBuffer.java:1119)
at org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead(ServerInputBuffer.java:95)
at org.glassfish.grizzly.http.io.InputBuffer.fill(InputBuffer.java:1143)
at org.glassfish.grizzly.http.io.InputBuffer.read(InputBuffer.java:353)
at org.glassfish.grizzly.http.server.NIOInputStreamImpl.read(NIOInputStreamImpl.java:83)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.glassfish.admin.payload.ZipPayloadImpl$Inbound$ZipEntryInputStream.read(ZipPayloadImpl.java:220)
at org.glassfish.admin.payload.PayloadFilesManager.extractFile(PayloadFilesManager.java:549)
... 46 more
Caused by: java.util.concurrent.TimeoutException
at org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(TemporarySelectorReader.java:126)
at org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(TemporarySelectorReader.java:75)
at org.glassfish.grizzly.AbstractReader.read(AbstractReader.java:72)
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:77)
... 80 more
]]

Some notes:

The ear is approx 27 MB.

Have run asadmin enable-secure-admin on the server. I am able to restart GlassFish_4.1 on the server and see its logs from Netbeans.

Had tried setting environment variable AS_ADMIN_READTIMEOUT="3600000" on the server within the init script but this did not help.

I wanted to rule out a configuration issue with my application as the issue so I did the following:

I creating some small projects in NetBeans_8.0.1:

Created a sample web application (no extras) from NetBeans template and it ran/deployed fine.
Created a sample web application (maven - no extras) from NetBeans template and it ran/deployed fine.
Created a sample EAR application (no extras) from NetBeans template and it ran/deployed fine.
Created a sample EAR application (maven - no extras) from NetBeans template and it ran/deployed fine.

So, added some dependencies to the sample maven EAR just to bulk up its size – again nothing fancy just eclipselink and primefaces stuff. The sample EAR now stood at about 26MB.

Tried to run (remote deploy) the sample again got the same java.io.IOException: java.util.concurrent.TimeoutException indicated above as I did for my actual application.



 Comments   
Comment by bhicks01 [ 17/Sep/14 ]

Another bug for the same issue is 21180: https://java.net/jira/browse/GLASSFISH-21180

Comment by dennis@gesker.com [ 30/Sep/14 ]

Maybe what is going on here has something to to with the interaction between Netbeans_8.0.1/Maven_3.2.3/GlassFish_4.1

I had a little bit of time yesterday so I tried to revisit the remote deploy issue.

I pointed my actual project via NetBeans at my remote Glassfish and again got the java.io.IOException: java.util.concurrent.TimeoutException – OK no change there.

However, I went to the Admin console and deployed – no errors where thrown but after clicking on my application none of the modules were visible. I tired deploying several times this same way and got the same results. EAR deployed but with no modules (as if the WAR and EJB/JAR were not present)

Just for the heck of it I dropped by EAR in the autodeploy directory and the application deployed completely correct. All modules were found and the context root was correct.

So, I took a step back and created a new/empty maven EAR project. Tried a remote deploy via Netbeans and threw no errors. However, in the address bar of the browser rather than the root context it displayed the full name of the WAR file "MyProject-web-1.0-SNAPSHOT" Going into the admin console and clicking on the application to launch that way the options did show the correct context root.

I tried to remote deploy the empty EAR a couple more times via Netbeans. Sometimes it was successful. Sometimes it failed with "java.io.IOException: java.util.concurrent.TimeoutException" and sometimes it failed with "java.io.EOFException"

I tired adding the following to the "maven-ear-plugin" section of my pom.xml of my EAR:

<modules>
<webModule>
<groupId>com.mycompany</groupId>
<artifactId>MyProject-web</artifactId>
<contextRoot>/MyProject-web</contextRoot>
</webModule>
<ejbModule>
<groupId>com.mycompany</groupId>
<artifactId>MyProject-ejb</artifactId>
</ejbModule>
</modules>

But the behavior remained the same. I'm new to Maven. So, the above is just my poking about but maybe it will give someone with a better understanding of the interrelationship of these three tools a hint that is useful.

I really would like to stick with maven as I like how it handles dependencies and keeps my repository clean but if it just gets in the way of seamless remote deploys I can drop back to standard project templates, too.

Any advice or more information I can collect?

Comment by bhicks01 [ 30/Sep/14 ]

I'm not using netbeans or maven for deployment. I am using a custom ant build system to run the asadmin remote deployment, and it is failing. In fact, we had our deploy command defaulted to "--upload=true", which would fail every time for some projects (this was done even for localhost deployment).

We've worked around this by scp'ing the jar/ear/war to the destination host, then running the asadmin command via an "ssh -q <asadmin deploy cmd>".





[GLASSFISH-20960] Batch job won't start when glassfish application versioning is used. Created: 19/Jan/14  Updated: 21/Sep/15

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.0
Fix Version/s: 4.1.1

Type: Bug Priority: Major
Reporter: jiggster Assignee: anthony.lai
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04, Mac OS X 10.9, Windows 7


Tags: 4_0_1-approved, 4_0_1-evangelists, application-versioning, batch

 Description   

To reproduce the issue take the webserverlog batch application sample from the java ee 7 tutorial, add the glassfish-web.xml file and make sure it contains the version-identifier element with the value, say, 1.0. Build and deploy the applicaiton, go to http://localhost:8080/webserverlog/ (assuming your instance listens on 8080) and start the batch job. The status of the job will change to STARTING, but the job itself won't start executing.

I can upload the source code of the modified version of the webserverlog sample application, but I don't know how to add the attachments.



 Comments   
Comment by jiggster [ 21/Jan/14 ]

Is there any chance that this issue will be evaluated any time soon?

Regards,
Jigg

Comment by Mahesh Kannan [ 16/Apr/14 ]

Mark for 4.0.1

Comment by Mahesh Kannan [ 16/Jul/14 ]

Assigning to Anthony

Comment by Mahesh Kannan [ 16/Jul/14 ]

The root cause is that setupContext in ManagedFuture (incorrectly) finds that the application is NOT enabled. The issue is
in ContextSetupProviderImpl.isApplicationEnabled(). This method probably needs to look into the application version field
before deciding if the app is disabled or not.

For example, after deploying an app with a different version, as you could see from the below output,
only app1:v3 is enabled. So, maybe ContextSetupProviderImpl.isApplicationEnabled() (or some
other utility method) should look into the versioning field of the app to decide correctly, if it is
enabled or not.

asadmin list-applications --long
NAME TYPE STATUS
app1 <ejb, web> disabled
app1:v1 <ejb, web> disabled
app1:v2 <ejb, web> disabled
app1v3 <ejb, web> enabled
Command list-applications executed successfully.

Here is the call stack when a batch job is submitted from a versioned app.

ManagedFutureTask::setupContext
ContextSetupProviderImpl::setup(ContextHandle contextHandleForSetup)
appName = handle.getInvocation().getAppName();
isApplicationEnabled(appName)

Unfortunately, isapplicationEnabled(appName) returns false thus causing
contextSetupException to IllegalStateException.

This causes ManagedfutureTask.run() method to be aborted. Hence the Batch job
is never executed.





[GLASSFISH-20624] man pages missing from concurrent-impl.jar Created: 11/Jun/13  Updated: 20/Dec/16

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.0_dev
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Mike Fitch Assignee: Hong Zhang
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

The man pages for the concurrency CLI commands do not display when you use the --help option. The reason is that they're not in the concurrent-impl.jar file. When concurrent/admin was refactored into concurrent/concurrent-impl, the dependency on the man pages was dropped from the pom.xml file, and so they stopped getting pulled in.



 Comments   
Comment by Hong Zhang [ 09/Jul/13 ]

Mike has a local fix for this, will review the fix and get the fix in.





[GLASSFISH-21561] The warning message "AS-CONCURRENT-00001" is written to server.log many times. Created: 15/Sep/16  Updated: 23/Jan/17

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: uno_kohei Assignee: rutujay
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

User application submits a task to ManagedExecutorService or ManagedScheduledExecutorService.
Once the task processing time exceeds the "hung-after-seconds" value of ManagedExecutorService or ManagedScheduledExecutorService, a warning message "AS-CONCURRENT-00001" is written to server.log file repeatedly every minute until the task finishes.

When a lot of tasks do hang, a large amount of same "AS-CONCURRENT-00001" messages are written to server.log file.
In that case, there is a risk that other important messages are missed or lost by rotating the server.log file.

Therefore, I hope to change behavior so that the "AS-CONCURRENT-00001" message is written only once for each task when the task processing time exceeds the "hung-after-seconds".

I also hope to maintain compatibility by adding an additional property to ManagedExecutorService and ManagedScheduledExecutorService, so user can specify the behavior when the task processing time exceeds the "hung-after-seconds".

The "hung-after-seconds" value of ManagedExecutorService can be specified by the command below:

asadmin set resources.managed-executor-service.<jndi-name>.hung-after-seconds


 Comments   
Comment by uno_kohei [ 14/Oct/16 ]

I propose a patch for fixing this bug.

This pach changes Managed(Scheduled)ExecutorService's behavior so that the "AS-CONCURRENT-00001" message is written only once for each task when the task processing time exceeds the "hung-after-seconds".

This patch requires another patch for "cu-javaee" project.
I'll create new issue for "cu-javaee" project, propose patch, and then link with each other.

Index: main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/ConcurrentRuntime.java
===================================================================
--- main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/ConcurrentRuntime.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/ConcurrentRuntime.java	(Working Copy)
@@ -49,6 +49,7 @@
 import org.glassfish.concurrent.runtime.deployer.ManagedScheduledExecutorServiceConfig;
 import org.glassfish.concurrent.runtime.deployer.ManagedThreadFactoryConfig;
 import org.glassfish.enterprise.concurrent.*;
+import org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl.ManagedThread;
 import org.glassfish.hk2.api.PostConstruct;
 import org.glassfish.hk2.api.PreDestroy;
 import org.glassfish.internal.data.ApplicationRegistry;
@@ -189,7 +190,8 @@
                 config.getTaskQueueCapacity(),
                 createContextService(config.getJndiName() + "-contextservice",
                         config.getContextInfo(), config.getContextInfoEnabled(), true),
-                AbstractManagedExecutorService.RejectPolicy.ABORT);
+                AbstractManagedExecutorService.RejectPolicy.ABORT,
+                config.isLogHungThreadOnce());
         if (managedExecutorServiceMap == null) {
             managedExecutorServiceMap = new HashMap();
         }
@@ -231,7 +233,8 @@
                 config.getThreadLifeTimeSeconds(),
                 createContextService(config.getJndiName() + "-contextservice",
                         config.getContextInfo(), config.getContextInfoEnabled(), true),
-                AbstractManagedExecutorService.RejectPolicy.ABORT);
+                AbstractManagedExecutorService.RejectPolicy.ABORT,
+                config.isLogHungThreadOnce());
         if (managedScheduledExecutorServiceMap == null) {
             managedScheduledExecutorServiceMap = new HashMap();
         }
@@ -339,7 +342,8 @@
                     0L,
                     createContextService(name + "-contextservice",
                             CONTEXT_INFO_CLASSLOADER, "true", false),
-                    AbstractManagedExecutorService.RejectPolicy.ABORT);
+                    AbstractManagedExecutorService.RejectPolicy.ABORT,
+                    false);
             internalScheduler.scheduleAtFixedRate(new HungTasksLogger(), 1L, 1L, TimeUnit.MINUTES);
         }
     }
@@ -372,23 +376,35 @@
             }
             for (ManagedExecutorServiceImpl mes: executorServices) {
                 Collection<AbstractManagedThread> hungThreads = mes.getHungThreads();
-                logHungThreads(hungThreads, mes.getManagedThreadFactory(), mes.getName());
+                logHungThreads(hungThreads, mes.isLogHungThreadOnce(), mes.getManagedThreadFactory(), mes.getName());
             }
             for (ManagedScheduledExecutorServiceImpl mses: scheduledExecutorServices) {
                 Collection<AbstractManagedThread> hungThreads = mses.getHungThreads();
-                logHungThreads(hungThreads, mses.getManagedThreadFactory(), mses.getName());
+                logHungThreads(hungThreads, mses.isLogHungThreadOnce(), mses.getManagedThreadFactory(), mses.getName());
             }
         }
 
-        private void logHungThreads(Collection<AbstractManagedThread> hungThreads, ManagedThreadFactoryImpl mtf, String mesName) {
+        private void logHungThreads(Collection<AbstractManagedThread> hungThreads, boolean logHungThreadOnce, ManagedThreadFactoryImpl mtf, String mesName) {
             if (hungThreads != null) {
                 for (AbstractManagedThread hungThread: hungThreads) {
-                    Object[] params = {hungThread.getTaskIdentityName(),
-                                       hungThread.getName(),
-                                       hungThread.getTaskRunTime(System.currentTimeMillis()) / 1000,
-                                       mtf.getHungTaskThreshold() / 1000,
-                                       mesName};
-                    logger.log(Level.WARNING, LogFacade.UNRESPONSIVE_TASK, params);
+                    boolean loggable = true;
+                    if(logHungThreadOnce && hungThread instanceof ManagedThread) {
+                        ManagedThread mt = (ManagedThread) hungThread;
+                        loggable = ! mt.isHungLogged();
+                    }
+                    if(loggable) {
+                        Object[] params = {hungThread.getTaskIdentityName(),
+                                           hungThread.getName(),
+                                           hungThread.getTaskRunTime(System.currentTimeMillis()) / 1000,
+                                           mtf.getHungTaskThreshold() / 1000,
+                                           mesName};
+                        logger.log(Level.WARNING, LogFacade.UNRESPONSIVE_TASK, params);
+
+                        if(logHungThreadOnce && hungThread instanceof ManagedThread) {
+                            ManagedThread mt = (ManagedThread) hungThread;
+                            mt.setHungLogged(true);
+                        }
+                    }
                 }
             }
         }
Index: main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedExecutorServiceConfig.java
===================================================================
--- main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedExecutorServiceConfig.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedExecutorServiceConfig.java	(Working Copy)
@@ -55,6 +55,7 @@
     private int maximumPoolSize;
     private int taskQueueCapacity;
     private long threadLifeTimeSeconds;
+    private boolean logHungThreadOnce = true;
 
     public ManagedExecutorServiceConfig(ManagedExecutorService config) {
         super(config.getJndiName(), config.getContextInfo(), config.getContextInfoEnabled());
@@ -66,6 +67,9 @@
         maximumPoolSize = parseInt(config.getMaximumPoolSize(), Integer.MAX_VALUE);
         taskQueueCapacity = parseInt(config.getTaskQueueCapacity(), Integer.MAX_VALUE);
         threadLifeTimeSeconds = parseLong(config.getThreadLifetimeSeconds(), 0L);
+        if("repeat".equalsIgnoreCase(config.getPropertyValue("log-hung-thread-policy"))) {
+            logHungThreadOnce = false;
+        }
     }
 
     public int getHungAfterSeconds() {
@@ -100,6 +104,10 @@
         return threadLifeTimeSeconds;
     }
 
+    public boolean isLogHungThreadOnce() {
+        return logHungThreadOnce;
+    }
+
     @Override
     TYPE getType() {
         return TYPE.MANAGED_EXECUTOR_SERVICE;
Index: main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedScheduledExecutorServiceConfig.java
===================================================================
--- main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedScheduledExecutorServiceConfig.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedScheduledExecutorServiceConfig.java	(Working Copy)
@@ -52,6 +52,7 @@
     private int corePoolSize;
     private long keepAliveSeconds;
     private long threadLifeTimeSeconds;
+    private boolean logHungThreadOnce = true;
 
     public ManagedScheduledExecutorServiceConfig(ManagedScheduledExecutorService config) {
         super(config.getJndiName(), config.getContextInfo(), config.getContextInfoEnabled());
@@ -61,6 +62,9 @@
         corePoolSize = parseInt(config.getCorePoolSize(), 0);
         keepAliveSeconds = parseLong(config.getKeepAliveSeconds(), 60);
         threadLifeTimeSeconds = parseLong(config.getThreadLifetimeSeconds(), 0L);
+        if("repeat".equalsIgnoreCase(config.getPropertyValue("log-hung-thread-policy"))) {
+            logHungThreadOnce = false;
+        }
     }
 
     public int getHungAfterSeconds() {
@@ -87,6 +91,10 @@
         return threadLifeTimeSeconds;
     }
 
+    public boolean isLogHungThreadOnce() {
+        return logHungThreadOnce;
+    }
+
     @Override
     public TYPE getType() {
         return TYPE.MANAGED_SCHEDULED_EXECUTOR_SERVICE;

If you want the previous behavior (before patch), set Managed(Scheduled)ExecutorService's additional property "log-hung-thread-policy" to "repeat".

asadmin set resources.managed-executor-service.<jndi-name>.property.log-hung-thread-policy=repeat
asadmin set resources.managed-scheduled-executor-service.<jndi-name>.property.log-hung-thread-policy=repeat




[GLASSFISH-21562] Changing the default value to set upper bounds of pool size of ManagedExecutorService and ManagedScheduledExecutorService Created: 16/Sep/16  Updated: 23/Jan/17

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.1
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: uno_kohei Assignee: rutujay
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

The default value of the configuration of the ManagedExecutorService is
set as follows.

ManagedExecutorService default value
core-pool-size 0
maximum-pool-size 2147483647 (Integer.MAX_VALUE)

The default value of the configuration of the ManagedScheduledExecutorService is set as follows.

ManagedScheduledExecutorService default value
core-pool-size 0

(maximum-pool-size does not exist in the ManagedScheduledExecutorService)

At the above-mentioned setting, there is no upper bounds of the number of threads in the thread pool of the ManagedExecutorService or ManagedScheduledExecutorService.
Therefore, a large amount of threads may be created by mistake of the application, and there is a risk of excessively consuming the resource of the server.

I hope to set upper bounds of the number of threads in the thread pool by changing the default value of the configuration of the ManagedExecutorService or ManagedScheduledExecutorService as follows. (64 is an example value enough for usual use)

ManagedExecutorService default value
core-pool-size 64
maximum-pool-size 64
ManagedScheduledExecutorService default value
core-pool-size 64


 Comments   
Comment by uno_kohei [ 14/Oct/16 ]

I propose a patch for changing default value.
This patch changes default value of core-pool-size and maximum-pool-size to 64.

Index: main/appserver/concurrent/concurrent-connector/src/main/java/org/glassfish/concurrent/config/ManagedExecutorService.java
===================================================================
--- main/appserver/concurrent/concurrent-connector/src/main/java/org/glassfish/concurrent/config/ManagedExecutorService.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-connector/src/main/java/org/glassfish/concurrent/config/ManagedExecutorService.java	(Working Copy)
@@ -82,7 +82,7 @@
      *
      * @return possible object is {@link String }
      */
-    @Attribute(defaultValue = ""+Integer.MAX_VALUE, dataType = Integer.class)
+    @Attribute(defaultValue = "64", dataType = Integer.class)
     @Min(value=0)
     String getMaximumPoolSize();
 
Index: main/appserver/concurrent/concurrent-connector/src/main/java/org/glassfish/concurrent/config/ManagedExecutorServiceBase.java
===================================================================
--- main/appserver/concurrent/concurrent-connector/src/main/java/org/glassfish/concurrent/config/ManagedExecutorServiceBase.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-connector/src/main/java/org/glassfish/concurrent/config/ManagedExecutorServiceBase.java	(Working Copy)
@@ -116,7 +116,7 @@
      *
      * @return possible object is {@link String }
      */
-    @Attribute(defaultValue = "0", dataType = Integer.class)
+    @Attribute(defaultValue = "64", dataType = Integer.class)
     @Min(value=0)
     String getCorePoolSize();
 
Index: main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/admin/CreateManagedExecutorService.java
===================================================================
--- main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/admin/CreateManagedExecutorService.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/admin/CreateManagedExecutorService.java	(Working Copy)
@@ -74,7 +74,7 @@
 @I18n("create.managed.executor.service")
 public class CreateManagedExecutorService extends CreateManagedExecutorServiceBase implements AdminCommand {
 
-    @Param(name="maximumpoolsize", alias="maximumPoolSize", defaultValue=""+Integer.MAX_VALUE, optional=true)
+    @Param(name="maximumpoolsize", alias="maximumPoolSize", defaultValue="64", optional=true)
     private Integer maximumpoolsize;
 
     @Param(name="taskqueuecapacity", alias="taskQueueCapacity", defaultValue=""+Integer.MAX_VALUE, optional=true)
Index: main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/admin/CreateManagedExecutorServiceBase.java
===================================================================
--- main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/admin/CreateManagedExecutorServiceBase.java	(Revision 64298)
+++ main/appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/admin/CreateManagedExecutorServiceBase.java	(Working Copy)
@@ -92,7 +92,7 @@
     @Param(name="hungafterseconds", alias="hungAfterSeconds", defaultValue="0", optional=true)
     protected Integer hungafterseconds;
 
-    @Param(name="corepoolsize", alias="corePoolSize", defaultValue="0", optional=true)
+    @Param(name="corepoolsize", alias="corePoolSize", defaultValue="64", optional=true)
     protected Integer corepoolsize;
 
     @Param(name="keepaliveseconds", alias="keepAliveSeconds", defaultValue="60", optional=true)
Index: appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedExecutorServiceConfig.java
===================================================================
--- appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedExecutorServiceConfig.java	(Revision 64298)
+++ appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedExecutorServiceConfig.java	(Working Copy)
@@ -61,9 +61,9 @@
         hungAfterSeconds = parseInt(config.getHungAfterSeconds(), 0);
         longRunningTasks = Boolean.valueOf(config.getLongRunningTasks());
         threadPriority = parseInt(config.getThreadPriority(), Thread.NORM_PRIORITY);
-        corePoolSize = parseInt(config.getCorePoolSize(), 0);
+        corePoolSize = parseInt(config.getCorePoolSize(), 64);
         keepAliveSeconds = parseLong(config.getKeepAliveSeconds(), 60);
-        maximumPoolSize = parseInt(config.getMaximumPoolSize(), Integer.MAX_VALUE);
+        maximumPoolSize = parseInt(config.getMaximumPoolSize(), 64);
         taskQueueCapacity = parseInt(config.getTaskQueueCapacity(), Integer.MAX_VALUE);
         threadLifeTimeSeconds = parseLong(config.getThreadLifetimeSeconds(), 0L);
     }
Index: appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedScheduledExecutorServiceConfig.java
===================================================================
--- appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedScheduledExecutorServiceConfig.java	(Revision 64298)
+++ appserver/concurrent/concurrent-impl/src/main/java/org/glassfish/concurrent/runtime/deployer/ManagedScheduledExecutorServiceConfig.java	(Working Copy)
@@ -58,7 +58,7 @@
         hungAfterSeconds = parseInt(config.getHungAfterSeconds(), 0);
         longRunningTasks = Boolean.valueOf(config.getLongRunningTasks());
         threadPriority = parseInt(config.getThreadPriority(), Thread.NORM_PRIORITY);
-        corePoolSize = parseInt(config.getCorePoolSize(), 0);
+        corePoolSize = parseInt(config.getCorePoolSize(), 64);
         keepAliveSeconds = parseLong(config.getKeepAliveSeconds(), 60);
         threadLifeTimeSeconds = parseLong(config.getThreadLifetimeSeconds(), 0L);
     }





[GLASSFISH-21660] Error in allocating a connection Created: 19/Jan/17  Updated: 30/Jan/17

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: tc_yopa Assignee: sameerpandit
Resolution: Unresolved Votes: 0
Labels: glassfish, jdk7, jpa, mysq
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 4.1
JPA 2.1
JDK 1.7


Tags: connectionpool, datasource, eclipselink, glassfish-4.1, jdbc, jpa, mysql

 Description   

Getting below exceptions,

[glassfish 4.1] [SEVERE] [poolmgr.system_exception] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource] [tid: _ThreadID=59105 _ThreadName=ajp-listener-1(350)] [timeMillis: 1484746152173] [levelValue: 1000] [[
RAR5031:System Exception
javax.resource.ResourceException: This Managed Connection is not valid as the physical connection is not usable
at com.sun.gjc.spi.ManagedConnectionImpl.checkIfValid(ManagedConnectionImpl.java:756)
at com.sun.gjc.spi.ManagedConnectionImpl.getLocalTransaction(ManagedConnectionImpl.java:531)
at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandle(ConnectorXAResource.java:250)

at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
at java.lang.Thread.run(Thread.java:745)
]]

[2017-01-18T18:59:12.174+0530] [glassfish 4.1] [SEVERE] [poolmgr.system_exception] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource] [tid: _ThreadID=59105 _ThreadName=ajp-listener-1(350)] [timeMillis: 1484746152174] [levelValue: 1000] [[
RAR5031:System Exception
com.sun.appserv.connectors.internal.api.PoolingException: javax.resource.ResourceException: This Managed Connection is not valid as the physical connection is not usable
at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandle(ConnectorXAResource.java:255)
at com.sun.enterprise.resource.ConnectorXAResource.start(ConnectorXAResource.java:136)
at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.enlistResource(JavaEETransactionManagerSimplified.java:386)

at com.sun.gjc.spi.ManagedConnectionImpl.getLocalTransaction(ManagedConnectionImpl.java:531)
at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandle(ConnectorXAResource.java:250)
... 96 more
]]

[glassfish 4.1] [WARNING] [poolmgr.err_enlisting_res_in_getconn] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.pool] [tid: _ThreadID=59105 _ThreadName=ajp-listener-1(350)] [timeMillis: 1484746152178] [levelValue: 900] [[
RAR7132: Unable to enlist the resource in transaction. Returned resource to pool. Pool name: [ Mysql_SOTC_Database_Pool ]]]

[2017-01-18T18:59:12.178+0530] [glassfish 4.1] [WARNING] [poolmgr.get_connection_failure] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors] [tid: _ThreadID=59105 _ThreadName=ajp-listener-1(350)] [timeMillis: 1484746152178] [levelValue: 900] [[
RAR5117 : Failed to obtain/create connection from connection pool [ Mysql_SOTC_Database_Pool ]. Reason : com.sun.appserv.connectors.internal.api.PoolingException: java.lang.RuntimeException: Got exception during XAResource.start:]]

[2017-01-18T18:59:12.179+0530] [glassfish 4.1] [WARNING] [jdbc.exc_get_conn] [javax.enterprise.resource.resourceadapter.com.sun.gjc.spi] [tid: _ThreadID=59105 _ThreadName=ajp-listener-1(350)] [timeMillis: 1484746152179] [levelValue: 900] [[
RAR5114 : Error allocating connection : [Error in allocating a connection. Cause: java.lang.RuntimeException: Got exception during XAResource.start:]]]

-javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
:Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: java.lang.RuntimeException: Got exception during XAResource.start:
-Error Code: 0

  • at org.eclipse.persistence.internal.jpa.QueryImpl.getDetailedException(QueryImpl.java:378) ~[org.eclipse.persistence.jpa.jar:?]
  • at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260) ~[org.eclipse.persistence.jpa.jar:?]
  • at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517) ~[org.eclipse.persistence.jpa.jar:?]
  • at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:400) ~[org.eclipse.persistence.jpa.jar:?]
  • at sun.reflect.GeneratedMethodAccessor6841.invoke(Unknown Source) ~[?:?]
  • at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) [nucleus-grizzly-all.jar:?]
  • at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
    -Caused by: org.eclipse.persistence.exceptions.DatabaseException:
    :Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: java.lang.RuntimeException: Got exception during XAResource.start:
    -Error Code: 0
  • at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:307) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:281) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.ResultSetMappingQuery.executeDatabaseQuery(ResultSetMappingQuery.java:316) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:798) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258) ~[org.eclipse.persistence.jpa.jar:?]
  • ... 151 more
    :Caused by: java.sql.SQLException: Error in allocating a connection. Cause: java.lang.RuntimeException: Got exception during XAResource.start:
  • at com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:121) ~[__ds_jdbc_ra.jar:?]
  • at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:307) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1625) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:613) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:281) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.ResultSetMappingQuery.executeDatabaseQuery(ResultSetMappingQuery.java:316) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:798) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258) ~[org.eclipse.persistence.jpa.jar:?]
  • ... 151 more
    :Caused by: javax.resource.spi.ResourceAllocationException: Error in allocating a connection. Cause: java.lang.RuntimeException: Got exception during XAResource.start:
  • at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:319) ~[connectors-runtime.jar:?]
  • at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:196) ~[connectors-runtime.jar:?]
  • at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:171) ~[connectors-runtime.jar:?]
  • at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:166) ~[connectors-runtime.jar:?]
  • at com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:114) ~[__ds_jdbc_ra.jar:?]
  • at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:307) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581) ~[org.eclipse.persistence.core.jar:?]


 Comments   
Comment by tc_yopa [ 19/Jan/17 ]

This occurs randomly, there is no consistent pattern to replicate it, but have observed more occurences of this during load testing cycle





[GLASSFISH-21659] No operations allowed after connection closed Created: 19/Jan/17  Updated: 30/Jan/17

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: tc_yopa Assignee: sameerpandit
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

GlassFish Server 4.1
JDK 1.7
JPA 2.1


Tags: concurrency, connectionpool, datasource, glassfish-4.1, jdbc, jpa

 Description   

Getting below error,
i have set autoReconnect=true
Changed isolation level to ReadCommitted : AppServer
Increased connection pool idle timeout value to 1200 secs from 300 secs
Connection Validation enabled
Still getting below error

-javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
:Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
-Error Code: 0

  • at org.eclipse.persistence.internal.jpa.QueryImpl.executeUpdate(QueryImpl.java:308) ~[org.eclipse.persistence.jpa.jar:?]
  • at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) [nucleus-grizzly-all.jar:?]
  • at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) [nucleus-grizzly-all.jar:?]
  • at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
    -Caused by: org.eclipse.persistence.exceptions.DatabaseException:
    :Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
    -Error Code: 0
  • at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1611) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelect(DatasourceCallQueryMechanism.java:251) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DataModifyQuery.executeDatabaseQuery(DataModifyQuery.java:85) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:798) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751) ~[org.eclipse.persistence.core.jar:?]
  • at org.eclipse.persistence.internal.jpa.QueryImpl.executeUpdate(QueryImpl.java:298) ~[org.eclipse.persistence.jpa.jar:?]
  • ... 148 more


 Comments   
Comment by tc_yopa [ 19/Jan/17 ]

This occurs randomly, there is no consistent pattern to replicate it, but have observed more occurences of this during load testing cycle

Comment by Yamini K B [ 23/Jan/17 ]

It would help to have more details on the scenario:
1. What kind of load testing?
2. Is this related to GLASSFISH-21661?

Comment by tc_yopa [ 24/Jan/17 ]

1. we did 2 kind of load testing
a. Soak test (12 hours) for 100 users
b. Stress Test (30 mins) for 500 users

we did 2-3 rounds of testing, where this issue occured 2 out of 3 times of test cycle and not always

2. Dont know whether both are related but both issues occured during same test cycle at different functional components

Comment by tc_yopa [ 24/Jan/17 ]

All the 3 issues 21661, 21660, 21659 occured at same load testing cycle, and only during load test, normally it works fine and cannot be recreated





[GLASSFISH-21661] java.lang.IllegalStateException: state.isBusy() : false Created: 19/Jan/17  Updated: 30/Jan/17

Status: Open
Project: glassfish
Component/s: concurrency
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: tc_yopa Assignee: sameerpandit
Resolution: Unresolved Votes: 0
Labels: glassfish, jpa, mysql
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glass fish 4.4
JPA 2.1
JDK 1.7


Tags: datasource, eclipselink, glassfish-4.1, jpa, mysql

 Description   

Getting below exception,

This occurs randomly, there is no consistent pattern to replicate it, but have observed more occurences of this during load testing cycle.

Error - java.lang.IllegalStateException: state.isBusy() : false
at com.sun.enterprise.resource.pool.ConnectionPool.resourceClosed(ConnectionPool.java:1008) ~[connectors-runtime.jar:?]
at com.sun.enterprise.resource.pool.PoolManagerImpl.putbackResourceToPool(PoolManagerImpl.java:426) ~[connectors-runtime.jar:?]
at com.sun.enterprise.resource.pool.PoolManagerImpl.resourceClosed(PoolManagerImpl.java:380) ~[connectors-runtime.jar:?]
at com.sun.enterprise.resource.listener.LocalTxConnectionEventListener.connectionClosed(LocalTxConnectionEventListener.java:77) ~[?:?]
at com.sun.gjc.spi.ManagedConnectionImpl.connectionClosed(ManagedConnectionImpl.java:783) ~[__ds_jdbc_ra.jar:?]
at com.sun.gjc.spi.base.ConnectionHolder.close(ConnectionHolder.java:217) ~[__ds_jdbc_ra.jar:?]
at com.sun.gjc.spi.jdbc40.ConnectionHolder40.close(ConnectionHolder40.java:587) ~[__ds_jdbc_ra.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeDatasourceConnection(DatabaseAccessor.java:493) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.closeConnection(DatasourceAccessor.java:520) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeConnection(DatabaseAccessor.java:518) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.decrementCallCount(DatasourceAccessor.java:290) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeStatement(DatabaseAccessor.java:411) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.releaseStatement(DatabaseAccessor.java:1665) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:703) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002) ~[org.eclipse.persistence.core.jar:?]
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570) ~[org.eclipse.persistence.core.jar:?]



 Comments   
Comment by Yamini K B [ 23/Jan/17 ]

Please provide more information to proceed with analysis:
1. What kind of app is deployed?
2. What does the env look like (servers in cluster? or standalone servers? resources configured?)
3. Can you please attach all the logs and domain.xml

Comment by tc_yopa [ 24/Jan/17 ]

1. Its an Web Application using EJB, Restful Services with JPA 2.1
2. StandAlone Servers with mysql datasource
3. domain.xml cannot be shared due to security issues,
please let me knw which part is needed in domain.xml





[GLASSFISH-21581] The thread pool's task queue is full, limit: 4096 Created: 07/Nov/16  Updated: 20/Feb/17

Status: In Progress
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: psgrandi Assignee: rutujay
Resolution: Unresolved Votes: 0
Labels: waiting_on_filer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows Server 2013



 Description   

Hi, we're dealing with recurrent issue with thread pool with Glassfish 4.1.1, where the pool are getting full and the domain stop responding.

Attached files:
domain.xml
server.log
thread dumps.

We did some researches for similar problems and for contingency we tried to change the thread pool size to -1, but got an error on domain startup:

Shutting down server due to startup exception
java.lang.IllegalArgumentException: poolsize < 1
at org.glassfish.grizzly.threadpool.AbstractThreadPool.<init>(AbstractThreadPool.java:141)
at org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:71)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)
at org.glassfish.grizzly.config.GenericGrizzlyListener.configureThreadPool(GenericGrizzlyListener.java:599)
at org.glassfish.grizzly.config.GenericGrizzlyListener.configure(GenericGrizzlyListener.java:307)
at com.sun.enterprise.v3.services.impl.GrizzlyProxy.initialize(GrizzlyProxy.java:124)
at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:539)
at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:490)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:326)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:374)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:471)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:228)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2072)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)



 Comments   
Comment by rutujay [ 10/Feb/17 ]

Please provide the steps to reproduce the problem.

Comment by rutujay [ 15/Feb/17 ]

The steps which I followed are,
1. started the server using asadmin start-domain

Waiting for domain1 to start .....
Successfully started the domain : domain1
domain Location: /home/yrutuja/glassfish-repo/glassfish4/glassfish/domains/domain1
Log File: /home/yrutuja/glassfish-repo/glassfish4/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.

2.Using the admin console localhost:4848 > Configurations -> server-config> Thread Pools -> hhtp-thread-pool -> max Queue Size = -1

3.Restart the domain using asadmin restart-domain

Successfully restarted the domain
Command restart-domain executed successfully.

The exception which you are facing occurs when you set max-thread-pool-size to -1, which is not advised.

If you followed some other steps other than mine, please let me know so that I can reproduce the bug.





[GLASSFISH-21128] Incorrect context in ManagedThreadFactory Created: 10/Jul/14  Updated: 20/Feb/17

Status: In Progress
Project: glassfish
Component/s: concurrency
Affects Version/s: 4.0
Fix Version/s: None

Type: Bug Priority: Major
Reporter: anthony.lai Assignee: rutujay
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

At page 3-27 of JSR236 spec, it is specified that the runnable that is executed will run with the context of the application component instance that created the ManagedThreadFactory instance.
But in the implementation of glassfish4, tasks will run with the context of the application component that called the newThread() method.






Generated at Tue Feb 28 13:59:52 UTC 2017 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.