[GLASSFISH-20387] Batch Database connection failure causes the domain/target need to be restarted every time. Created: 23/Apr/13  Updated: 25/Apr/13

Status: Open
Project: glassfish
Component/s: sqe-test
Affects Version/s: 4.0_b85
Fix Version/s: None

Type: Bug Priority: Major
Reporter: arunkumar_s Assignee: arunkumar_s
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Steps:

1) Created a cluster with an instance.
2) Start the derby database.
3) Deploy a batch app to the cluster and start job and wait till it gets completed.
4) List batch job in CLI returns the completed batch job.

Just simulated database connection failure by killing the database and started it again.

5)Now try to list batch jobs again in CLI

bash-4.1$ asadmin list-batch-jobs --target test-cluster -l
remote failure: java.lang.NullPointerException
An error occurred during replication
Failure: Command _ListBatchJobs failed on server instance cl-instance1: remote failure: java.sql.SQLNonTransientConnectionException: Insufficient data while reading from the network - expected a minimum of 6 bytes and received only 0 bytes. The connection has been terminated.
Command list-batch-jobs failed.

Not sure it is expected behavior or not. Every time database connection is dead and comes back again, it requires the target needs to be restarted.



 Comments   
Comment by Mahesh Kannan [ 23/Apr/13 ]

Arun,
Could you post the entire stack trace (as opposed to just the message?)

Scott: when this exception occurs ( java.sql.SQLNonTransientConnectionException ) may be RI can close the connection and re-acquire another connection?

Comment by Mahesh Kannan [ 23/Apr/13 ]

Arun, Can you try this on a simple web app that uses JPA? Or any other plain jdbc app?
This way we can evaluate if the problem is specific to batch RI or is it a more of a ConnectionPool problem.

Comment by ScottKurz [ 23/Apr/13 ]

I'm not sure what you mean exactly by

"Every time database connection is dead and comes back again, it requires the target needs to be restarted."

As far as I'm aware, the batch RI doesn't cache its Connection(s) but gets a new one each time from the Datasource. It's not like we're holding onto a bad connection.

Comment by Mahesh Kannan [ 24/Apr/13 ]

Since, Batch RI is not caching the connection, this is most likely GlassFish runtime issue.

Arun, you need to turn on validation for the connection pool. Connection Pool config bean
has two methods:

@Attribute (defaultValue="false", dataType=Boolean.class)
String getIsConnectionValidationRequired();

@Attribute (defaultValue="table")
@Pattern(regexp="(auto-commit|meta-data|custom-validation|table)")
String getConnectionValidationMethod();

Try if setting these attributes resolves the issue. (I have asked Jagadish to send
pointers / examples for turning on connection validation)

Comment by Jagadish [ 24/Apr/13 ]

Yes, you need to enable connection-validation in the "jdbc-connection-pool" of GlassFish and prefer the connection-validation-method to be "table" and provide a valid table-name that is available in the database. This will help to avoid stale connections in the pool.

Reference :
http://docs.oracle.com/cd/E19316-01/820-4343/abehw/index.html
https://blogs.oracle.com/JagadishPrasath/entry/connection_validation_in_glassfish_jdbc





Generated at Fri Sep 04 13:53:53 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.