glassfish
  1. glassfish
  2. GLASSFISH-20387

Batch Database connection failure causes the domain/target need to be restarted every time.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0_b85
    • Fix Version/s: None
    • Component/s: sqe-test
    • Labels:
      None

      Description

      Steps:

      1) Created a cluster with an instance.
      2) Start the derby database.
      3) Deploy a batch app to the cluster and start job and wait till it gets completed.
      4) List batch job in CLI returns the completed batch job.

      Just simulated database connection failure by killing the database and started it again.

      5)Now try to list batch jobs again in CLI

      bash-4.1$ asadmin list-batch-jobs --target test-cluster -l
      remote failure: java.lang.NullPointerException
      An error occurred during replication
      Failure: Command _ListBatchJobs failed on server instance cl-instance1: remote failure: java.sql.SQLNonTransientConnectionException: Insufficient data while reading from the network - expected a minimum of 6 bytes and received only 0 bytes. The connection has been terminated.
      Command list-batch-jobs failed.

      Not sure it is expected behavior or not. Every time database connection is dead and comes back again, it requires the target needs to be restarted.

        Activity

        Hide
        Mahesh Kannan added a comment -

        Arun,
        Could you post the entire stack trace (as opposed to just the message?)

        Scott: when this exception occurs ( java.sql.SQLNonTransientConnectionException ) may be RI can close the connection and re-acquire another connection?

        Show
        Mahesh Kannan added a comment - Arun, Could you post the entire stack trace (as opposed to just the message?) Scott: when this exception occurs ( java.sql.SQLNonTransientConnectionException ) may be RI can close the connection and re-acquire another connection?
        Hide
        Mahesh Kannan added a comment -

        Arun, Can you try this on a simple web app that uses JPA? Or any other plain jdbc app?
        This way we can evaluate if the problem is specific to batch RI or is it a more of a ConnectionPool problem.

        Show
        Mahesh Kannan added a comment - Arun, Can you try this on a simple web app that uses JPA? Or any other plain jdbc app? This way we can evaluate if the problem is specific to batch RI or is it a more of a ConnectionPool problem.
        Hide
        ScottKurz added a comment -

        I'm not sure what you mean exactly by

        "Every time database connection is dead and comes back again, it requires the target needs to be restarted."

        As far as I'm aware, the batch RI doesn't cache its Connection(s) but gets a new one each time from the Datasource. It's not like we're holding onto a bad connection.

        Show
        ScottKurz added a comment - I'm not sure what you mean exactly by "Every time database connection is dead and comes back again, it requires the target needs to be restarted." As far as I'm aware, the batch RI doesn't cache its Connection(s) but gets a new one each time from the Datasource. It's not like we're holding onto a bad connection.
        Hide
        Mahesh Kannan added a comment -

        Since, Batch RI is not caching the connection, this is most likely GlassFish runtime issue.

        Arun, you need to turn on validation for the connection pool. Connection Pool config bean
        has two methods:

        @Attribute (defaultValue="false", dataType=Boolean.class)
        String getIsConnectionValidationRequired();

        @Attribute (defaultValue="table")
        @Pattern(regexp="(auto-commit|meta-data|custom-validation|table)")
        String getConnectionValidationMethod();

        Try if setting these attributes resolves the issue. (I have asked Jagadish to send
        pointers / examples for turning on connection validation)

        Show
        Mahesh Kannan added a comment - Since, Batch RI is not caching the connection, this is most likely GlassFish runtime issue. Arun, you need to turn on validation for the connection pool. Connection Pool config bean has two methods: @Attribute (defaultValue="false", dataType=Boolean.class) String getIsConnectionValidationRequired(); @Attribute (defaultValue="table") @Pattern(regexp="(auto-commit|meta-data|custom-validation|table)") String getConnectionValidationMethod(); Try if setting these attributes resolves the issue. (I have asked Jagadish to send pointers / examples for turning on connection validation)
        Hide
        Jagadish added a comment -

        Yes, you need to enable connection-validation in the "jdbc-connection-pool" of GlassFish and prefer the connection-validation-method to be "table" and provide a valid table-name that is available in the database. This will help to avoid stale connections in the pool.

        Reference :
        http://docs.oracle.com/cd/E19316-01/820-4343/abehw/index.html
        https://blogs.oracle.com/JagadishPrasath/entry/connection_validation_in_glassfish_jdbc

        Show
        Jagadish added a comment - Yes, you need to enable connection-validation in the "jdbc-connection-pool" of GlassFish and prefer the connection-validation-method to be "table" and provide a valid table-name that is available in the database. This will help to avoid stale connections in the pool. Reference : http://docs.oracle.com/cd/E19316-01/820-4343/abehw/index.html https://blogs.oracle.com/JagadishPrasath/entry/connection_validation_in_glassfish_jdbc

          People

          • Assignee:
            arunkumar_s
            Reporter:
            arunkumar_s
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: