grizzly
  1. grizzly
  2. GRIZZLY-988

possible file descriptor leak on ConnectorHandler.connect(...)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.1
    • Component/s: framework
    • Labels:
      None
    • Environment:

      When ConnectorHandler.connect(...) fails the file descriptor is not getting released

      Activity

      Hide
      eugenb added a comment -

      We wrote a dummy web service that sleeps for 2 seconds. Then we set the request-timeout-seconds="1" in the domain.xml so the server interrupts the threads that are running the dummy web service. A SoapUI script was set up and we hammered the server with 40 threads calling the dummy web service every 200 ms. We used this to reproduce the original issue with the old version of Grizzly. After upgrading grizzly and corba-orb we noticed the original issue with rising CPU usage was fixed. We then started using other applications that were already deployed on the server and noticed the number of open FIFO and anon pipes continuously rose. The increase of open file descriptors happens only when we access our web applications while we are running the stress test (our dummy web service). The server starts with about 30 FIFO and 10 anon pipes. After about 8 hours of having two instances of our web app open (it pulls data from a database every 2 minutes) the number of open FIFO pipes rises to 400 and anon pipes to 200. We're running a small script on the server which checks

      lsof -u glassfish | grep anon | wc -l

      and

      lsof -u glassfish | grep FIFO | wc -l

      every second and prints the results out if they have changed. We hoped this would help us pinpoint the exact moment when the application did something and the number of open pipes rose. So far it's been mostly random, sometimes taking 2 or 5 or 15 minutes before there are any changes. But the number of open pipes slowly rises and will eventually reach the maximum defined in the server settings after which the server becomes unresponsive.

      Show
      eugenb added a comment - We wrote a dummy web service that sleeps for 2 seconds. Then we set the request-timeout-seconds="1" in the domain.xml so the server interrupts the threads that are running the dummy web service. A SoapUI script was set up and we hammered the server with 40 threads calling the dummy web service every 200 ms. We used this to reproduce the original issue with the old version of Grizzly. After upgrading grizzly and corba-orb we noticed the original issue with rising CPU usage was fixed. We then started using other applications that were already deployed on the server and noticed the number of open FIFO and anon pipes continuously rose. The increase of open file descriptors happens only when we access our web applications while we are running the stress test (our dummy web service). The server starts with about 30 FIFO and 10 anon pipes. After about 8 hours of having two instances of our web app open (it pulls data from a database every 2 minutes) the number of open FIFO pipes rises to 400 and anon pipes to 200. We're running a small script on the server which checks lsof -u glassfish | grep anon | wc -l and lsof -u glassfish | grep FIFO | wc -l every second and prints the results out if they have changed. We hoped this would help us pinpoint the exact moment when the application did something and the number of open pipes rose. So far it's been mostly random, sometimes taking 2 or 5 or 15 minutes before there are any changes. But the number of open pipes slowly rises and will eventually reach the maximum defined in the server settings after which the server becomes unresponsive.
      Hide
      eugenb added a comment -

      So, we have finnaly figured it out how to reproduce the issue. Here is a zip with:

      • demo application (Netbeans project) that has one web service that reads a huge file (this was pretty much the simplest way to create a load on thread, since with sleep we couldn't reproduce the issue)
      • grizzly 1.9.59 jars (that we downloaded from Maven central)
      • glassfish-cobra jar (that we downloaded from here: https://java.net/jira/browse/GLASSFISH-16217)
      • linux shell script that will monitor FIFO and ANON descriptors on file system, along with opened TCP connections for Glassfish
      • pdf file with our findings, and instructions on reproducing the issue

      https://db.tt/8LWSSZki

      Best regards

      Show
      eugenb added a comment - So, we have finnaly figured it out how to reproduce the issue. Here is a zip with: demo application (Netbeans project) that has one web service that reads a huge file (this was pretty much the simplest way to create a load on thread, since with sleep we couldn't reproduce the issue) grizzly 1.9.59 jars (that we downloaded from Maven central) glassfish-cobra jar (that we downloaded from here: https://java.net/jira/browse/GLASSFISH-16217 ) linux shell script that will monitor FIFO and ANON descriptors on file system, along with opened TCP connections for Glassfish pdf file with our findings, and instructions on reproducing the issue https://db.tt/8LWSSZki Best regards
      Hide
      vkovac added a comment - - edited

      Has this been addressed? I'm having the same issue as eugenb. It's marked as fixed but I'm still having issues even though I'm using the newer version of grizzly (1.9.59)

      Show
      vkovac added a comment - - edited Has this been addressed? I'm having the same issue as eugenb. It's marked as fixed but I'm still having issues even though I'm using the newer version of grizzly (1.9.59)
      Hide
      oleksiys added a comment -

      finally i have glassfish 3.1.2.2 compiled - will try to reproduce the issue tomorrow.

      Show
      oleksiys added a comment - finally i have glassfish 3.1.2.2 compiled - will try to reproduce the issue tomorrow.
      Hide
      oleksiys added a comment -

      i filed another issue to track the problem reported by eugenb
      https://java.net/jira/browse/GRIZZLY-1690

      Show
      oleksiys added a comment - i filed another issue to track the problem reported by eugenb https://java.net/jira/browse/GRIZZLY-1690

        People

        • Assignee:
          Unassigned
          Reporter:
          oleksiys
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved: