We wrote a dummy web service that sleeps for 2 seconds. Then we set the request-timeout-seconds="1" in the domain.xml so the server interrupts the threads that are running the dummy web service. A SoapUI script was set up and we hammered the server with 40 threads calling the dummy web service every 200 ms. We used this to reproduce the original issue with the old version of Grizzly. After upgrading grizzly and corba-orb we noticed the original issue with rising CPU usage was fixed. We then started using other applications that were already deployed on the server and noticed the number of open FIFO and anon pipes continuously rose. The increase of open file descriptors happens only when we access our web applications while we are running the stress test (our dummy web service). The server starts with about 30 FIFO and 10 anon pipes. After about 8 hours of having two instances of our web app open (it pulls data from a database every 2 minutes) the number of open FIFO pipes rises to 400 and anon pipes to 200. We're running a small script on the server which checks
lsof -u glassfish | grep anon | wc -l
lsof -u glassfish | grep FIFO | wc -l
every second and prints the results out if they have changed. We hoped this would help us pinpoint the exact moment when the application did something and the number of open pipes rose. So far it's been mostly random, sometimes taking 2 or 5 or 15 minutes before there are any changes. But the number of open pipes slowly rises and will eventually reach the maximum defined in the server settings after which the server becomes unresponsive.